fedora-infrastructure

Clone
Source Code
GIT

#7503 fedpkg build: Kerberos authentication fails: unable to obtain a session
Closed: Fixed 5 years ago by churchyard. Opened 5 years ago by churchyard.

Closed: Fixed
Reopen Issue

I run fedpkg --release f30 build --nowait --scratch --srpm

and i get:

Kerberos authentication fails: unable to obtain a session
Could not execute build: Could not login to https://koji.fedoraproject.org/kojihub

So I kinit, it works. After 1 minute it doesn't work again. I just needed to kinit 10 times in 15 minutes.

I've seen several similar issues reported recently in various places, usually closed because kinit helps. However I expected I should kinit onece a day, not once a minute. How do i debug this?

I wonder if this could be related to the DB issues we're seeing

It seems we should be recovering at the DB level, could you see if that changes anything for you?

Monitoring the kinit need. So far so good.

Took 10 minutes and again:

Kerberos authentication fails: unable to obtain a session
Could not execute build: Could not login to https://koji.fedoraproject.org/kojihub

After 30 minutes, again.

Funny enough:

[bodhi (master %)]$ git pull && fedpkg build --nowait 
From ssh://pkgs.fedoraproject.org/rpms/bodhi
   38b20cc..294ebf7  master     -> origin/master
Updating 38b20cc..294ebf7
Fast-forward
 bodhi.spec | 90 ++++++++++++------------------------------------------------------------------------------
 1 file changed, 12 insertions(+), 78 deletions(-)
Found a gating.yaml file in the repo and it is properly configured
Kerberos authentication fails: unable to obtain a session
Could not execute build: Could not login to https://koji.fedoraproject.org/kojihub
[bodhi (master %)]$ fedpkg build --nowait 
Found a gating.yaml file in the repo and it is properly configured
Building bodhi-3.12.0-101.fc30 for rawhide
Created task: 32026227
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=32026227

There was no kinit in between, I've just tried again.

Can you try now?

It seems like it's a bug in f29 httpd where it gets into some kind of odd state and restarting it clears it up.

I'll report any odd behavior. However I'm not expecting any heavy fedpkg / Koji usage today.

So far so good. Thanks. Will reopen if anything goes south.

Metadata Update from @churchyard:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
5 years ago

I'm having the same issue now:

[pbrezina ~/workspace/packages/fedora/authselect (f29 %)]$ fedpkg -d -v scratch-build --srpm authselect-1.0.3-1.fc29.src.rpm
Initiating a koji session to https://koji.fedoraproject.org/kojihub
Logging into https://koji.fedoraproject.org/kojihub with Kerberos authentication.
Kerberos authentication fails: unable to obtain a session
Logout kojisession
Could not execute scratch_build: Could not login to https://koji.fedoraproject.org/kojihub
Traceback (most recent call last):
  File "/usr/bin/fedpkg", line 11, in <module>
    load_entry_point('fedpkg==1.36', 'console_scripts', 'fedpkg')()
  File "/usr/lib/python3.7/site-packages/fedpkg/__main__.py", line 86, in main
    sys.exit(client.args.command())
  File "/usr/lib/python3.7/site-packages/pyrpkg/cli.py", line 2282, in scratch_build
    return self.build()
  File "/usr/lib/python3.7/site-packages/pyrpkg/cli.py", line 1611, in build
    task_id = self._build(sets=sets)
  File "/usr/lib/python3.7/site-packages/fedpkg/cli.py", line 1101, in _build
    return super(fedpkgClient, self)._build(sets)
  File "/usr/lib/python3.7/site-packages/pyrpkg/cli.py", line 1657, in _build
    url = self._handle_srpm_option()
  File "/usr/lib/python3.7/site-packages/pyrpkg/cli.py", line 1539, in _handle_srpm_option
    return self._upload_srpm_for_build()
  File "/usr/lib/python3.7/site-packages/pyrpkg/cli.py", line 1519, in _upload_srpm_for_build
    self.cmd.koji_upload(self.args.srpm, uniquepath, callback=callback)
  File "/usr/lib/python3.7/site-packages/pyrpkg/__init__.py", line 2419, in koji_upload
    if not self.kojisession:
  File "/usr/lib/python3.7/site-packages/pyrpkg/__init__.py", line 255, in kojisession
    self.load_kojisession()
  File "/usr/lib/python3.7/site-packages/pyrpkg/__init__.py", line 412, in load_kojisession
    self.login_koji_session(koji_config, self._kojisession)
  File "/usr/lib/python3.7/site-packages/pyrpkg/__init__.py", line 382, in login_koji_session
    raise rpkgError('Could not login to %s' % koji_config['server'])
pyrpkg.errors.rpkgError: Could not login to https://koji.fedoraproject.org/kojihub

I was able to issue the scratch build twice in the morning, but does not work anymore.

@pbrezina Could you provide the output of klist -A please?
It is very possible that your kerberos ticket has expired and you need to re-kinit, and there's some known bugs where the refreshed tickets don't get stored until you run a full kdestroy -A.

kdestroy -A did not help

[pbrezina ~/workspace/packages/fedora/authselect (f30 %)]$ kinit pbrezina@FEDORAPROJECT.ORG
Password for pbrezina@FEDORAPROJECT.ORG: 
[pbrezina ~/workspace/packages/fedora/authselect (f30 %)]$ fedpkg scratch-build --srpm authselect-1.0.3-1.fc30.src.rpm 
Kerberos authentication fails: unable to obtain a session
Could not execute scratch_build: Could not login to https://koji.fedoraproject.org/kojihub
[pbrezina ~/workspace/packages/fedora/authselect (f30 %)]$ klist -A
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: pbrezina@FEDORAPROJECT.ORG

Valid starting       Expires              Service principal
02/26/2019 15:05:02  02/27/2019 15:05:00  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
    renew until 03/05/2019 15:05:00
02/26/2019 15:05:06  02/27/2019 15:05:00  HTTP/proxy01.fedoraproject.org@
    renew until 03/05/2019 15:05:00
02/26/2019 15:05:06  02/27/2019 15:05:00  HTTP/proxy01.fedoraproject.org@FEDORAPROJECT.ORG
    renew until 03/05/2019 15:05:00
[pbrezina ~/workspace/packages/fedora/authselect (f30 %)]$ kdestroy -A
[pbrezina ~/workspace/packages/fedora/authselect (f30 %)]$ kinit pbrezina@FEDORAPROJECT.ORG
Password for pbrezina@FEDORAPROJECT.ORG: 
[pbrezina ~/workspace/packages/fedora/authselect (f30 %)]$ klist -A
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: pbrezina@FEDORAPROJECT.ORG

Valid starting       Expires              Service principal
02/26/2019 15:05:37  02/27/2019 15:05:35  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
    renew until 03/05/2019 15:05:35
[pbrezina ~/workspace/packages/fedora/authselect (f30 %)]$ fedpkg scratch-build --srpm authselect-1.0.3-1.fc30.src.rpm 
Kerberos authentication fails: unable to obtain a session
Could not execute scratch_build: Could not login to https://koji.fedoraproject.org/kojihub
[pbrezina ~/workspace/packages/fedora/authselect (f30 %)]$

The problem still persists. I cannot create a build nor scratch build in koji.

So, based on what I see from your klist -A output, you don't have rdns = false in /etc/krb5.conf.
For Fedora services, you need the default (which is rdns = false) in there, under [libdefaults].

Thank you, this seems to help. However, man krb5.conf says that default is true (*) and I've been using fedpkg for a long time with the same Kerberos settings and I never had this issue before. Did anything change?

*rdns   If this flag is true, reverse name lookup will be used in addition to forward name lookup to  canonicalizing  hostnames  for  use  in  service  principal  names.   If
              dns_canonicalize_hostname is set to false, this flag has no effect.  The default value is true.

The default used to be false so something did change in that it is now true

So, the krb5 upstream default is True, indeed.
But Fedora's krb5-libs package has a krb5.conf that sets it to False.
So it's the Fedora default, not per se the upstream default.

Using fedpkg-1.37-1.fc30.noarch I also cannot build a scratch build now, with the same error message. I tried with the rdns change, but it prevents me to kinit with error "kinit: Password incorrect while getting initial credentials". When I change the rdns to true I can use kinit, but cannot issue fedpkg scratch-build .....

My workaround is:
a) edit /etc/krb5.conf and set rdns to true
b) do kinit
c) edit /etc/krb5.conf and set rdns to false
d) do fedpkg scratch-build ....

This is kind of suboptimal.

Err, and now I can kinit with rdns=false too (after I did kinit against company server with rdns=false). This is weird. It's with krb5-workstation-1.17-14.fc30.x86_64 if it helps.

@mcrha rdns set to true does not have any effect to the kinit.
This sounds like random failures.
Could you attach a log when you get an error with KRB5_TRACE=/dev/stderr kinit ...?

rdns set to true does not have any effect to the kinit.

That had been my observation. I'm not questioning whether it was a correct or a wrong observation.

I just retried and I cannot reproduce the failure. I agree it could be just a random failure. Let's skip it. I'm sorry for the noise.

@puiterijk I think I can provide a band-aid :smile:

Only KRB5_TRACE is not sufficient, you need -d as well

% KRB5_TRACE=/dev/stderr koji -d hello   
2019-10-09 16:03:19,573 [DEBUG] koji: Opening new requests session
2019-10-09 16:03:19,573 [DEBUG] koji: Opening new requests session
[17400] 1570601000.462935: ccselect module realm chose cache FILE:/tmp/krb5cc_11793 with client principal dchen@FEDORAPROJECT.ORG for server principal HTTP/proxy10.fedoraproject.org@FEDORAPROJECT.ORG
[17400] 1570601000.462936: Getting credentials dchen@FEDORAPROJECT.ORG -> HTTP/proxy10.fedoraproject.org@FEDORAPROJECT.ORG using ccache FILE:/tmp/krb5cc_11793
[17400] 1570601000.462937: Retrieving dchen@FEDORAPROJECT.ORG -> HTTP/proxy10.fedoraproject.org@FEDORAPROJECT.ORG from FILE:/tmp/krb5cc_11793 with result: 0/Success
[17400] 1570601000.462939: Creating authenticator for dchen@FEDORAPROJECT.ORG -> HTTP/proxy10.fedoraproject.org@FEDORAPROJECT.ORG, seqnum 1065235898, subkey aes256-cts/BCFD, session key aes256-cts/05B5
2019-10-09 16:03:21,418 [DEBUG] koji: Opening new requests session
2019-10-09 16:03:21,418 [DEBUG] koji: gssapi auth failed: requests_kerberos.exceptions.MutualAuthenticationError: Unable to authenticate <Response [200]>

Traceback (most recent call last):
  File "/usr/bin/koji", line 336, in <module>
    rv = locals()[command].__call__(options, session, args)
  File "/usr/lib/python3.6/site-packages/koji_cli/commands.py", line 7304, in handle_moshimoshi
    activate_session(session, options)
  File "/usr/lib/python3.6/site-packages/koji_cli/lib.py", line 577, in activate_session
    session.krb_login(proxyuser=runas)
  File "/usr/lib/python3.6/site-packages/koji/__init__.py", line 2209, in krb_login
    if self.gssapi_login(principal, keytab, ccache, proxyuser=proxyuser):
  File "/usr/lib/python3.6/site-packages/koji/__init__.py", line 2363, in gssapi_login
    raise AuthError('unable to obtain a session')
koji.AuthError: unable to obtain a session

It looks like the mutual auth does not work, not even optional mutual auth works, it needs to be disabled.

To apply my "band-aid",
1. open /usr/lib/python3.6/site-packages/koji/__init__.py (or corresponding python2 equivalent) as root

  1. Search def gssapi_login
  2. In the function, replace kwargs = {} with
    kwargs = { 'mutual_authentication': requests_kerberos.DISABLED } # FIXME

That's it.

Login to comment on this ticket.

Metadata
None
None
None
None
Needs Review
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.