#2422 gssapi auth broken for builders in 1.22.0
Opened 2 months ago by kevin. Modified 21 days ago

Upgraded fedora to 1.22.0, but then all the builders could no longer auth. ;(

First issue:

krbservice = host
krb_rdns = false

used to be allowed, but now cause a error on startup. No biggie, but then:

Aug 06 00:32:57 buildvm-a64-01.iad2.fedoraproject.org systemd[1]: Started Koji build server.                       
Aug 06 00:32:57 buildvm-a64-01.iad2.fedoraproject.org kojid[4228]: Traceback (most recent call last):
Aug 06 00:32:57 buildvm-a64-01.iad2.fedoraproject.org kojid[4228]:   File "/usr/sbin/kojid", line 6615, in <module>
Aug 06 00:32:57 buildvm-a64-01.iad2.fedoraproject.org kojid[4228]:     session.gssapi_login(principal=krb_principal,
Aug 06 00:32:57 buildvm-a64-01.iad2.fedoraproject.org kojid[4228]:   File "/usr/lib/python3.8/site-packages/koji/__i
nit__.py", line 2521, in gssapi_login                                                                              
Aug 06 00:32:57 buildvm-a64-01.iad2.fedoraproject.org kojid[4228]:     raise AuthError('unable to obtain a session')
Aug 06 00:32:57 buildvm-a64-01.iad2.fedoraproject.org kojid[4228]: koji.AuthError: unable to obtain a session      
Aug 06 00:32:57 buildvm-a64-01.iad2.fedoraproject.org systemd[1]: kojid.service: Main process exited, code=exited, s
tatus=1/FAILURE

kerberos debug:

[4385] 1596677505.493288: Retrieving "compile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJEC
T.ORG from FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for "co
mpile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJECT.ORG                                   
[4385] 1596677505.493292: Retrieving "compile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJEC
T.ORG from FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for "co
mpile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJECT.ORG                                   
[4385] 1596677505.493296: Retrieving "compile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJEC
T.ORG from FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for "co
mpile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJECT.ORG                                   
[4385] 1596677505.493300: Retrieving "compile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJEC
T.ORG from FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for "co
mpile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJECT.ORG                                   
Traceback (most recent call last):                                                                                  
  File "/usr/sbin/kojid", line 6615, in <module>                                                                    
    session.gssapi_login(principal=krb_principal,                                                                   
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2521, in gssapi_login                              
    raise AuthError('unable to obtain a session')                                                                   
koji.AuthError: unable to obtain a session 

Thats with:

krb_principal = compile/buildvm-a64-01.iad2.fedoraproject.org@FEDORAPROJECT.ORG
keytab = /etc/kojid/kojid.keytab

Just using the keytab:

[4395] 1596677911.046198: Retrieving compile\/buildvm-a64-01.iad2.fedoraproject.org/example.com@FEDORAPROJECT.ORG fr
om FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for compile\/bu
ildvm-a64-01.iad2.fedoraproject.org/example.com@FEDORAPROJECT.ORG                                                   
[4395] 1596677911.046202: Retrieving compile\/buildvm-a64-01.iad2.fedoraproject.org/example.com@FEDORAPROJECT.ORG fr
om FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for compile\/bu
ildvm-a64-01.iad2.fedoraproject.org/example.com@FEDORAPROJECT.ORG                                                   
[4395] 1596677911.046206: Retrieving compile\/buildvm-a64-01.iad2.fedoraproject.org/example.com@FEDORAPROJECT.ORG fr
om FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for compile\/bu
ildvm-a64-01.iad2.fedoraproject.org/example.com@FEDORAPROJECT.ORG                                                   
[4395] 1596677911.046210: Retrieving compile\/buildvm-a64-01.iad2.fedoraproject.org/example.com@FEDORAPROJECT.ORG fr
om FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for compile\/bu
ildvm-a64-01.iad2.fedoraproject.org/example.com@FEDORAPROJECT.ORG                                                   
Traceback (most recent call last):                                                                                  
  File "/usr/sbin/kojid", line 6615, in <module>                                                                    
    session.gssapi_login(principal=krb_principal,
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2521, in gssapi_login
    raise AuthError('unable to obtain a session')
koji.AuthError: unable to obtain a session

The keytab:

/etc/kojid/kojid.keytab: Kerberos Keytab file, realm=FEDORAPROJECT.ORG, principal=compile/buildvm-a64-01.iad2.fedoraproject.org, type=1, date=Thu Jun 4 14:59:00 2020, kvno=1

This may be some issue with our config. @puiterwijk may know off the top of his head, but I don't understand why it would suddently break unless requests-gssapi is doing something that requests-kerberos doesn't.


Metadata Update from @tkopecek:
- Custom field Size adjusted to None

2 months ago
krbservice = host
krb_rdns = false

Yeah, those were only supported in the old krbV auth, which was dropped. Still, we should have noted this specifically in the migration notes, and we should probably do so retroactively. I'll file a separate issue for that.

ETA: #2425

@kevin Is it just builders then? I presume cli works. Does kojira? What platform(s) are we failing on?

@julian8628 Have you hit this?

No..

does setting rdns = false in /etc/krb5.conf work?

@kevin Is it just builders then? I presume cli works. Does kojira? What platform(s) are we failing on?

builders and kojira and web auth all fail. cli works. Fedora 32 all around.

@julian8628 Have you hit this?

No..

does setting rdns = false in /etc/krb5.conf work?

no, we have that set already by default.

Please note:

[4385] 1596677505.493296: Retrieving "compile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJEC
T.ORG from FILE:/etc/kojid/kojid.keytab (vno 0, enctype 0) with result: -1765328203/No key table entry found for "co
mpile\/buildvm-a64-01.iad2.fedoraproject.org/fedoraproject.org"@FEDORAPROJECT.ORG                                   

This has nothing to do with rdns or the sorts: rdns is only applicable to determining the remote principal. If the error had been like No principal proxy01.phx2.fedoraproject.org@FEDORAPROJECT.ORG (i.e. the target principal), it would've been more likely to be rdns-related.
Given the specific error, I have a feeling that something is doing extra escaping the client principal before passing it into python-requests-gssapi.

Given the specific error, I have a feeling that something is doing extra escaping the client principal before passing it into python-requests-gssapi.

I don't suppose... that something could be putting the quotes into the kojid.conf file?

This is a bug in python3-gssapi (which underpins python3-requests-gssapi) I believe.
It automatically appends /buildvm-a64-01.iad2.fedoraproject.org@FEDORAPROJECT.ORG to whatever is passed in as principal.
Koji passes the principal exactly correct to python-requests-gssapi, which passes it as-is (after converting to a GssapiName to python-gssapi, and then the gssapi.raw.creds part automatically adds random stuff and escapes.
I have not dived further into this, but I'm relatively sure that this is not a Koji bug.

In the Fedora case, it turned out setting krb_principal = compile made python-gssapi append the exact correct things to get compile/buildvm-.....@FEDORAPROJECT.ORG and the builder to start succesfully.

Sadly it turns out that setting it to compile isn't a full workaround, kojira also fails auth, but it's principal is 'kojira/koji.fedoraproject.org@FEDORAPROJECT.ORG' :(

Without having looked at the code, that sounds like the wrong NameType gets used. I will look further.

Yeah, I think it's the wrong NameType. You can check this by modifying compat.py (approximately /usr/lib/python3.7/site-packages/requests_gssapi/compat.py ) - on line 49, replace gssapi.NameType.hostbased_service with gssapi.NameType.user.

If someone can confirm that fixes it, I'll make a fix upstream and release to Fedora.

Yes, I can confirm that does fix it. I guess thats in python-requests-gssapi?

For future reference, this appears to be the upstream fix: https://github.com/pythongssapi/requests-gssapi/pull/28

Correct, that's the fix I wrote upstream, but there's nothing in it beyond the patch above and test suite fallout.

I confirm https://bodhi.fedoraproject.org/updates/FEDORA-2020-134329dd96 fixes this issue for me in Fedora 32 with kojira and kojid in my private Koji instance.

Do I understand it correctly that the issue is solved by requests-gssapi fix or do we want to plan this for 1.22.1?

Yep. It's fixed here now with requests-gssapi fix. Might be something to note in release notes or something for others that hit it?

and I might push out a new fedora/epel package with a requires for the new requests-gssapi version...

But otherwise I think we can close this. Thanks for all the help everyone... very appreciated.

Am I correct in thinking that this bug has been in requests-gssapi since the beginning? If so, we should probably have Koji check the lib version before attempting to use keytab auth.

Note that you don't have to run kojid to verify this. The cli will show the same error if you configure it to use keytab authentication.

Mike McLean pagure@pagure.io writes:

Am I correct in thinking that this bug has been in requests-gssapi
since the beginning? If so, we should probably have Koji check the lib
version before attempting to use keytab auth.

I'm against this practice in general, but this isn't my project :)

I'm against this practice in general, but this isn't my project :)

It's not ideal, but neither is showing a cryptic error to the user because there is a bug in the lib installed on their system and we can tell them that with a simple check. What would you suggest instead?

Metadata Update from @tkopecek:
- Issue set to the milestone: 1.24

21 days ago

Login to comment on this ticket.

Metadata