#10098 Can't login to koji stg with kerberos ticket
Closed: Fixed 2 years ago by asaleh. Opened 2 years ago by asaleh.

When I run:

kdestroy -A
kinit asaleh@STG.FEDORAPROJECT.ORG
export KRB5_TRACE=/dev/stdout
koji -s https://koji.stg.fedoraproject.org/kojihub call hello

I get

[41105] 1626167889.048888: ccselect module realm chose cache KCM:1000 with client principal asaleh@STG.FEDORAPROJECT.ORG for server principal HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
[41105] 1626167889.048889: Getting credentials asaleh@STG.FEDORAPROJECT.ORG -> HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG using ccache KCM:1000
[41105] 1626167889.048890: Retrieving asaleh@STG.FEDORAPROJECT.ORG -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:1000 with result: -1765328243/Matching credential not found
[41105] 1626167889.048891: Retrieving asaleh@STG.FEDORAPROJECT.ORG -> HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG from KCM:1000 with result: -1765328243/Matching credential not found
[41105] 1626167889.048892: Retrieving asaleh@STG.FEDORAPROJECT.ORG -> krbtgt/STG.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG from KCM:1000 with result: 0/Success
[41105] 1626167889.048893: Starting with TGT for client realm: asaleh@STG.FEDORAPROJECT.ORG -> krbtgt/STG.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG
[41105] 1626167889.048894: Requesting tickets for HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG, referrals on
[41105] 1626167889.048895: Generated subkey for TGS request: aes256-cts/94EE
[41105] 1626167889.048896: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[41105] 1626167889.048898: Encoding request body and padata into FAST request
[41105] 1626167889.048899: Sending request (1032 bytes) to STG.FEDORAPROJECT.ORG
[41105] 1626167889.048900: Resolving hostname id.stg.fedoraproject.org
[41105] 1626167889.048901: TLS certificate name matched "id.stg.fedoraproject.org"
[41105] 1626167889.048902: Sending HTTPS request to https 38.145.60.33:443
[41105] 1626167889.048903: Received answer (499 bytes) from https 38.145.60.33:443
[41105] 1626167889.048904: Terminating TCP connection to https 38.145.60.33:443
[41105] 1626167889.048905: Sending DNS URI query for _kerberos.STG.FEDORAPROJECT.ORG.
[41105] 1626167889.048906: URI answer: 10 1 "krb5srv:m:kkdcp:https://id.stg.fedoraproject.org/KdcProxy/"
[41105] 1626167889.048907: Response was from primary KDC
[41105] 1626167889.048908: Decoding FAST response
[41105] 1626167889.048909: TGS request result: -1765328377/Server HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG not found in Kerberos database
[41105] 1626167889.048910: Requesting tickets for HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG, referrals off
[41105] 1626167889.048911: Generated subkey for TGS request: aes256-cts/E3FD
[41105] 1626167889.048912: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[41105] 1626167889.048914: Encoding request body and padata into FAST request
[41105] 1626167889.048915: Sending request (1032 bytes) to STG.FEDORAPROJECT.ORG
[41105] 1626167889.048916: Resolving hostname id.stg.fedoraproject.org
[41105] 1626167889.048917: TLS certificate name matched "id.stg.fedoraproject.org"
[41105] 1626167889.048918: Sending HTTPS request to https 38.145.60.33:443
[41105] 1626167890.192266: Received answer (497 bytes) from https 38.145.60.33:443
[41105] 1626167890.192267: Terminating TCP connection to https 38.145.60.33:443
[41105] 1626167890.192268: Sending DNS URI query for _kerberos.STG.FEDORAPROJECT.ORG.
[41105] 1626167890.192269: URI answer: 10 1 "krb5srv:m:kkdcp:https://id.stg.fedoraproject.org/KdcProxy/"
[41105] 1626167890.192270: Response was from primary KDC
[41105] 1626167890.192271: Decoding FAST response
[41105] 1626167890.192272: TGS request result: -1765328377/Server HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG not found in Kerberos database
[41105] 1626167890.192278: ccselect module realm chose cache KCM:1000 with client principal asaleh@STG.FEDORAPROJECT.ORG for server principal HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
[41105] 1626167890.192279: Getting credentials asaleh@STG.FEDORAPROJECT.ORG -> HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG using ccache KCM:1000
[41105] 1626167890.192280: Retrieving asaleh@STG.FEDORAPROJECT.ORG -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:1000 with result: -1765328243/Matching credential not found
[41105] 1626167890.192281: Retrieving asaleh@STG.FEDORAPROJECT.ORG -> HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG from KCM:1000 with result: -1765328243/Matching credential not found
[41105] 1626167890.192282: Retrieving asaleh@STG.FEDORAPROJECT.ORG -> krbtgt/STG.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG from KCM:1000 with result: 0/Success
[41105] 1626167890.192283: Starting with TGT for client realm: asaleh@STG.FEDORAPROJECT.ORG -> krbtgt/STG.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG
[41105] 1626167890.192284: Requesting tickets for HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG, referrals on
[41105] 1626167890.192285: Generated subkey for TGS request: aes256-cts/9BFE
[41105] 1626167890.192286: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[41105] 1626167890.192288: Encoding request body and padata into FAST request
[41105] 1626167890.192289: Sending request (1032 bytes) to STG.FEDORAPROJECT.ORG
[41105] 1626167890.192290: Resolving hostname id.stg.fedoraproject.org
[41105] 1626167890.192291: TLS certificate name matched "id.stg.fedoraproject.org"
[41105] 1626167890.192292: Sending HTTPS request to https 38.145.60.32:443
[41105] 1626167890.192293: Received answer (499 bytes) from https 38.145.60.32:443
[41105] 1626167890.192294: Terminating TCP connection to https 38.145.60.32:443
[41105] 1626167890.192295: Sending DNS URI query for _kerberos.STG.FEDORAPROJECT.ORG.
[41105] 1626167890.192296: URI answer: 10 1 "krb5srv:m:kkdcp:https://id.stg.fedoraproject.org/KdcProxy/"
[41105] 1626167890.192297: Response was from primary KDC
[41105] 1626167890.192298: Decoding FAST response
[41105] 1626167890.192299: TGS request result: -1765328377/Server HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG not found in Kerberos database
[41105] 1626167890.192300: Requesting tickets for HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG, referrals off
[41105] 1626167890.192301: Generated subkey for TGS request: aes256-cts/70F8
[41105] 1626167890.192302: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[41105] 1626167890.192304: Encoding request body and padata into FAST request
[41105] 1626167890.192305: Sending request (1032 bytes) to STG.FEDORAPROJECT.ORG
[41105] 1626167890.192306: Resolving hostname id.stg.fedoraproject.org
[41105] 1626167890.192307: TLS certificate name matched "id.stg.fedoraproject.org"
[41105] 1626167890.192308: Sending HTTPS request to https 38.145.60.33:443
[41105] 1626167891.192868: Received answer (499 bytes) from https 38.145.60.33:443
[41105] 1626167891.192869: Terminating TCP connection to https 38.145.60.33:443
[41105] 1626167891.192870: Sending DNS URI query for _kerberos.STG.FEDORAPROJECT.ORG.
[41105] 1626167891.192871: URI answer: 10 1 "krb5srv:m:kkdcp:https://id.stg.fedoraproject.org/KdcProxy/"
[41105] 1626167891.192872: Response was from primary KDC
[41105] 1626167891.192873: Decoding FAST response
[41105] 1626167891.192874: TGS request result: -1765328377/Server HTTP/wildcard.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG not found in Kerberos database
2021-07-13 11:18:11,199 [ERROR] koji: (gssapi auth failed: requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://koji.stg.fedoraproject.org/kojihub/ssllogin)
Use following documentation to debug kerberos/gssapi auth issues. https://docs.pagure.org/koji/kerberos_gssapi_debug/
2021-07-13 11:18:11,201 [ERROR] koji: GSSAPIAuthError: unable to obtain a session (gssapi auth failed: requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://koji.stg.fedoraproject.org/kojihub/ssllogin)
Use following documentation to debug kerberos/gssapi auth issues. https://docs.pagure.org/koji/kerberos_gssapi_debug/

I have tried to authorize against fasjson, and that seems to have worked:

curl -u : --negotiate https://fasjson.stg.fedoraproject.org/v1/me/

resutls in

[41147] 1626167957.293056: ccselect module realm chose cache KCM:1000 with client principal asaleh@STG.FEDORAPROJECT.ORG for server principal HTTP/fasjson.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
[41147] 1626167957.293057: Getting credentials asaleh@STG.FEDORAPROJECT.ORG -> HTTP/fasjson.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG using ccache KCM:1000
[41147] 1626167957.293058: Retrieving asaleh@STG.FEDORAPROJECT.ORG -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from KCM:1000 with result: -1765328243/Matching credential not found
[41147] 1626167957.293059: Retrieving asaleh@STG.FEDORAPROJECT.ORG -> HTTP/fasjson.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG from KCM:1000 with result: 0/Success
[41147] 1626167957.293061: Creating authenticator for asaleh@STG.FEDORAPROJECT.ORG -> HTTP/fasjson.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG, seqnum 362436137, subkey aes256-cts/A011, session key aes256-cts/14EA
{"result": {"dn": "uid=asaleh,cn=users,cn=accounts,dc=stg,dc=fedoraproject,dc=org", "username": "asaleh", "service": null, "uri": "https://fasjson.stg.fedoraproject.org/v1/users/asaleh/"}}

Metadata Update from @mohanboddu:
- Issue tagged with: medium-gain, medium-trouble, ops

2 years ago

Please check that:

rdns = false

is set in your /etc/krb5.conf file?

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

2 years ago

Yes, I have set it in [libdefaults]

My krb5.conf:

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
    spake_preauth_groups = edwards25519
    default_realm = FEDORAPROJECT.ORG
    default_ccache_name = KEYRING:persistent:%{uid}
    dns_canonicalize_hostname = true

Could this have been the sssd issue?

Can you try a 'stg-koji hello' again now and see if it's working or failing?

@asaleh is it working for you now? It's still working fine here...

Oh, I just noticed also that your krb5.conf doesn't have:

includedir /etc/krb5.conf.d/

at the top, so it would not pick up the config from fedora-packager thats in there. Can you add that and see if it works?

Any news here? Is this still happening? If so, can we schedule a time to look at it interactively and get to the bottom of it?

Hi, I have doublechecked the config and I have the includedir there. Having a interactive debug sounds good :)

Try setting dns_canonicalize_hostname = false

If that doesn't work, I guess we will need to strace the stg-koji hello and try and figure out where it's getting that wildcard instead of koji for the service principal.

Thanks so much, it seems dns_canonicalize_hostname = false was the one that fixed it.

Metadata Update from @asaleh:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

wow. Cool. I didn't think that could be it... and sorry for not thinking to try it sooner. ;(

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog