#10666 problems with Kerberized web access
Closed: Fixed 2 years ago by kevin. Opened 2 years ago by loveshack.

I've been trying Kerberos authN for web access following the instructions for copr. (This is with Debian 11's Firefox 91.8.0esr and krb5-user 1.18.3, in case that matters.)

First I tried copr via "login with gssapi" and got a 401 error. Debug output showed

[2364047] 1651143627.433333: Retrying loveshack@FEDORAPROJECT.ORG -> HTTP/copr.fedorainfracloud.org@FEDORAPROJECT.ORG with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_1000)
[2364047] 1651143627.433334: Server has referral realm; starting with HTTP/copr.fedorainfracloud.org@FEDORAPROJECT.ORG
[2364047] 1651143627.433335: Retrieving loveshack@FEDORAPROJECT.ORG -> krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG from FILE:/tmp/krb5cc_1000 with result: 0/Success
[2364047] 1651143627.433336: Starting with TGT for client realm: loveshack@FEDORAPROJECT.ORG -> krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
[2364047] 1651143627.433337: Requesting tickets for HTTP/copr.fedorainfracloud.org@FEDORAPROJECT.ORG, referrals on
[2364047] 1651143627.433338: Generated subkey for TGS request: aes256-cts/B105
[2364047] 1651143627.433339: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-cts, aes128-sha2, camellia128-cts
[2364047] 1651143627.433341: Encoding request body and padata into FAST request
[2364047] 1651143627.433342: Sending request (979 bytes) to FEDORAPROJECT.ORG
[2364047] 1651143627.433343: Resolving hostname id.fedoraproject.org
[2364047] 1651143627.433344: TLS certificate name matched "id.fedoraproject.org"
[2364047] 1651143627.433345: Sending HTTPS request to https 38.145.60.20:443
[2364047] 1651143628.061736: Received answer (475 bytes) from https 38.145.60.20:443
[2364047] 1651143628.061737: Terminating TCP connection to https 38.145.60.20:443
[2364047] 1651143628.061738: Sending DNS URI query for _kerberos.FEDORAPROJECT.ORG.
[2364047] 1651143628.061739: URI answer: 10 1 "krb5srv:m:kkdcp:https://id.fedoraproject.org/KdcProxy/"
[2364047] 1651143628.061740: Response was from master KDC
[2364047] 1651143628.061741: Decoding FAST response
[2364047] 1651143628.061742: TGS request result: -1765328377/Server HTTP/copr.fedorainfracloud.org@FEDORAPROJECT.ORG not found in Kerberos database
[2364047] 1651143628.061743: Local realm referral failed; trying fallback realm FEDORAINFRACLOUD.ORG

It appears to be specific to copr, but presumably it's not a general problem. I tried the koji web frontend afterwards, which worked, getting me service tickets for HTTP/id.fedoraproject.org and HTTP/koji.fedoraproject.org. Do you have any suggestions what might be wrong for copr and how to debug it, assuming it's at my end? I have long history with kerberos, but it's years since I could use spnego and there still seems to be very little practical information around (I'd be interested in any good source). I'm glad to see it in use, anyway!

A related problem: Later I tried to log into bugzilla with fedora credentials and got a 500 error from id.fedoraproject.org that seemed to be associated with the previous attempts (which were in a new session). I tried kdestroy, with no luck, and then cleared the fedoraproject.org and redhat.com cookies and found I could then get a Kerberos login with a new TGT.

Incidentally, it might be worth mentioning in instructions that browser sandboxing can break this. I had to test without my normal firejail and haven't yet had investigated whitelisting in firejail to make it work.


So, oddly I can duplicate in firefox, but not chrome. :(

I wonder if this is a firefox bug. It definitely used to work...

cc @praiskup

Working here with firefox-99.0-1.fc36.x86_64 without firejail.

In command line can be tested with:

KRB5_TRACE=/dev/stderr curl -u : --negotiate  https://copr.fedorainfracloud.org/api_3/gssapi_login/

Debian 11's Firefox 91.8.0esr

I can not easily test this configuration ^^^. I know that in Fedora we have configured FAST which seems to be last useful log entry in the log above.

My log continues like:

[3104056] 1651209857.017887: Decoding FAST response
[3104056] 1651209857.017888: FAST reply key: aes256-cts/7DAE
[3104056] 1651209857.017889: TGS reply is for praiskup@FEDORAPROJECT.ORG -> HTTP/copr-fe.aws.fedoraproject.org@FEDORAPROJECT.ORG with session key aes256-cts/BF2E
[3104056] 1651209857.017890: TGS request result: 0/Success
...

Configuration is here and this conf plugin.

For taking tickets, we use fkinit script. (I have two-auth enabled)

$ klist
Ticket cache: KCM:17122
Default principal: praiskup@FEDORAPROJECT.ORG

Valid starting       Expires              Service principal
04/29/2022 07:21:22  04/30/2022 07:21:07  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
        renew until 05/06/2022 07:21:07

I'm not sure otherwise, perhaps some expert @abbra ?

@kevin wrote:

So, oddly I can duplicate in firefox, but not chrome. :(

Meh, but you are on Fedora I guess, with the pre-installed fedora-packager configuration.
Therefore I'm lost here :-/

From logs in the comments above, the difference is in the client behavior to not follow CNAME resolution and using HTTP/copr... principal instead of resolving a CNAME of copr.... hostname and using HTTP/copr-fe... target principal.

I am using dns_canonicalize_hostname = fallback and I am failing to get the HTTP/copr.fedorainfracloud.org ticket but succeeding with HTTP/copr-fe.aws... one because of the fallback.

[I initially sent this in a mail reply, and that got bounced by pagure for some unspecified policy violation -- heaven knows what. I'll make a separate issue for that anyway.]

Thanks for the hints. I'm a bit confused now how the temporary profile
I was trying with was configured. I thought I was following
https://fedoraproject.org/wiki/Infrastructure/Kerberos but I think I
must have had network.negotiate-auth.trusted-uris set somehow.

Anyhow, if I set trusted-uris to
'.fedoraproject.org,.fedorainfracloud.org,.redhat.com (the
instructions for copr do actually mention adding to it) and remove
dns_canonicalize_hostname = false from krb5.conf, I can do gssapi to
copr. However, I still get the 500 error subsequently trying to access
bugzilla.

I also looked at RHEL8 (a bit painful) on the remote VM I use with an
under-resourced laptop this end. That works, and I found that
trusted-uris for its firefox installation is set to https://, which
the wiki page above says is only necessary for old firefox. It looks as
if that page needs review, at least for that and the
dns_canonicalize_hostname setting.

The main thing I don't understand is the significance of trusted-uris,
and specifically whether it's a security issue to have it set to allow
all sites, if that's what https:// means; I'd assume it is, given the
need to set it, but I know there are Red Hat experts. (Obviously not
your problem, but it's not even documented anywhere I've found what
format is accepted in the firefox config, which is obviously not just
URIs...)

HTH. Let me know if there's anything I can do to diagnose the bugzilla
problem. It's not a big deal being able to have spnego working in this
case, of course, but it's a nuisance if having it in one place breaks
another.

I have said years ago that we should be using just 'https://' as a prefix and we did at that as a default in Fedora and RHEL. It is secure enough because the only point who will ever see you are attempting to talk to an out of realm host is your KDC. If your KDC does not have cross realm trust with something that owns the principal you are asking for, the whole processing would stop here. The target host would never see it.

Red Hat IT folks ignored our recommendations and went for more complex configuration. It is their choice, as they have to support it you definitely don't need to set individual domains.

FYI, bugzilla auth should be fixed now. (It has nothing to do with kerberos, it's using SAML2 and there's a bug on our side that sometimes causes the SAML2 data to be empty. I've refreshed it and will be looking to figure out again how it happened)

So, it sounds like this is fixed now? (Except I am not sure why I am seeing this with firefox)

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

2 years ago

ok, I got a new TGT and everything is working fine for me with firefox now too.

I think this is solved now? please re-open if there's further work to be done.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog