@mikem any ideas (esp. in relation to #2244) Does it make sense to use these options somehow with gssapi?

do the options in /etc/krb5.conf help?

• rdns
• dns_canonicalize_hostname

Yes, setting rdns = false in /etc/krb5.conf solves this, but we don't want to have to set this option for all kerberos requests within the machines where the clients may be installed.

I've created PR #2271 to share my solution to this issue, please let me know what you think and if there's a better way of handling it.

It would be great to avoid adding more Kerberos code to Koji.

There is a "dns_canonicalize_hostname=fallback" option available in MIT kerberos 1.18+. Does this work for your use-case?

Alternatively, https://github.com/pythongssapi/requests-gssapi would give us a "Hostname Override" and "Explicit Principal" options without having to add this into Koji. See #882

Edit: I see you mentioned you tried Hostname Override for requests-kerberos above. I think I need to understand more about your DNS setup and the exact versions you're running on the client and hub in order to try to replicate this.

Technically, this isn't Kerberos code, it just picks one Hub IP address to talk to consistently. :smile: Nothing changes if you have a single Hub, and if you have several all this does is load-balance across them by session instead of by request.

I didn't try dns_canonicalize_hostname=fallback, but setting it to false didn't solve the issue. Regardless, we can't make changes to the host's krb5.conf file just to accommodate Koji.

Hostname override in requests-kerberos did not work, but I do not know if those options for requests-gssapi would work around this issue.

All our tests were done with a 1.21 Hub configured for GSSAPI authentication. We mostly tested 1.21 CLIs, but we verified that 1.19 CLIs have the same behavior. As I mentioned just now in PR #2271, the DNS alias resolves to A records for the IPs of the machines running the Hub.

python-requests-kerberos and glibc have been buggy over the years with hostname canonicalization. It would not surprise me to find that hostname_override was buggy here also. What operating system are you running on your clients and servers? What are the exact builds/NVRs for your clients' python-kerberos and python-requests-kerberos versions? I would like to set up an environment to match what you are doing.

There are two options that do not involve patching Koji:
A) Setting dns_canonicalize_hostname=fallback (really the best option imho, but not available on el7)
B) Storing all three SPN entries in both hosts' keytabs.

Would you please help me understand more about the restrictions you face in your environment that makes these two options hard? Maybe we can help write a helper script for you to combine those SPN entries into your Apache keytab (for example) if you are rotating those keytabs regularly or something.

This feature as implemented in #2271 adds complexity to an area of Koji that is already very complex, making it harder to QE effectively over time. One of the goals with dropping the old-style krbV authentication code was to reduce the test matrix and focus on fewer, well-tested solutions.

I asked about rdns on the MIT kerberos list, and the folks there convinced me that there are still users who want or need rdns = true. I think I understand more now about why you would have to set this.

If requests-kerberos does not always send the correct auth header for the server that requests+urllib3 chooses, I would like to track that bug in requests-kerberos (and verify requests-gssapi behavior as well).

Do you see these errors with simple commands like koji hello or more long-lived Koji commands?

I would much prefer to solve this in requests. The reason we migrated to requests in the first place was to avoid having to dig deeply into the transport like this.

Also of note: the krb_rdns and krb_canon_host are artifacts of the old krbV auth and I'm not sure we can sanely duplicate their original behavior in gssapi. The krb_rdns is probably ok as it pretty much lines up with the rdns setting in kerberos, but krb_canon_host is a little odd as I recall.

Here's the reproducer I'm thinking of developing outside of Koji.

1) Set up two httpd servers in a DNS round-robin configuration for "www.example.com"
2) Configure mod_auth_gssapi to protect simple static pages on both servers
3) Ensure rdns = true in /etc/krb5.conf on the client
4) Run requests to www.example.com (with requests-kerberos) in a loop

If we can get it to send the wrong auth header, it will be easier to report to the requests-kerberos issue tracker. And then of course we need to verify this with requests-gssapi.

From stepping through the urllib3 code that requests uses, socket.getaddrinfo() will return multiple records if IPv6 is enabled on the client and AAAA records are present for the server, which makes me think this could affect dual-stack environments even apart from DNS round-robin scenarios.

@ktdreyer following your comments, I have dug deeper into why we have rdns = true in our infrastructure and it seems the historical reasons for this may no longer be relevant. We're going to start the process of migrating to rdns = false, which would solve this issue for us.

Having said that, it is true that this may affect other users so it's worth following up a bit more. Your plan for a reproducer sounds good. Empirically, we found that running the client inside a docker container seemed to increase the likelihood of triggering this issue. We were also running kdestroy & kinit before each client run.

I don't think this will affect dual-stack environments because the PTR records would point to the same hostname, so the requested principal will be the same.

So, is there a call for fixing this, or is everybody fine with current state?

Closing it for now as there is no known problem in the wilderness.

Sorry to reopen this issue, but I'm looking to upgrade our Koji instance and I would like to drop our custom patch in favor of a better solution. Initially I thought we would be able to change globally to rdns = false, but it turns out we won't be able to because apparently it's not supported by FreeIPA.

To recap: when rdns = true is set, the DNS of the server to connect to gets resolved twice: once when opening the socket to connect to and again by the krb5 libraries. If the DNS is a load-balanced alias, you might get two different hosts and kerberos authentication will fail.

I can reproduce this with this simple "client" and two Koji hubs:

import sys
import logging
import requests
import requests_gssapi

logger = logging.getLogger(__name__)

logging.basicConfig(level=logging.DEBUG)

r = requests.post(
    "https://kojitesthub.host.com/kojihub/ssllogin",
    auth=requests_gssapi.HTTPSPNEGOAuth(),
    verify=False)

if r.status_code == requests.codes.ok:
    print('Login successful!')
else:
    print('Authentication failed')
    sys.exit(1)

I also patched urllib3.util.connection.create_connection()so it prints the IP address it connects to, and with some KRB5_TRACEing I get this:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): kojitesthub.host.com:443
 ** URLLIB HACK ** connected to <ip address of hosta>:443
DEBUG:urllib3.connectionpool:https://kojitesthub.host.com:443 "POST /kojihub/ssllogin HTTP/1.1" 401 381
DEBUG:requests_gssapi.gssapi_:handle_401(): Handling: 401
[368] 1615301655.463485: ccselect module realm chose cache FILE:/tmp/krb5cc_0 with client principal user@HOST.COM for server principal HTTP/hostb.host.com@HOST.COM
[368] 1615301655.463486: Getting credentials user@HOST.COM -> HTTP/hostb.host.com@HOST.COM using ccache FILE:/tmp/krb5cc_0
[368] 1615301655.463487: Retrieving user@HOST.COM -> HTTP/hostb.host.com@HOST.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[368] 1615301655.463489: Creating authenticator for user@HOST.COM -> HTTP/hostb.host.com@HOST.COM, seqnum 340164123, subkey aes256-cts/6F68, session key aes256-cts/66D2
DEBUG:requests_gssapi.gssapi_:authenticate_user(): Authorization header: Negotiate YIIQ...
DEBUG:urllib3.connectionpool:https://kojitesthub.host.com:443 "POST /kojihub/ssllogin HTTP/1.1" 401 381
DEBUG:requests_gssapi.gssapi_:authenticate_user(): returning <Response [401]>
DEBUG:requests_gssapi.gssapi_:handle_401(): returning <Response [401]>
DEBUG:requests_gssapi.gssapi_:handle_response(): returning <Response [401]>
DEBUG:requests_gssapi.gssapi_:handle_response() has seen 0 401 responses
DEBUG:requests_gssapi.gssapi_:handle_401(): Handling: 401
[368] 1615301655.463496: ccselect module realm chose cache FILE:/tmp/krb5cc_0 with client principal user@HOST.COM for server principal HTTP/hostb.host.com@HOST.COM
[368] 1615301655.463497: Getting credentials user@HOST.COM -> HTTP/hostb.host.com@HOST.COM using ccache FILE:/tmp/krb5cc_0
[368] 1615301655.463498: Retrieving user@HOST.COM -> HTTP/hostb.host.com@HOST.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[368] 1615301655.463500: Creating authenticator for user@HOST.COM -> HTTP/hostb.host.com@HOST.COM, seqnum 831550428, subkey aes256-cts/B1D9, session key aes256-cts/66D2
DEBUG:requests_gssapi.gssapi_:authenticate_user(): Authorization header: Negotiate YIIQ...
DEBUG:urllib3.connectionpool:https://kojitesthub.host.com:443 "POST /kojihub/ssllogin HTTP/1.1" 401 381
DEBUG:requests_gssapi.gssapi_:authenticate_user(): returning <Response [401]>
DEBUG:requests_gssapi.gssapi_:handle_401(): returning <Response [401]>
DEBUG:requests_gssapi.gssapi_:handle_response(): returning <Response [401]>
DEBUG:requests_gssapi.gssapi_:handle_response() has seen 1 401 responses
DEBUG:requests_gssapi.gssapi_:handle_response(): returning 401 <Response [401]>
Authentication failed

Here you can see that urllib3 resolved kojitesthub.host.com to host A, but krb5_libs resolved it to host B. Host A then gets presented a ticket for host B and authentication fails.

While playing around with my simple client, I discovered that I could make it work by simply changing line 12 to:

    auth=requests_gssapi.HTTPSPNEGOAuth(target_name=Name("HTTP/kojitesthub.host.com", NameType.kerberos_principal)),

Doing this tells gssapi to request a ticket for the service principal instead of resolving the DNS. This is essentially what the now removed option krb_rdns used to do.

In that spirit, I'd like to propose bringing that option (and krbservice) back and doing something like this. This isn't fully tested, but at least moshimoshi works, and I wanted to get some feedback before continuing down this line. This might also be a good time to drop the deprecated HTTPKerberosAuth altogether, but that would break compatibility with requests_kerberos.

By the way, is there a plan to deprecate python-requests-kerberos?

Does anybody have any thoughts on my proposal to bring back krb_rdns?

Basically yes, @mikem @julian8628 ?

I wonder if @puiterwijk has any input here?

(Hmm, pagure seems to have lost my first comment....)

It looks like we have at least version 1.2.2 of python-requests-gssapi everywhere we support. So, I wonder if there's value in any of the cases in where the gssapi module is missing.

Also, I'm not sure that the krb_rdns option is really the correct name for the behavior this implements.

To extend my comment above, the behavior enabled by krb_rdns in the patch, doesn't seem to be rdns behavior.

The krb5 option is described as follows:

rdns — If this flag is true, reverse name lookup will be used in addition to forward name lookup to canonicalizing hostnames for use in service principal names. If dns_canonicalize_hostname is set to false, this flag has no effect. The default value is true.

This code simply seems to override the target_name based on the host from the uri and the krbservice option. I don't see how it does any sort of reverse lookup or otherwise instructs the gssapi lib to do so.

Perhaps it would be cleaner to just make this conditional on the krbservice option being set and leave krb_rdns out.

At any rate, we'll need a PR before we can proceed much further

Closing due to inactivity. Reopen if you'd like to continue discussion.