The vast majority of our hosts are Kerberized NFS clients with /home auto-mounted on-demand from an NFS server which requires sec=krb5p.

For this reason, it is critical for us that GSSAPI credential delegation works properly for remote logins via ssh. If it does not, the user will be denied access to their /home directory upon logging in to a host remotely.

For our RHEL7 and RHEL8 hosts, credential delegation works properly if using the kernel persistent keyring for the Kerberos credential cache. For our RHEL8 hosts, we are testing whether we can replace the kernel persistent keyring cache with KCM.

On RHEL8, we're using the latest openssh-server and sssd-kcm packages:

$ rpm -q openssh-server sssd-kcm
openssh-server-7.8p1-4.el8.x86_64
sssd-kcm-2.0.0-43.el8_0.3.x86_64

We have encountered an issue with OpenSSH sshd credential delegation when using KCM as the Kerberos credential cache. Specifically: if credentials already exist in KCM for the user in question, KCM always discards the incoming credentials from sshd in preference to the credentials that already exist in the cache. This is true even if the existing credentials in KCM are expired.

Here's a demonstration of the issue.

On my desktop host, I have current credentials cached:

$ klist

Ticket cache: KEYRING:persistent:123456789:krb_ccache_fgQUgAM
Default principal: username@EXAMPLE.ORG

Valid starting       Expires              Service principal
2019-05-24 11:26:29  2019-05-25 11:26:25  krbtgt/EXAMPLE.ORG@EXAMPLE.ORG
        renew until 2019-05-28 13:38:47

But if I ssh to a particular remote RHEL8 host and run klist, this is what I see:

$ ssh remoteserver.example.org klist
Could not chdir to home directory /home/username: Permission denied
/usr/bin/xauth:  timeout in locking authority file /home/username/.Xauthority
bash: /home/username/.bashrc: Permission denied
Ticket cache: KCM:987654321:90786
Default principal: username@EXAMPLE.ORG

Valid starting       Expires              Service principal
2019-05-22 01:09:06  2019-05-22 23:39:49  nfs/nfsserver.example.org@EXAMPLE.ORG
        renew until 2019-05-27 11:44:41
2019-05-21 23:44:41  2019-05-22 23:39:49  krbtgt/EXAMPLE.ORG@EXAMPLE.ORG
        renew until 2019-05-27 11:44:41
2019-05-22 17:38:45  2019-05-22 23:39:49  cifs/dfsserver.example.org@EXAMPLE.ORG
        renew until 2019-05-27 11:44:41

The credentials KCM is caching are from a login session 2 days ago. They are expired, which is why I am denied access to my auto-mounted NFS home directory. But KCM discarded the new, unexpired, incoming credentials from sshd, preferring instead to cling to the expired credentials that were already cached.

If I open a separate login session on remoteserver.example.org and run kdestroy -A, then (and only then) will KCM accept the new forwarded credentials from sshd:

On remoteserver.example.org:

$ kdestroy -A
$ klist -A

Now, repeating the ssh klist command:

$ ssh remoteserver.example.org klist
Ticket cache: KCM:987654321:39355
Default principal: username@EXAMPLE.ORG

Valid starting       Expires              Service principal
2019-05-24 12:27:22  2019-05-25 11:26:25  krbtgt/EXAMPLE.ORG@EXAMPLE.ORG
        renew until 2019-05-28 13:38:47
2019-05-24 12:27:23  2019-05-25 11:26:25  nfs/nfsserver.example.org@EXAMPLE.ORG
        renew until 2019-05-28 13:38:47

Voilà: KCM accepts the new forwarded credentials, and thus I have access to my auto-mounted NFS home directory, so there are no permission denied errors.

So, in summary, KCM's behavior of never accepting new delegated credentials if there are existing credentials already in the cache breaks remote logins, and is a dealbreaker for using KCM. It means that if you remotely login to host that is using KCM as the Kerberos credentials cache via sshd, credential delegation only works properly the very first time. After that, credential delegation will always fail. And when the existing delegated credentials expire, any subsequent attempt to use those credentials within that login session will fail, producing authentication errors.

It is important to note that this is a subtlety different problem than the issue of KCM not automatically renewing credentials that it did not acquire itself (e.g., through pam_sss.so). Even if KCM did automatically renew all credentials, that would not solve this problem, because eventually KCM's existing credentials would expire, and then this problem would reoccur.

Similarly, adding expired-credentials-reaper functionality to KCM would not address this issue, either, unless the reaper check was activated on-demand, every time credentials were accessed. On-demand reaping would incur a performance penalty, and since KCM is already about an order of magnitude slower than using the kernel persistent keyring, this is undesirable. (I'd like to see a expired credentials reaper, but it should be a background task, the same as the credential renewal task).

The real issue here is that KCM needs to be smarter about what it does with delegated credentials when there are already existing credentials in the cache. KCM's current behavior (never take the new credentials; always keep the existing credentials) is clearly wrong, and leads to massive breakage for sites that rely on credential delegation to work.

But it is not clear that the inverse behavior—always overwriting the existing credentials with the new credentials—is necessarily the correct behavior, either. For example, what if the incoming credentials expire in 5 minutes, while the existing credentials do not expire for another 20 hours? Even if a future KCM renewed all credentials, unless the site has set krb5_renew_interval to something unreasonably short (e.g., 3 minutes), this will create a window where the existing credentials have expired.

Perhaps the best solution would be a new option for KCM. E.g.:

delegation_acceptance_criteria (string)

Controls how KCM behaves when a new credentials delegation request is made for a user (e.g., via sshd), but there are already existing credentials for the user in the cache.

If set to always, KCM always accepts the new credentials being delegated, overwriting the existing credentials.

If set to newer, KCM replaces the existing credentials with the new credentials, but only if the new credentials are newer (that is, have an expiration date that is farther in the future). If the existing credentials have a longer lifetime remaining, they are retained, and the incoming credentials are discarded.

If set to never, KCM always discards the new credentials being delegated, and keeps the existing credentials. This is true even if the existing credentials have expired but the new credentials have not. (This is probably not what you want.)

If not specified, the default is always.

Note that if there are no existing credentials already in the cache, KCM always accepts the new credentials. This option only affects KCM's behavior when a new delegation request is made for a user and there are already existing credentials for the user in the cache.

I'm not sure that never should even be an option. Yes, it's KCM's current behavior, but I cannot think of a use case where pre-existing credentials should always win, even if expired.

We're treating this as a low-priority issue, because the easiest work-around is to simply go back to using the kernel persistent keyring for the Kerberos credentials cache. But the advantages of KCM over the kernel persistent keyring are compelling, so we'd like to eventually move to KCM.