The Problem

I ran the following test program (where "sgallagh" is a username in the SSSD cache, not /etc/passwd):

#include <pwd.h>

int main(int argc, char **argv)
{
    size_t i;
    for (i = 0; i < 100000; i++) {
        getpwnam("sgallagh");
    }
}

With {{{nsswitch.conf}}} set to:

passwd:     files sss

and nscd disabled, the script runs for:

real    0m1.404s
user    0m0.756s
sys 0m0.637s

If I reverse the order in nsswitch.conf:

passwd:     sss files

the results are several orders of magnitude faster:

real    0m0.067s
user    0m0.034s
sys 0m0.032s

The reason for this is that we recommend that users disable nscd when using SSSD, but classically nscd would be used to also maintain an in-memory fast cache for the contents of /etc/passwd, /etc/group, etc.

The problem with ordering SSSD first in nsswitch.conf in the current situation is that we will be changing behavior (currently, it's expected that local entries should always be accessed before remote ones, thus effectively allowing local overrides.

Proposed Solution

SSSD provides a reliable fast cache these days. I think that we should modify the NSS Responder so that it behaves in the following manner:

1) Take code from the "proxy" provider and dlopen() nss_files.so.2
2) Populate the SSSD fastcache with the results of a {{{[get|set|end]pwent()}}} routine (along with groups and services). (This may require adding code to the fastcache to allow entries to be persistent-unless-deleted)
3) Add an inotify watch (and fallback to polling) on each of {{{/etc/passwd}}}, {{{/etc/group}}}, {{{/etc/services}}} and reread the contents if the file has been modified. We should update the contents of the persistent fastcache with these new entries.

With the fastcache answering the entries for the local files before processing gets to SSSD, we can provide comparable performance to nscd for handling local files and also allow much faster lookups to the SSSD entries. With this change, we should modify nsswitch.conf to be:

passwd:     sss files
shadow:     files sss
group:      sss files
initgroups: files [notfound=ignore] sss
netgroups:  files sss

(note: shadow is left alone because SSSD does not respond to it. Netgroups are left alone because there exists no present interface to get all netgroups, so we would have to write our own. We could consider that as a separate enhancement if we choose to.)

Non-goals

We will not be reordering processing of the PAM stack, so pam_unix.so should remain sorted ahead of pam_sss.so.

I am explicitly not reordering {{{initgroups()}}} lookups as well, because this is most often used during a login where it is acceptable for there to be a slightly longer lookup time in favor of having the most up-to-date information available. This can be revisited in the future.

Potential Risks

The primary risk here is with reordering the nsswitch configuration. We need to make sure that we test this configuration fully both during initial system startup and when SSSD is shut down or crashed to ensure that we are never locking out local users (such as root).

We need to figure out how to address the following edge-case:
1) SSSD has been running and has created the fast cache.
2) SSSD has crashed or otherwise been shut down.
3) A user is deleted or modified in the /etc/passwd file using a tool such as libuser.
4) The fastcache continues to return the old user entry.

We do not need to worry about newly-added entries, as they should just fall through to the 'files' lookup (but that should be tested).

The deleted user is probably less of an issue than the modified user. A deleted user might still show up in a lookup, but they are not going to be able to log in, so there should not be a security risk. A modified user or group entry is a greater concern.

Some options:
- Add logic to the sss_client to check whether SSSD is alive before allowing access to the fastcache. This would probably cause a slowdown on all lookups, so it's not really an option.
- Extend libuser (and friends) so that it deletes entries from our fastcache when modifications are made to entries. This might be a lot of work and a lot of places to touch. A variant of this option would be for us to instruct libuser to always delete our /var/lib/sss/mc/* files any time they perform a {{{write}}} action.

Other solutions are surely welcome.