I wrote a quick and dirty script to attempt to figure out the "time" after which the failure condition occurs. It appears to be 15 minutes:

groups: 91
at Mon Dec 4 11:38:28 AST 2017
groups: 2
at Mon Dec 4 11:53:33 AST 2017

With 15 minutes in mind, I returned to the logs (debug level 7), and see

(Mon Dec 4 11:38:13 2017) [sssd[be[my.domain.com]]] [sdap_get_tgt_recv] (0x0400): Child responded: 0 [FILE:/var/lib/sss/db/ccache_MY.DOMAIN.COM], expired on [1512437893]
(Mon Dec 4 11:38:13 2017) [sssd[be[my.domain.com]]] [sdap_cli_auth_step] (0x0100): expire timeout is 900
(Mon Dec 4 11:38:13 2017) [sssd[be[my.domain.com]]] [sdap_cli_auth_step] (0x1000): the connection will expire at 1512402793

so it seems as if the cache (upon expiry) can't be properly updated? Yet it can be properly populated on clean start?

The expire timeout in the log messages of sdap_cli_auth_step() is the lifetime of the Kerberos ticket and the connection to the LDAP server, this is not related to cached data.

The 'Incomplete group object' error message indicates that a group the user is a member of has a missing GID or GID=0 and the group is not properly flagged as non-POSIX group in the cache. Do you know what kind of group Users@MY.DOMAIN is in AD? The backend is the only process which should update the GID, so please check sssd_my.domain.log for any activity for this group. Feel free to attach the (sanitized) log to this ticket so that I can have a look as well.

so...it's not specifically 15 minutes:

groups: 91
at Mon Dec 4 15:01:31 AST 2017
groups: 2
at Mon Dec 4 15:06:31 AST 2017

but it certainly smells like multiples of 300 seconds (ldap_enumeration_refresh_timeout ?)...I continue to peruse logs.

edit:
This 300 second (5 min) failure was when explicitly setting ldap_user_search_base. Presumably, then, the 15 minute (900) second version is perhaps due to reconnection_retries = 3.

log analysis continues

during initial/success -> failure

egrep 'nss_protocol_fill_initgr|enumeration'

(Mon Dec 4 16:49:14 2017) [sssd[be[my.domain.com]]] [dp_get_options] (0x0400): Option ldap_enumeration_refresh_timeout has value 300
(Mon Dec 4 16:49:14 2017) [sssd[be[my.domain.com]]] [dp_get_options] (0x0400): Option ldap_enumeration_search_timeout has value 60
(Mon Dec 4 16:49:14 2017) [sssd[be[my.domain.com]]] [sdap_id_setup_tasks] (0x0400): Setting up enumeration for my.domain.com
(Mon Dec 4 16:49:14 2017) [sssd[be[my.domain.com]]] [be_ptask_create] (0x0400): Periodic task [enumeration] was created
(Mon Dec 4 16:49:14 2017) [sssd[be[my.domain.com]]] [be_ptask_schedule] (0x0400): Task [enumeration]: scheduling task 0 seconds from now [1512420554]
(Mon Dec 4 16:49:14 2017) [sssd[be[my.domain.com]]] [be_ptask_execute] (0x0400): Task [enumeration]: executing task, timeout 300 seconds
(Mon Dec 4 16:49:23 2017) [sssd[be[my.domain.com]]] [be_ptask_done] (0x0400): Task [enumeration]: finished successfully
(Mon Dec 4 16:49:23 2017) [sssd[be[my.domain.com]]] [be_ptask_schedule] (0x0400): Task [enumeration]: scheduling task 300 seconds from last execution time [1512420854]
(Mon Dec 4 16:54:14 2017) [sssd[be[my.domain.com]]] [be_ptask_execute] (0x0400): Task [enumeration]: executing task, timeout 300 seconds
(Mon Dec 4 16:54:14 2017) [sssd[be[my.domain.com]]] [be_ptask_done] (0x0400): Task [enumeration]: finished successfully
(Mon Dec 4 16:54:14 2017) [sssd[be[my.domain.com]]] [be_ptask_schedule] (0x0400): Task [enumeration]: scheduling task 300 seconds from last execution time [1512421154]
(Mon Dec 4 16:59:14 2017) [sssd[be[my.domain.com]]] [be_ptask_execute] (0x0400): Task [enumeration]: executing task, timeout 300 seconds
(Mon Dec 4 16:59:14 2017) [sssd[be[my.domain.com]]] [be_ptask_done] (0x0400): Task [enumeration]: finished successfully
(Mon Dec 4 16:59:14 2017) [sssd[be[my.domain.com]]] [be_ptask_schedule] (0x0400): Task [enumeration]: scheduling task 300 seconds from last execution time [1512421454]
(Mon Dec 4 17:04:14 2017) [sssd[be[my.domain.com]]] [be_ptask_execute] (0x0400): Task [enumeration]: executing task, timeout 300 seconds
(Mon Dec 4 17:04:19 2017) [sssd[be[my.domain.com]]] [be_ptask_done] (0x0400): Task [enumeration]: finished successfully
(Mon Dec 4 17:04:19 2017) [sssd[be[my.domain.com]]] [be_ptask_schedule] (0x0400): Task [enumeration]: scheduling task 300 seconds from last execution time [1512421754]
(Mon Dec 4 17:04:29 2017) [sssd[be[my.domain.com]]] [be_ptask_destructor] (0x0400): Terminating periodic task [enumeration]

(Mon Dec 4 17:04:29 2017) [sssd[nss]] [nss_protocol_fill_initgr] (0x0020): Incomplete group object [name=Users@my.domain.com,cn=groups,cn=my.domain.com,cn=sysdb] for initgroups! Aborting.

./test.sh
groups: 92
at Mon Dec 4 16:49:25 AST 2017
groups: 2
at Mon Dec 4 17:04:29 AST 2017

!/usr/bin/env bash

set -e

systemctl stop sssd
rm -rf /var/lib/sss/{mc,db}/ /var/log/sssd/
systemctl restart sssd

groupcount=$(id MYADUSER|tr ',' '\n' |wc -l)
echo -e "groups: ${groupcount}\n\tat $(date)"

while true ; do
groupcount=$(id MYADUSER|tr ',' '\n' |wc -l)
if [[ "${groupcount}" -lt 10 ]] ; then
echo -e "groups: ${groupcount}\n\tat $(date)"
break
fi
sleep 30
done

systemctl stop sssd
sleep 5
rsync -avP /var/log/sssd /var/tmp/
systemctl restart sssd

I missed that you have 'enumerate = True'.

Can you check if it happens with 'enumerate = False' as well?

BTW if you use sssd with Active Directory then you might try to disable lookup in global catalog or replicate POSIX attributes to GC.

man sssd-ad -> ad_enable_gc

But I agree that sanitized log files + sssd configuration would be helpful.

BTW you might check following page https://docs.pagure.org/SSSD.sssd/users/troubleshooting.html or even more on here https://pagure.io/docs/SSSD/sssd/

interesting. after ~3 years of successfully running SSSD in production with enumeration = true, now setting it to false seems to prevent the loss of groups described in initial ticket description. I continue to monitor testing and will confirm

On (05/12/17 15:34), Robin P Blanchard wrote:

interesting. after ~3 years of successfully running SSSD in production with enumeration = true, now setting it to false seems to prevent the loss of groups described in initial ticket description. I continue to monitor testing and will confirm

I would expect that something was changed on directory server after 3 years :-)

LS

any theories as to why it would enumerate successfully on clean (no cache present) start, then subsequently fail (and never self-resolve) ?

On (05/12/17 16:11), Robin P Blanchard wrote:

any theories as to why it would enumerate successfully on clean (no cache present) start, then subsequently fail ?

It is impossible to say without more data/logs. I had only a wild guess.

LS

sadly, enumerate = false ultimately leads to the same end-state: all supplemental groups disappear. will attempt to sanitize the mountain of logs....

We would really like to help but we can't.

Please either provide sanitized log files + steps which you did
https://docs.pagure.org/SSSD.sssd/users/reporting_bugs.html

Or try to follow troubleshooting page which might help you to find problem yourself.
https://docs.pagure.org/SSSD.sssd/users/troubleshooting.html

I believe this may be closed. The final failed test was inappropriately run as an AD user (for whom the SSSD config changed whilst being logged in). enumerate = false has since returned all groups.

thank you for listening, your suggestions, and your patience.

I believe this may be closed. The final failed test was inappropriately run as an AD user (for whom the SSSD config changed whilst being logged in). enumerate = false has since returned all groups.
thank you for listening, your suggestions, and your patience.

I'm glad it works now.

SSSD is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in SSSD's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/SSSD/sssd/issues/4616

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.