SSSD / sssd

Clone

#3534 caching issues with authorized_keys
Closed: Invalid 6 years ago Opened 6 years ago by vorgangsnummer.

Closed: Invalid
Reopen Issue

Running sssd version 1.15.2 or 1.15.3 on CentOS 7.4.1708.

I have a working sssd setup which enables me to sign in using SSH public keys stored in Active Directory.

User are able to log in only once during cache lifetime (by default 90 minutes), otherwise they are denied access. See the merged log of a failed (re)login at https://pastebin.com/Jqc52MWH. For logs with debug_level = 9 see https://pastebin.com/0uQ8MCuS

/etc/sssd/sssd.conf

[sssd]
domains = EXAMPLE.COM
default_domain_suffix = EXAMPLE.COM
config_file_version = 2
debug_level = 7
services = nss, pam, ssh

[domain/EXAMPLE.COM]
ad_domain = EXAMPLE.COM
debug_level = 7
krb5_realm = EXAMPLE.COM
realmd_tags = manages-system joined-with-samba
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = ad
ldap_user_ssh_public_key = sshPublicKey
ad_access_filter = DOM:EXAMPLE.COM:(memberOf:1.2.840.113556.1.4.1941:=CN=ACL_DEV_APAC_developers,OU=ACL,OU=Group,OU=EXAMPLE,DC=EXAMPLE,DC=COM)

[ssh]
debug_level = 7

[nss]
debug_level = 7

Same config works on 1.13

Do you really need to use default_domain_suffix = EXAMPLE.COM ?

Because it might explain following line and reason why it is ssh key is not found.

(Wed Oct  4 09:18:45 2017) [sssd[ssh]] [cache_req_search_send] (0x0400): CR #4: Looking up myadmin@example.com@example.com

It's the only way I found to enable login to ssh with username@server.

Disabled default_domain_suffix, deleted cache and tried again:

2nd try failed again:
https://pastebin.com/32nhfupp

It's the only way I found to enable login to ssh with username@server.
Disabled default_domain_suffix, deleted cache and tried again:
2nd try failed again:
https://pastebin.com/32nhfupp

I can see that user was found few times:

(Wed Oct  4 11:26:24 2017) [sssd[ssh]] [cache_req_search_send] (0x0400): CR #9: Returning [myadmin@example.com] from cache
(Wed Oct  4 11:26:24 2017) [sssd[ssh]] [cache_req_search_ncache_filter] (0x0400): CR #9: This request type does not support filtering result by negative cache
(Wed Oct  4 11:26:24 2017) [sssd[ssh]] [cache_req_create_and_add_result] (0x0400): CR #9: Found 1 entries in domain EXAMPLE.COM
(Wed Oct  4 11:26:24 2017) [sssd[ssh]] [cache_req_done] (0x0400): CR #9: Finished: Success

Could you provide output of following command?
ldb_search -H /var/lib/sss/db/cache_$domain.ldb '(objectClass=user)'

I am interested whether ssh key is cached or no.

 
# ldbsearch -H /var/lib/sss/db/cache_EXAMPLE.COM.ldb '(objectClass=user)'
asq: Unable to register control with rootdse!
# record 1
dn: name=myadmin@example.com,cn=users,cn=EXAMPLE.COM,cn=sysdb
createTimestamp: 1507116287
fullName: myadmin
gecos: myadmin
gidNumber: 873800513
name: myadmin@example.com
objectClass: user
uidNumber: 873801607
objectSIDString: S-1-5-21-3737882482-2546886929-1415981180-1607
uniqueID: 807f8d7c-2ef1-4bbd-97c6-f8e6ba924564
originalDN: CN=myadmin,OU=Priviledged,OU=User,OU=EXAMPLE,DC=EXAMPLE,DC=COM
originalMemberOf: CN=G_Superadmin,OU=Role,OU=Group,OU=EXAMPLE,DC=EXAMPLE,DC=COM
originalModifyTimestamp: 20170810100427.0Z
entryUSN: 26708
userPrincipalName: myadmin@EXAMPLE.COM
adUserAccountControl: 66048
nameAlias: myadmin@example.com
isPosix: TRUE
lastUpdate: 1507116287
dataExpireTimestamp: 1507121687
initgrExpireTimestamp: 1507121721
ldap_access_filter_allow: TRUE
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1106@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1107@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1108@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1109@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1110@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1111@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1112@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1113@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1114@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1115@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1116@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1117@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1118@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1119@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1120@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1121@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1122@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1136@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1139@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=S-1-5-21-3737882482-2546886929-1415981180-1603@example.com,cn=gr
 oups,cn=EXAMPLE.COM,cn=sysdb
memberof: name=Domain Users@example.com,cn=groups,cn=EXAMPLE.COM,cn=sysdb
distinguishedName: name=myadmin@example.com,cn=users,cn=EXAMPLE.COM,cn=sy
 sdb

# returned 1 records
# 1 entries
# 0 referrals

I installed ldb-tools and used ldbsearch in case that matters.

OK ssh key is not cached.

Are you sure that use with DN CN=myadmin,OU=Priviledged,OU=User,OU=EXAMPLE,DC=EXAMPLE,DC=COM has attribute sshPublicKey in AD? because IIRC such attribute is not standard in AD. But you might add custom LDIF schema to AD.

If yes then it would be good to see also related domain log files and not only log files from ssh https://pagure.io/SSSD/sssd/issue/3534#comment-470311
Edited 6 years ago by lslebodn

Yes I'm sure. I'm able to log in if I clear the cache.

$ date ; ssh -l myadmin@EXAMPLE.COM 10.95.35.126 date; date
Wed Oct  4 13:54:37 CEST 2017
Wed Oct  4 11:54:40 UTC 2017
Wed Oct  4 13:54:41 CEST 2017

See https://pastebin.com/RV8iFwr2 for a log

Yes I'm sure. I'm able to log in if I clear the cache.
$ date ; ssh -l myadmin@EXAMPLE.COM 10.95.35.126 date; date
Wed Oct 4 13:54:37 CEST 2017
Wed Oct 4 11:54:40 UTC 2017
Wed Oct 4 13:54:41 CEST 2017

See https://pastebin.com/RV8iFwr2 for a log

I can see that key was downloaded for second time

(Wed Oct  4 11:54:39 2017) [sssd[be[EXAMPLE.COM]]] [sdap_attrs_add_ldap_attr] (0x2000): Adding sshPublicKey [c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUtQaTdqK3N6bkRLL3ZabFhPRUFkcjB5R2w0aFNKUFRTcTdtTDBHLzRmMFEgam9uYXRoYW52b2d0QEpvbmF0aGFucy1NQlAuZnJpdHouYm94] to attributes of [myadmin@example.com].

I assume that https://pagure.io/SSSD/sssd/issue/3534#comment-470311 did not refresh anything from AD because entry in cache was still valid. It is impossible to say without domain log files.
And it should be enough to invalidate entry with sss_cache -u myadmin

Anyway it seems that there is bug with ssh responder and default_domain_suffix in sssd-1.15.x

There seems to be no refresh at all. I only see dns traffic in tcpdump when connecting.

# tcpdump -n host 10.95.2.30 or host 10.95.2.14
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:10:44.792146 IP 10.95.35.126.59146 > 10.95.2.14.domain: 33046+ PTR? 2.255.168.192.in-addr.arpa. (44)
12:10:44.796481 IP 10.95.2.14.domain > 10.95.35.126.59146: 33046 1/0/0 PTR ip-192-168-255-2.ec2.internal. (87)
12:10:44.796596 IP 10.95.35.126.54907 > 10.95.2.14.domain: 30294+ A? ip-192-168-255-2.ec2.internal. (47)
12:10:44.799038 IP 10.95.2.14.domain > 10.95.35.126.54907: 30294 1/0/0 A 192.168.255.2 (63)
12:10:46.122755 IP 10.95.35.126.42476 > 10.95.2.14.domain: 42951+ A? ip-192-168-255-2.ec2.internal. (47)
12:10:46.122767 IP 10.95.35.126.42476 > 10.95.2.14.domain: 7238+ AAAA? ip-192-168-255-2.ec2.internal. (47)
12:10:46.123797 IP 10.95.2.14.domain > 10.95.35.126.42476: 42951 1/0/0 A 192.168.255.2 (63)
12:10:46.128146 IP 10.95.2.14.domain > 10.95.35.126.42476: 7238 0/1/0 (108)

Invalidating the cache for the user helps, yes. Also setting entry_cache_user_timeout = 5 enables a login every 5 seconds.

Anyway it seems that there is bug with ssh responder and default_domain_suffix in sssd-1.15.x

All posted logs since https://pagure.io/SSSD/sssd/issue/3534#comment-470294 where without default_domain_suffix!

Anyway it seems that there is bug with ssh responder and default_domain_suffix in sssd-1.15.x

All posted logs since https://pagure.io/SSSD/sssd/issue/3534#comment-470294 where without default_domain_suffix!

Sure but IIUC it works without default_domain_suffix + invalidating cache entries (sssd_cache -u user / sss_cache -U). So the only bug is with default_domain_suffix (initial log files)

User are able to log in only once during cache lifetime (by default 90 minutes), otherwise they are denied access. See the merged log of a failed (re)login at https://pastebin.com/Jqc52MWH. For logs with debug_level = 9 see https://pastebin.com/0uQ8MCuS

The problem is the same with and without default_domain_suffix:
I can log in once for a user and then have to wait for a cache refresh due to timeout or invalidation.

According to some of the pasted logs the user is looked up via the Global Catalog from time to time. Do you replicate the sshPublicKey attribute to the Global Catalog? If not setting 'ad_enable_gc = false' in the [domain/...] section of sssd.conf might help, please see man sssd-ad for details.

The attribute was not replicated to the global catalog. Enabling replication or setting ad_enable_gc = false enables multiple logins.

On (04/10/17 13:28), Jonathan Vogt wrote:

vorgangsnummer added a new comment to an issue you are following:
The attribute was not replicated to the global catalog. Enabling replication or setting `ad_enable_gc = false` enables multiple logins.

I'm glad workaround works for you now. And thank you for bug report

LS

The global catalog entry should be solved in a systematic way with https://pagure.io/SSSD/sssd/issue/3538

btw I'm not even able to reproduce the issue with the double qualification of the username and looking up user's public keys works fine for me even with default_domain_suffix set to the only domain I have in the config file. Even the debug message has the name qualified only once:

(Thu Oct  5 14:41:42 2017) [sssd[ssh]] [cache_req_search_send] (0x0400): CR #3: Looking up puser@win.trust.test

Therefore I suggest closing this ticket.

btw I'm not even able to reproduce the issue with the double qualificatio=
n of the username and looking up user's public keys works fine for me even =
with default_domain_suffix set to the only domain I have in the config file=
. Even the debug message has the name qualified only once:
That=E2=80=99s what I tried to bring across. default_domain_suffix set or=
not does not change the behaviour.
(Thu Oct 5 14:41:42 2017) [sssd[ssh]] [cache_req_search_send] (0x0400): = CR #3: Looking up puser@win.trust.test
=20
Therefore I suggest closing this ticket.
https://pagure.io/SSSD/sssd/issue/3538 https://pagure.io/SSSD/sssd/issue/3= 538 describes the actually issue way better I guess, so fell free to clos=
e.
``
=20
To reply, visit the link below or just reply to this email
https://pagure.io/SSSD/sssd/issue/3534

--=20
https://goo.gl/cGQj8c
This message and the information contained herein is proprietary and=20
confidential and subject to the AllCloud policy statement, you may review=
=20
it here https://goo.gl/QejddU.

OK, closing.

Metadata Update from @jhrozek:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)
6 years ago

SSSD is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in SSSD's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/SSSD/sssd/issues/4560

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Login to comment on this ticket.

Metadata
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.