#4095 ldap backend goes offline when user with subdomain connects
Opened 4 months ago by sternber. Modified 4 months ago

Environment:
sssd-1.16.2-13.el7_6.8.x86_64
CentOS Linux release 7.6.1810
x86_64

opendj-6.5.0-1 ldap server

Bug:
We don't run any subdomains. So all user login attempts with subdomain
come from brute force attacks.
When a user with an subdomain tries to connect like 'smith1234@foo' .
It results in a crash of the ldap backend the following users can't login for the next minutes.

Steps to reproduce

ssh -l smith1234@foo max-disp004

Problem seems to be:

[sssd[be[LDAP]]] [sdap_get_generic_op_finished] (0x0040): Unexpected result from ldap: Insufficient access(50), You do not have sufficient privileges to perform an unindexed search
[sssd[be[LDAP]]] [generic_ext_search_handler] (0x0040): sdap_get_generic_ext_recv failed [5]: Input/output error

Workaround is in sssd.conf re_expression = (?P<name>[^@]+$)

Expected:
The backend shouldn't go offline when the the ldap query don't work. It would be helpful
to know the ldap query, which failed

Logs:
/var/log/secure

Oct 10 13:32:00 max-disp004 sshd[342457]: Invalid user smith1234@foo from 42.169.42.32 port 51672
Oct 10 13:32:00 max-disp004 sshd[342457]: input_userauth_request: invalid user smith1234@foo [preauth]
Oct 10 13:32:03 max-display004 sshd[342457]: pam_unix(sshd:auth): check pass; user unknown
Oct 10 13:32:03 max-display004 sshd[342457]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=pzx3446.foo.de
Oct 10 13:32:05 max-disp004 sshd[342457]: Failed password for invalid user smith1234@foo from 42.169.42.32 port 51672 ssh2
Oct 10 13:32:46 max-disp004 sshd[342572]: Invalid user foomaster from 42.169.42.98 port 38388

/var/log/sssd/sssd_LDAP.log

(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Subdomains #3003]: Finished. Target is not supported with this configuration.
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_get_generic_op_finished] (0x0040): Unexpected result from ldap: Insufficient access(50), You do not have sufficient privileges to perform an unindexed search
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [generic_ext_search_handler] (0x0040): sdap_get_generic_ext_recv failed [5]: Input/output error
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_get_users_done] (0x0040): Failed to retrieve users [5][Input/output error].
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [resolv_gethostbyname_files_send] (0x0100): Trying to resolve A record of 'ldap03.foo.de' in files
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap03.foo.de' as 'resolving name'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap03.foo.de' as 'name resolved'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_cli_auth_step] (0x0100): expire timeout is 900
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [fo_set_port_status] (0x0100): Marking port 1389 of server 'ldap03.foo.de' as 'working'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap03.foo.de' as 'working'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_get_generic_op_finished] (0x0040): Unexpected result from ldap: Insufficient access(50), You do not have sufficient privileges to perform an unindexed search
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [generic_ext_search_handler] (0x0040): sdap_get_generic_ext_recv failed [5]: Input/output error
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_get_users_done] (0x0040): Failed to retrieve users [5][Input/output error].
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [resolv_gethostbyname_files_send] (0x0100): Trying to resolve A record of 'ldap04.foo.de' in files
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap04.foo.de' as 'resolving name'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap04.foo.de' as 'name resolved'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_cli_auth_step] (0x0100): expire timeout is 900
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [fo_set_port_status] (0x0100): Marking port 1389 of server 'ldap04.foo.de' as 'working'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap04.foo.de' as 'working'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_get_generic_op_finished] (0x0040): Unexpected result from ldap: Insufficient access(50), You do not have sufficient privileges to perform an unindexed search
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [generic_ext_search_handler] (0x0040): sdap_get_generic_ext_recv failed [5]: Input/output error
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_get_users_done] (0x0040): Failed to retrieve users [5][Input/output error].
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [get_server_status] (0x0100): Hostname resolution expired, resetting the server status of 'ldap.foo.de'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap.foo.de' as 'name not resolved'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [get_port_status] (0x0080): SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [get_port_status] (0x0100): Resetting the status of port 1389 for server 'ldap.foo.de'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [resolv_gethostbyname_files_send] (0x0100): Trying to resolve A record of 'ldap.foo.de' in files
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap.foo.de' as 'resolving name'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [resolv_gethostbyname_files_send] (0x0100): Trying to resolve AAAA record of 'it-ldap-slave.desy.de' in files
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [resolv_gethostbyname_dns_query] (0x0100): Trying to resolve A record of 'ldap.foo.de' in DNS
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap.foo.de' as 'name resolved'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_cli_auth_step] (0x0100): expire timeout is 900
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [fo_set_port_status] (0x0100): Marking port 1389 of server 'ldap.foo.de' as 'working'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [set_server_common_status] (0x0100): Marking server 'ldap.foo.de' as 'working'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_get_generic_op_finished] (0x0040): Unexpected result from ldap: Insufficient access(50), You do not have sufficient privileges to perform an unindexed search
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [generic_ext_search_handler] (0x0040): sdap_get_generic_ext_recv failed [5]: Input/output error
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_get_users_done] (0x0040): Failed to retrieve users [5][Input/output error].
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [get_port_status] (0x0080): SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [get_port_status] (0x0080): SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [get_port_status] (0x0080): SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [fo_resolve_service_send] (0x0020): No available servers for service 'LDAP'
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [sdap_id_op_connect_done] (0x0020): Failed to connect, going offline (5 [Input/output error])
(Thu Oct 10 13:32:00 2019) [sssd[be[LDAP]]] [be_run_offline_cb] (0x0080): Going offline. Running callbacks.
(Thu Oct 10 13:32:03 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3005]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:03 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3006]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:03 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3007]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:10 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3008]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:10 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3009]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:10 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3010]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:31 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3011]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:46 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3012]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:49 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3013]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:49 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3014]: Finished. Backend is currently offline.
(Thu Oct 10 13:32:49 2019) [sssd[be[LDAP]]] [dp_req_reply_gen_error] (0x0080): DP Request [Account #3015]: Finished. Backend is currently offline.

sssd.conf

[sssd]
domains = LDAP
services = nss, pam, sudo
config_file_version = 2

[nss]
filter_users = root
filter_groups = root
override_homedir = /home/%u


[pam]
debug_level = 4
pam_pwd_expiration_warning = 10

[domain/LDAP]
debug_level = 4
chpass_provider = none
ldap_schema = rfc2307bis

sudo_provider = none

id_provider = ldap
ldap_uri = ldap://ldap.foo.de:1389,ldap://ldap03.foo.de:1389,ldap://ldap04.foo.de:1389
ldap_search_base = ou=RGY,o=DESY,c=DE
ldap_group_member = uniqueMember

auth_provider = krb5
krb5_server = kerb2.foo.de:88,kerb3.foo.de:88,kerb1.foo.de:88
krb5_realm = FOO.DE

cache_credentials = false
enumerate = false

Hi,

if the name contains an @ character and the part after the @ character does not match to an existing domain SSSD assumes that the input might be an email address or Kerberos principal and tries to find a user with this email or principal. If you increase the debug_level in the [domain/...] section you should see the related search filter in the logs.

I agree that going offline is not the best here. But I have to think a bit about the best solution because in general a Insufficient access(50) error might also indicate the current LDAP server has an unexpected or broken configuration and switching to a different LDAP server (with a proper configuration) might help.

As long as your names do not contain @ your workaroud re_expression = (?P<name>[^@]+$) makes sense.

bye,
Sumit

Hello!

ok I found the culprit

calling ldap_search_ext with [(&(|(krbPrincipalName=bls@max)(mail=bls@max)(krbPrincipalName=bls\\@max@FOO.DE))(objectclass=posixAccount)(uid=*)(&(uidNumber=*)(!(uidNumber=0))))][ou=RGY,o=FOO,c=DE]

Our account schema don't have a krbPrincipalName attribut. So it can not be indexed,
and ldapsearch as anonymous user is not allowed to perform an unindexed searches.

Not sure if OpenDJ should allow the search on unknown attributes, or if sssd should
not simply assume that every ldapserver has account schemas with krbPrincipalName attributs?

I also tried to set ldap_user_principal = uid then the ldap backend stays online, but I can't
login anymore :-/

Hi,

I think it is not unusual that LDAP servers do not handle unknown attributes well, 389ds show a similar behavior. As a side note, what I do not understand is why search filter components with unknown attributes and equal-operator cannot just be treated as not-matched, because since the attribute does not exists it cannot match and you do not have to search for it.

The search for mail and principal is just a fallback if the user could not be found directly, so I would say using it to cover as much use-cases as possible with the default installation is justified.

You have to set ldap_user_principal to an attribute name which exists in the schema but is not used in the user objects. This attribute is not only used for searches but the value, if exists, is taken as the user principal for Kerberos authentication and for this must be a proper Kerberos principal. Even is the uid attribute contains the name part of the principal (the part before the @ character) the realm part is expected as well and is not automatically added is missing.

bye,
Sumit

Hello!

In my opinion the behaviour from OpenDJ/389ds is correct. If you don't have the
right to search the attribute, it shouldn't give you an answer.

As a feature request it would be nice to configure that the ldap_user_principal shouldn't
be used at least for the login. In our case for example I see 30 break in attempts in the
last 2h (and we have already fail2ban)
The would result in that login node will not be accessible.

bye and thanks for your help

Sven

We could do something like this: when ldap_user_principal is unset we'd skip using it. The problem is that we currently have no way to unset defaults via config file because setting an option to empty value will simply ignore it and will reuse the default.

sssd-empty-ldap_user_principal.patch

Login to comment on this ticket.

Metadata
Attachments 1