#3189 STARTTLS connections to OpenLDAP fail with 1.14.1
Closed: Fixed None Opened 3 years ago by lollipopman.

Connections to OpenLDAP 2.4.31 with STARTTLS fails with 1.14.1, however the same configuration works correctly with 1.13.4. I have git bisected to find the breaking commmit, 75e66c3.

After reverting this commit off of 1.14.1 STARTTLS connections work again successfully.

This was tested on Debian 7.0(Wheezy)

Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 fd=13 ACCEPT from IP=192.168.1.33:38139 (IP=0.0.0.0:389)
Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 op=0 EXT oid=1.3.6.1.4.1.1466.20037
Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 op=0 STARTTLS
Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 op=0 RESULT oid= err=0 text=
Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 fd=13 closed (TLS negotiation failure)


Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 fd=13 ACCEPT from IP=192.168.1.33:38139 (IP=0.0.0.0:389)
Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 op=0 EXT oid=1.3.6.1.4.1.1466.20037
Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 op=0 STARTTLS
Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 op=0 RESULT oid= err=0 text=
Sep 15 02:31:43 biz.bar slapd[13061]: conn=35716 fd=13 closed (TLS negotiation failure)

Thank you for the bug report and the work in bisecting the failed patch. We checked the patch with other SSSD developers and can't see a reason why this patch would break TLS and our own tests that use TLS are all passing with both OpenLDAP and 389DS, so I'd like to ask you for more information.

Could you please attach SSSD debug logs (debug_level=10 in the [domain] section please) to this ticket?

Hi, any luck getting those logs?

Sumit suggested the issue might be in us not removing the O_NONBLOCK flag with the new code which might cause some calls in libldap to just return without doing anything actually.

apologies for the delay, I will try to get some debug output today

Logs attached, the behavior seems slightly different from what I first observed when I bisected, as I don't see the STARTTLS failures, perhaps, because we are no longer using a simple bind in sssd. However the failure still occurs, and sssd does not work.

The pristine log actually looks like if sssd was crashing..could you please check for sssd_be crashes in the syslog?

The failure in the "revert" log doesn't tell me much except that sssd is thinking it cannot contact the LDAP server. Here I would like to ask for strace logs, as described in https://fedorahosted.org/sssd/wiki/DevelTips#UsingstracetotracktheSSSDprocesses

Hi, sorry to nag you, but did you have a chance to take a look at the syslog if sssd indeed crashes for you?

I have added two new straces following the instructions you provided. However I was not able to turn the debug level up to 10 on sssd_be. When setting the debug level to 10, I was no longer able to reproduce the problem. It appears that setting the debug level to 10 somehow changes the behavior of the program and masks the bug.

Please let me know what other information I can provide. Thanks, Jesse

Thank you, then I guess my hunch about sssd crashing was not correct?

In the meantime, there was also a bug open in the Debian tracker, that describes a similar, if not the same issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=840617

Is your openldap server also running on Debian? I'm asking because on RH/Fedora, except the latest development distribution, openldap was linked with NSS for crypto, which might make a difference here..

We are running openldap on Debian as well, below is the version information, and the dynamically linked libraries, including gnutls.

Thanks, Jesse

> /usr/sbin/slapd -VVV
@(#) $OpenLDAP: slapd  (May  2 2016 20:31:10) $
        pbuilder@chimera:/build/openldap-2.4.31/debian/build/servers/slapd

Included static backends:
    config
    ldif

> ldd /usr/sbin/slapd
        linux-vdso.so.1 =>  (0x00007fffee5ff000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f532e2c4000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f532e0b5000)
        libdb-5.1.so => /usr/lib/x86_64-linux-gnu/libdb-5.1.so (0x00007f532dd31000)
        libodbc.so.1 => /usr/lib/x86_64-linux-gnu/libodbc.so.1 (0x00007f532dac9000)
        libslp.so.1 => /usr/lib/libslp.so.1 (0x00007f532d8b7000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f532d69b000)
        libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f532d3db000)
        libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f532d15c000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f532cf24000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f532cd0e000)
        libslapi-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libslapi-2.4.so.2 (0x00007f532caee000)
        libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f532c8e4000)
        libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007f532c6d9000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f532c4bd000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f532c12f000)
        libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f532bf17000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f532bd12000)
        libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f532bb01000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f532b8ea000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f532b6d8000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f532b4d4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f532e8a4000)

Replying to [comment:11 lollipopman]:

We are running openldap on Debian as well, below is the version information, and the dynamically linked libraries, including gnutls.

Thanks, Jesse

{{{

/usr/sbin/slapd -VVV
@(#) $OpenLDAP: slapd (May 2 2016 20:31:10) $
pbuilder@chimera:/build/openldap-2.4.31/debian/build/servers/slapd
}}}
Server seems to be on debian-old-stable(wheezy)
IMHO, for this case it is important what is a version of libldap on client side? Do you you debian testing(stretch)?

BTW Could you test with different Directory server?
freeIPA, Active Directory or it is reproducible just with opendldap server.

Fields changed

cc: => lslebodn

Fwiw, I think I encountered the same bug, also on Debian. Both my SSSD server and the OpenLDAP server run Debian sid.

Downgrading sssd as described above helped. Unfortunately I don't currently have much time to pursue this problem, but I can provide version numbers of specific packages if needed.

The problem occurs both with STARTTLS and when using an ldaps:// URL.

Replying to [comment:12 lslebodn]:

Replying to [comment:11 lollipopman]:

We are running openldap on Debian as well, below is the version information, and the dynamically linked libraries, including gnutls.

Thanks, Jesse

{{{

/usr/sbin/slapd -VVV
@(#) $OpenLDAP: slapd (May 2 2016 20:31:10) $
pbuilder@chimera:/build/openldap-2.4.31/debian/build/servers/slapd
}}}
Server seems to be on debian-old-stable(wheezy)
IMHO, for this case it is important what is a version of libldap on client side? Do you you debian testing(stretch)?

A user named "Freg" showed up Today on #sssd with the very same issue. Freg has been using Debian Stretch and the version of libldap on the client side is 2.4.42+dfsg-2+b3. Freg also mentioned that the OpenLDAP server (this one on Debian Jessie) is 2.4.40+dfsg-1+deb8u2.

I spent some time with troubleshooting the issue.

And the problem is with libldap is compiled with gnutls on debian. It works well with nss on fedora/el and I also tried to compile libldap + openssl on debian and it works well.

lollipopman correctly identified problematic commit 75e66c3.

The main change in the commit is that O_NONBLOCK is not set just around syscall connect in request sssd_async_connect_send -> sssd_async_connect_send.
But it is permanently set in function sssd_async_socket_init_send. And it looks like gnutls does not like it.

The problem is not just with authentication + start_tls.
But there is the same problem with ldaps:// in ldap_uri

Fields changed

owner: somebody => lslebodn
status: new => assigned

Fields changed

patch: 0 => 1

Fields changed

milestone: NEEDS_TRIAGE => SSSD 1.14.3

no clone to RHBZ as this bug only affects client libraries linked with gnuTLS

rhbz: => 0

master:

sssd-1-14:

resolution: => fixed
status: assigned => closed

Metadata Update from @lollipopman:
- Issue assigned to lslebodn
- Issue set to the milestone: SSSD 1.14.3

3 years ago

SSSD is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in SSSD's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/SSSD/sssd/issues/4222

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Login to comment on this ticket.

Metadata