389-ds-base

Clone
Source Code
GIT

#49436 [abrt] 389-ds-base: SLL_Next(): ns-slapd killed by SIGSEGV
Closed: wontfix 8 years ago Opened 8 years ago by firstyear.

Closed: wontfix
Reopen Issue

Ticket was cloned from Red Hat Bugzilla (product Red Hat Enterprise Linux 7): Bug 1505046

=================================================================
==30050== ERROR: AddressSanitizer: attempting double-free on 0x60060097f720:
    #0 0x7fea5d26edd9 in __interceptor_free _asan_rtl_
    #1 0x7fea5cbdc718 in slapi_ch_free /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/ch_malloc.c:270
addr2line: Dwarf Error: Offset (104626) greater than or equal to .debug_str size (79829).
addr2line: Dwarf Error: Offset (104669) greater than or equal to .debug_str size (79829).
    #2 0x7fea5cd1fa94 in vattr_map_entry_free /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/vattr.c:1867
    #3 0x7fea5cd1fade in vattr_he_cleanup_fn /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/vattr.c:1875
    #4 0x7fea5ad8cb6f in PL_HashTableEnumerateEntries /usr/src/debug/nspr-4.17.0/lib/ds/../../nspr/lib/ds/plhash.c:374
0x60060097f720 is located 0 bytes inside of 24-byte region [0x60060097f720,0x60060097f738)
freed by thread T0 here:
    #0 0x7fea5d26edd9 in __interceptor_free _asan_rtl_
    #1 0x7fea5cbdc718 in slapi_ch_free /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/ch_malloc.c:270
    #2 0x7fea5cd1fa94 in vattr_map_entry_free /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/vattr.c:1867
    #3 0x7fea5cd1fade in vattr_he_cleanup_fn /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/vattr.c:1875
    #4 0x7fea5ad8cb6f in PL_HashTableEnumerateEntries /usr/src/debug/nspr-4.17.0/lib/ds/../../nspr/lib/ds/plhash.c:374
previously allocated by thread T0 here:
    #0 0x7fea5d26eff5 in calloc _asan_rtl_
    #1 0x7fea5cbdc2d8 in slapi_ch_calloc /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/ch_malloc.c:180
    #2 0x7fea5cd1f54d in slapi_vattrspi_register_internal /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/vattr.c:1585
    #3 0x7fea516766d8 in cos_cache_init /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/plugins/cos/cos_cache.c:326
    #4 0x7fea51670387 in cos_start /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/plugins/cos/cos.c:200
    #5 0x7fea5cc99ac3 in plugin_call_func /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/plugin.c:2028
    #6 0x7fea5cc9a571 in plugin_call_one /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/plugin.c:1978
    #7 0x55b0029eb79d in ?? /usr/src/debug/389-ds-base-1.4.0.1.20171030git2086d052e/ldap/servers/slapd/main.c:1114
    #8 0x7fea59ae737a in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
==30050== ABORTING

This crash makes very little sense. So vattr_cleanup calls vattr_map_destroy which in turn triggers PL_HashTableEnumerateEntries and vattr_he_cleanup_fn.

IE:

./ldap/servers/slapd/vattr.c:83:static void vattr_map_destroy(void);
./ldap/servers/slapd/vattr.c:124: vattr_map_destroy();
./ldap/servers/slapd/vattr.c:1881:vattr_map_destroy(void)

83 is the declaration, 124 is the call in vattr_cleanup and 1881 is the impl.

vattr_cleanup is a similar story:

ldap/servers/slapd/proto-slap.h:1408:void vattr_cleanup(void);
ldap/servers/slapd/vattr.c:121:vattr_cleanup()
ldap/servers/slapd/main.c:1181: vattr_cleanup();

proto-slap is the declaration, 121 the impl, and mail.c 1181 is the only call to it.

The only flaw I can really see here right now is that during the cleanup we do not take the vattr map lock, but at this point, the server should be only running a single thread of execution anyway.

So what this leads me to believe is that we have a situation where a virtual attribute is added TWICE into the cache, and this leads to the double free. Because we free it once early in the enumeration, and then again later during the same loop.

So the bug is likely in slapi_vattrspi_register_internal or cos_cache_init (or close by).

This means the best solution could be to make the insertion of values to the vattr cache stricter to find when we are double-adding values.

Metadata Update from @firstyear:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1505046
8 years ago

Metadata Update from @firstyear:
- Custom field component adjusted to None
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None
- Custom field type adjusted to None
- Custom field version adjusted to None
- Issue set to the milestone: 1.3.7.0 (was: 0.0 NEEDS_TRIAGE)
8 years ago

Metadata Update from @firstyear:
- Issue assigned to firstyear
8 years ago

Okay, I now understand this issue thanks to @vashirov

Here are two important back traces:

Breakpoint 1, slapi_vattrspi_regattr (h=0x56505d392b00, 
    type_name_to_register=0x56505a9a0b00 "multiLineDescription override ", DN=DN@entry=0x0, 
    hint=hint@entry=0x0) at ldap/servers/slapd/vattr.c:1608
1608    {
(gdb) bt
#0  0x00007f9e62e16f80 in slapi_vattrspi_regattr (h=0x56505d392b00, type_name_to_register=0x56505a9a0b00 "multiLineDescription override ", DN=DN@entry=0x0, hint=hint@entry=0x0)
    at ldap/servers/slapd/vattr.c:1608
#1  0x00007f9e577c416e in cos_dn_defs_cb (e=0x56505aade000, callback_data=0x7f9e26493c60)
    at ldap/servers/plugins/cos/cos_cache.c:863
#2  0x00007f9e62defd04 in send_ldap_search_entry_ext (pb=pb@entry=0x56505a9592c0, e=0x56505aade000, ectrls=ectrls@entry=0x0, attrs=0x0, attrsonly=0, send_result=send_result@entry=0, nentries=nentries@entry=0, urls=urls@entry=0x0) at ldap/servers/slapd/result.c:1495
#3  0x00007f9e62df04fc in send_ldap_search_entry (pb=pb@entry=0x56505a9592c0, e=<optimized out>, ectrls=ectrls@entry=0x0, attrs=<optimized out>, attrsonly=<optimized out>) at ldap/servers/slapd/result.c:1038
#4  0x00007f9e62dcbac1 in iterate (send_result=1, be=<optimized out>, pr_statp=0x7f9e2648f594, pagesize=<optimized out>, pnentries=0x7f9e2648f628, pb=0x56505a9592c0) at ldap/servers/slapd/opshared.c:1380
#5  0x00007f9e62dcbac1 in send_results_ext (pb=pb@entry=0x56505a9592c0, nentries=nentries@entry=0x7f9e2648f628, pagesize=-1, pr_stat=pr_stat@entry=0x7f9e2648f594, send_result=1)
    at ldap/servers/slapd/opshared.c:1611
#6  0x00007f9e62dcd601 in op_shared_search (pb=pb@entry=0x56505a9592c0, send_result=send_result@entry=1)
    at ldap/servers/slapd/opshared.c:811
#7  0x00007f9e62ddebc7 in search_internal_callback_pb (pb=pb@entry=0x56505a9592c0, callback_data=callback_data@entry=0x7f9e26493c60, prc=prc@entry=0x0, psec=psec@entry=0x7f9e577c3b60 <cos_dn_defs_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:726
#8  0x00007f9e62ddf0e9 in slapi_search_internal_callback_pb (pb=pb@entry=0x56505a9592c0, callback_data=callback_data@entry=0x7f9e26493c60, prc=prc@entry=0x0, psec=psec@entry=0x7f9e577c3b60 <cos_dn_defs_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:518
#9  0x00007f9e577c5d06 in cos_cache_add_dn_defs (pDefs=0x56505d3627e0, dn=0x56505a918c90 "o=cos.com")
---Type <return> to continue, or q <return> to quit---
    at ldap/servers/plugins/cos/cos_cache.c:1055
#10 0x00007f9e577c5d06 in cos_cache_build_definition_list (vattr_cacheable=0x56505d362808, pDefs=0x56505d3627e0) at ldap/servers/plugins/cos/cos_cache.c:670
#11 0x00007f9e577c5d06 in cos_cache_create_unlock () at ldap/servers/plugins/cos/cos_cache.c:462
#12 0x00007f9e577c5d06 in cos_cache_creation_lock () at ldap/servers/plugins/cos/cos_cache.c:590
#13 0x00007f9e577c60fd in cos_cache_wait_on_change (arg=<optimized out>)
    at ldap/servers/plugins/cos/cos_cache.c:423
#14 0x00007f9e60b9fc8b in _pt_root (arg=0x56505d372000) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#15 0x00007f9e6053fdd5 in start_thread (arg=0x7f9e26494700) at pthread_create.c:308
#16 0x00007f9e5fbed9ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113



Breakpoint 1, slapi_vattrspi_regattr (h=0x56505d392b00, 
    type_name_to_register=0x56505d393440 "multiLineDescription", DN=DN@entry=0x0, hint=hint@entry=0x0)
    at ldap/servers/slapd/vattr.c:1608
1608    {
(gdb) bt
#0  0x00007f9e62e16f80 in slapi_vattrspi_regattr (h=0x56505d392b00, type_name_to_register=0x56505d393440 "multiLineDescription", DN=DN@entry=0x0, hint=hint@entry=0x0) at ldap/servers/slapd/vattr.c:1608
#1  0x00007f9e577c416e in cos_dn_defs_cb (e=0x56505aa65d40, callback_data=0x7f9e26493c60)
    at ldap/servers/plugins/cos/cos_cache.c:863
#2  0x00007f9e62defd04 in send_ldap_search_entry_ext (pb=pb@entry=0x56505a9592c0, e=0x56505aa65d40, ectrls=ectrls@entry=0x0, attrs=0x0, attrsonly=0, send_result=send_result@entry=0, nentries=nentries@entry=0, urls=urls@entry=0x0) at ldap/servers/slapd/result.c:1495
#3  0x00007f9e62df04fc in send_ldap_search_entry (pb=pb@entry=0x56505a9592c0, e=<optimized out>, ectrls=ectrls@entry=0x0, attrs=<optimized out>, attrsonly=<optimized out>) at ldap/servers/slapd/result.c:1038
#4  0x00007f9e62dcbac1 in iterate (send_result=1, be=<optimized out>, pr_statp=0x7f9e2648f594, pagesize=<optimized out>, pnentries=0x7f9e2648f628, pb=0x56505a9592c0) at ldap/servers/slapd/opshared.c:1380
#5  0x00007f9e62dcbac1 in send_results_ext (pb=pb@entry=0x56505a9592c0, nentries=nentries@entry=0x7f9e2648f628, pagesize=-1, pr_stat=pr_stat@entry=0x7f9e2648f594, send_result=1)
    at ldap/servers/slapd/opshared.c:1611
#6  0x00007f9e62dcd601 in op_shared_search (pb=pb@entry=0x56505a9592c0, send_result=send_result@entry=1)
    at ldap/servers/slapd/opshared.c:811
#7  0x00007f9e62ddebc7 in search_internal_callback_pb (pb=pb@entry=0x56505a9592c0, callback_data=callback_data@entry=0x7f9e26493c60, prc=prc@entry=0x0, psec=psec@entry=0x7f9e577c3b60 <cos_dn_defs_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:726
#8  0x00007f9e62ddf0e9 in slapi_search_internal_callback_pb (pb=pb@entry=0x56505a9592c0, callback_data=callback_data@entry=0x7f9e26493c60, prc=prc@entry=0x0, psec=psec@entry=0x7f9e577c3b60 <cos_dn_defs_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:518
#9  0x00007f9e577c5d06 in cos_cache_add_dn_defs (pDefs=0x56505d3627e0, dn=0x56505a9191e0 "o=cos1.com")
    at ldap/servers/plugins/cos/cos_cache.c:1055
---Type <return> to continue, or q <return> to quit---
#10 0x00007f9e577c5d06 in cos_cache_build_definition_list (vattr_cacheable=0x56505d362808, pDefs=0x56505d3627e0) at ldap/servers/plugins/cos/cos_cache.c:670
#11 0x00007f9e577c5d06 in cos_cache_create_unlock () at ldap/servers/plugins/cos/cos_cache.c:462
#12 0x00007f9e577c5d06 in cos_cache_creation_lock () at ldap/servers/plugins/cos/cos_cache.c:590
#13 0x00007f9e577c60fd in cos_cache_wait_on_change (arg=<optimized out>)
    at ldap/servers/plugins/cos/cos_cache.c:423
#14 0x00007f9e60b9fc8b in _pt_root (arg=0x56505d372000) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#15 0x00007f9e6053fdd5 in start_thread (arg=0x7f9e26494700) at pthread_create.c:308
#16 0x00007f9e5fbed9ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

What's important here is that in cos_dn_defs_cb, we register two seperate attributes to the virtual attribute system, but we use the same vattr_sp_handle (h=0x56505d392b00).

This is because in cos, with regattr, we use the handle which is a global. Note that we only use this handle at plugin START and to register these attrs, we never free or use it again otherwise.

In the vattr system however, we uniquely identify vattr_maps by the attr name not by their sp_handles. So when we register like this, we register the same sp_handle TWICE into the vattr_map code.

Finally, this leads to our issue - as we shutdown, we free the first vattr_map and it's handle, then we free the second - leading to the doubly free.

Due to the way the handles here work, it would be best to create a handle per attribute we register instead of per plugin to avoid this. Alternately, we need to refcount on the handles at insertion time.

@vashirov Would appreciate you running this through the COS test suites to make sure this works.

@firstyear TET CoS tests completed without any crash.

Metadata Update from @mreynolds:
- Custom field reviewstatus adjusted to ack (was: None)
8 years ago

commit ee4428a
To ssh://git@pagure.io/389-ds-base.git
03df576..ee4428a master -> master

commit 8ca21dd
To ssh://git@pagure.io/389-ds-base.git
2ffdac2..8ca21dd 389-ds-base-1.3.7 -> 389-ds-base-1.3.7

Metadata Update from @firstyear:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
8 years ago

828b46b..c81a5af 389-ds-base-1.3.6 -> 389-ds-base-1.3.6

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/2495

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: fixed)
5 years ago

Log in to comment on this ticket.

Metadata
None
None
None
None
ack
None
Attachments 1
0001-Ticket-49436-double-free-in-COS-in-some-condi...
Attached 8 years ago View Comment
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.