#48987 Heap use after free in dblayer_close_indexes
Closed: Fixed None Opened 3 years ago by firstyear.

=================================================================
==13107==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100047a598 at pc 0x7f555aa14e48 bp 0x7f554a9027c0 sp 0x7f554a9027b0
WRITE of size 8 at 0x61100047a598 thread T24
llvm-symbolizer: for the -functions option: Cannot find option named 'true'!
    #0 0x7f555aa14e47 in dblayer_close_indexes /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/dblayer.c:2518
    #1 0x7f555aa15343 in dblayer_instance_close /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/dblayer.c:2562
    #2 0x7f555aabe3f8 in ldbm_instance_delete_instance_entry_callback /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/ldbm_instance_config.c:1239
    #3 0x7f556a2cc072 in dse_call_callback /home/william/development/389ds/ds/ldap/servers/slapd/dse.c:2634
    #4 0x7f556a2cb798 in dse_delete /home/william/development/389ds/ds/ldap/servers/slapd/dse.c:2496
    #5 0x7f556a2b11db in op_shared_delete /home/william/development/389ds/ds/ldap/servers/slapd/delete.c:333
    #6 0x7f556a2b02ab in do_delete /home/william/development/389ds/ds/ldap/servers/slapd/delete.c:97
    #7 0x41f74d in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:617
    #8 0x4252d2 in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1759
    #9 0x7f55677857de in _pt_root /usr/src/debug/nspr-4.12.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216
    #10 0x7f55675456b9 in start_thread /usr/src/debug/glibc-2.24-91-g7e625f7/nptl/pthread_create.c:333
    #11 0x7f55672803ce in __GI___clone /usr/src/debug////////glibc-2.24-91-g7e625f7/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:105

0x61100047a598 is located 24 bytes inside of 208-byte region [0x61100047a580,0x61100047a650)
freed by thread T20 here:
    #0 0x7f556a804ba0 in __interceptor_free _asan_rtl_
    #1 0x7f556a2a4e72 in slapi_ch_free /home/william/development/389ds/ds/ldap/servers/slapd/ch_malloc.c:292
    #2 0x7f555aa8640e in attrinfo_delete /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/ldbm_attr.c:62
    #3 0x7f555ab0a620 in vlvIndex_delete /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/vlv_srch.c:547
    #4 0x7f555ab0902e in vlvSearch_delete /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/vlv_srch.c:224
    #5 0x7f555aafbf31 in vlv_DeleteSearchEntry /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/vlv.c:123
    #6 0x7f556a2cc072 in dse_call_callback /home/william/development/389ds/ds/ldap/servers/slapd/dse.c:2634
    #7 0x7f556a2cb798 in dse_delete /home/william/development/389ds/ds/ldap/servers/slapd/dse.c:2496
    #8 0x7f556a2b11db in op_shared_delete /home/william/development/389ds/ds/ldap/servers/slapd/delete.c:333
    #9 0x7f556a2b02ab in do_delete /home/william/development/389ds/ds/ldap/servers/slapd/delete.c:97
    #10 0x41f74d in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:617
    #11 0x4252d2 in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1759
    #12 0x7f55677857de in _pt_root /usr/src/debug/nspr-4.12.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216

previously allocated by thread T33 here:
    #0 0x7f556a8050c0 in calloc _asan_rtl_
    #1 0x7f556a2a49ca in slapi_ch_calloc /home/william/development/389ds/ds/ldap/servers/slapd/ch_malloc.c:188
    #2 0x7f555aa861e9 in attrinfo_new /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/ldbm_attr.c:37
    #3 0x7f555ab09ff7 in vlvIndex_new /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/vlv_srch.c:509
    #4 0x7f555aafca38 in vlv_init_index_entry /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/vlv.c:243
    #5 0x7f556a2cc072 in dse_call_callback /home/william/development/389ds/ds/ldap/servers/slapd/dse.c:2634
    #6 0x7f556a2c883a in do_dse_search /home/william/development/389ds/ds/ldap/servers/slapd/dse.c:1643
    #7 0x7f556a2c8ef9 in dse_search /home/william/development/389ds/ds/ldap/servers/slapd/dse.c:1757
    #8 0x7f556a35be36 in op_shared_search /home/william/development/389ds/ds/ldap/servers/slapd/opshared.c:807
    #9 0x7f556a3983f3 in search_internal_callback_pb /home/william/development/389ds/ds/ldap/servers/slapd/plugin_internal_op.c:783
    #10 0x7f556a3979b6 in search_internal_pb /home/william/development/389ds/ds/ldap/servers/slapd/plugin_internal_op.c:636
    #11 0x7f556a397229 in slapi_search_internal /home/william/development/389ds/ds/ldap/servers/slapd/plugin_internal_op.c:480
    #12 0x7f555aafd7d2 in vlv_init /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/vlv.c:397
    #13 0x7f555aa6c43d in bulk_import_start /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/import-threads.c:3146
    #14 0x7f555aa6e08f in ldbm_back_wire_import /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/import-threads.c:3418
    #15 0x7f556a2a25a0 in process_bulk_import_op /home/william/development/389ds/ds/ldap/servers/slapd/bulk_import.c:140
    #16 0x7f556a2a2188 in slapi_start_bulk_import /home/william/development/389ds/ds/ldap/servers/slapd/bulk_import.c:47
    #17 0x7f555977eb1b in multimaster_extop_StartNSDS50ReplicationRequest /home/william/development/389ds/ds/ldap/servers/plugins/replication/repl_extop.c:955
    #18 0x7f556a383d9b in plugin_call_exop_plugins /home/william/development/389ds/ds/ldap/servers/slapd/plugin.c:546
    #19 0x43bcab in do_extended /home/william/development/389ds/ds/ldap/servers/slapd/extendop.c:354
    #20 0x41faf8 in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:680
    #21 0x4252d2 in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1759
    #22 0x7f55677857de in _pt_root /usr/src/debug/nspr-4.12.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216

Thread T24 created by T0 here:
    #0 0x7f556a76f538 in pthread_create _asan_rtl_
    #1 0x7f55677854ba in _PR_CreateThread /usr/src/debug/nspr-4.12.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457

Thread T20 created by T0 here:
    #0 0x7f556a76f538 in pthread_create _asan_rtl_
    #1 0x7f55677854ba in _PR_CreateThread /usr/src/debug/nspr-4.12.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457

Thread T33 created by T0 here:
    #0 0x7f556a76f538 in pthread_create _asan_rtl_
    #1 0x7f55677854ba in _PR_CreateThread /usr/src/debug/nspr-4.12.0/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457

SUMMARY: AddressSanitizer: heap-use-after-free (/opt/dirsrv/lib/dirsrv/plugins/libback-ldbm.so+0x70e47)
Shadow bytes around the buggy address:
  0x0c2280087460: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c2280087470: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c2280087480: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c2280087490: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c22800874a0: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c22800874b0: fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c22800874c0: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x0c22800874d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c22800874e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c22800874f0: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
  0x0c2280087500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==13107==ABORTING

Triggered by ticket47966_test.py


Here is the code in question that triggers the use after free:

{{{

next = handle->dblayer_handle_next;
*((dblayer_handle )handle->dblayer_handle_ai_backpointer) = NULL; //<<-- This one
slapi_ch_free((void
)&handle);

}}}

Now, according to the header:

{{{
struct tag_dblayer_handle
{
...
void dblayer_handle_ai_backpointer; / Voodo magic pointer to the place where we store a·
pointer to this handle in the attrinfo structure
/
};
}}}

Which doesn't make a lot of sense. Looking at it, this value is only in three places:

{{{
./ldap/servers/slapd/back-ldbm/back-ldbm.h:685: void dblayer_handle_ai_backpointer; / Voodo magic pointer to the place where we store a
./ldap/servers/slapd/back-ldbm/dblayer.c:2519:
((dblayer_handle
)handle->dblayer_handle_ai_backpointer) = NULL;
./ldap/servers/slapd/back-ldbm/dblayer.c:3145: handle->dblayer_handle_ai_backpointer = &(a->ai_dblayer);
}}}

Looking at this, we can see one is the definition, one is the clearing of the pointer. It looks like we set this value in dblayer.c 3145, but never use it.

{{{
int dblayer_get_index_file(backend be, struct attrinfo a, DB** ppDB, int open_flags)
...
a->ai_dblayer = handle;
handle->dblayer_handle_ai_backpointer = &(a->ai_dblayer);
}}}

So thinking about it, I think there are a few possibilities.

First, we NULL the dblayer_handle_ai_backpointer in attrinfo_delete(), because we have access to it via a->ai_dblayer->dblayer_handle_ai_backpointer, and we can then remove the offending code in dblayer.c:2519.

Alternately, we can not call attrinfo_delete in vlvIndex_delete() and various other places, opting to use attrinfo_delete in dblayer.c:2519

Or finally, we can not set the backpointer at all. We don't use it, or reference it anywhere! So why both setting something that we don't use?

Thoughts?

Could it be possible to run the test with this patch?
0001-Ticket-48987-Heap-use-after-free-in-dblayer_close_in.patch​

Thanks!!

Hmmm I think that patch causes a different issue to occur now.

I think I'll have to investigate this later when I have more time to spend on the problem.

{{{ ================================================================= ==25879==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000642d60 at pc 0x7f1a8e8d1d71 bp 0x7f1a652f1830 sp 0x7f1a652f1820 READ of size 8 at 0x603000642d60 thread T76 #0 0x7f1a8e8d1d70 in dblayer_get_index_file /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/dblayer.c:3072:0 #1 0x7f1a8e95ac24 in _entryrdn_open_index /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/ldbm_entryrdn.c:1618:0 #2 0x7f1a8e95526c in entryrdn_index_entry /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/ldbm_entryrdn.c:214:0 #3 0x7f1a8e91b2e4 in foreman_do_entryrdn /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/import-threads.c:2417:0 #4 0x7f1a8e91c105 in import_foreman /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/import-threads.c:2565:0 #5 0x7f1a9aa695cb in _pt_root /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216:0 #6 0x7f1a9a82b5c9 in start_thread /usr/src/debug/glibc-2.23-81-g2eda04e/nptl/pthread_create.c:333:0 #7 0x7f1a9a564f6c in __clone /usr/src/debug////////glibc-2.23-81-g2eda04e/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109:0 0x603000642d60 is located 0 bytes inside of 32-byte region [0x603000642d60,0x603000642d80) freed by thread T62 here: #0 0x7f1a9dacbb00 in __interceptor_free _asan_rtl_:0 #1 0x7f1a9d5756b6 in slapi_ch_free /home/william/development/389ds/ds/ldap/servers/slapd/ch_malloc.c:274:0 #2 0x7f1a8e8cf55b in dblayer_close_indexes /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/dblayer.c:2496:0 #3 0x7f1a8e8cf955 in dblayer_instance_close /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/dblayer.c:2539:0 #4 0x7f1a8e91f833 in bulk_import_start /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/import-threads.c:3131:0 #5 0x7f1a8e921435 in ldbm_back_wire_import /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/import-threads.c:3419:0 #6 0x7f1a9d572ce5 in process_bulk_import_op /home/william/development/389ds/ds/ldap/servers/slapd/bulk_import.c:140:0 #7 0x7f1a9d5728df in slapi_start_bulk_import /home/william/development/389ds/ds/ldap/servers/slapd/bulk_import.c:47:0 #8 0x7f1a8d63b518 in multimaster_extop_StartNSDS50ReplicationRequest /home/william/development/389ds/ds/ldap/servers/plugins/replication/repl_extop.c:965:0 #9 0x7f1a9d64ca73 in plugin_call_exop_plugins /home/william/development/389ds/ds/ldap/servers/slapd/plugin.c:547:0 #10 0x437969 in do_extended /home/william/development/389ds/ds/ldap/servers/slapd/extendop.c:348:0 #11 0x41e8ad in connection_dispatch_operation /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:676:0 #12 0x42387a in connection_threadmain /home/william/development/389ds/ds/ldap/servers/slapd/connection.c:1760:0 #13 0x7f1a9aa695cb in _pt_root /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:216:0 previously allocated by thread T0 here: #0 0x7f1a9dacc020 in calloc _asan_rtl_:0 #1 0x7f1a9d575274 in slapi_ch_calloc /home/william/development/389ds/ds/ldap/servers/slapd/ch_malloc.c:183:0 #2 0x7f1a8e8d1f7a in dblayer_get_index_file /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/dblayer.c:3096:0 #3 0x7f1a8e95ac24 in _entryrdn_open_index /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/ldbm_entryrdn.c:1618:0 #4 0x7f1a8e955c9d in entryrdn_index_read_ext /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/ldbm_entryrdn.c:376:0 #5 0x7f1a8e8e756b in dn2entry_ext /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/dn2entry.c:63:0 #6 0x7f1a8e8ed227 in find_entry_internal_dn /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/findentry.c:106:0 #7 0x7f1a8e8edd1d in find_entry_internal /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/findentry.c:333:0 #8 0x7f1a8e8eddb0 in find_entry /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/findentry.c:352:0 #9 0x7f1a8e979889 in ldbm_back_search /home/william/development/389ds/ds/ldap/servers/slapd/back-ldbm/ldbm_search.c:587:0 #10 0x7f1a9d626619 in op_shared_search /home/william/development/389ds/ds/ldap/servers/slapd/opshared.c:807:0 #11 0x7f1a9d660182 in search_internal_callback_pb /home/william/development/389ds/ds/ldap/servers/slapd/plugin_internal_op.c:783:0 #12 0x7f1a9d65f306 in slapi_search_internal_callback_pb /home/william/development/389ds/ds/ldap/servers/slapd/plugin_internal_op.c:564:0 #13 0x7f1a9125e6f2 in aclinit_search_and_update_aci /home/william/development/389ds/ds/ldap/servers/plugins/acl/aclinit.c:273:0 #14 0x7f1a9125e263 in aclinit_main /home/william/development/389ds/ds/ldap/servers/plugins/acl/aclinit.c:146:0 #15 0x7f1a9127d29f in aclplugin_init /home/william/development/389ds/ds/ldap/servers/plugins/acl/aclplugin.c:270:0 #16 0x7f1a9d6537b8 in plugin_call_func /home/william/development/389ds/ds/ldap/servers/slapd/plugin.c:2066:0 #17 0x7f1a9d653494 in plugin_call_one /home/william/development/389ds/ds/ldap/servers/slapd/plugin.c:2014:0 #18 0x7f1a9d6523c0 in plugin_dependency_startall /home/william/development/389ds/ds/ldap/servers/slapd/plugin.c:1786:0 #19 0x7f1a9d653423 in plugin_startall /home/william/development/389ds/ds/ldap/servers/slapd/plugin.c:1986:0 #20 0x43e0b5 in main /home/william/development/389ds/ds/ldap/servers/slapd/main.c:1081:0 #21 0x7f1a9a482730 in __libc_start_main /usr/src/debug/glibc-2.23-81-g2eda04e/csu/../csu/libc-start.c:289:0 Thread T76 created by T75 here: #0 0x7f1a9da36498 in pthread_create _asan_rtl_:0 #1 0x7f1a9aa692aa in _PR_CreateThread /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457:0 Thread T75 created by T62 here: #0 0x7f1a9da36498 in pthread_create _asan_rtl_:0 #1 0x7f1a9aa692aa in _PR_CreateThread /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457:0 Thread T62 created by T0 here: #0 0x7f1a9da36498 in pthread_create _asan_rtl_:0 #1 0x7f1a9aa692aa in _PR_CreateThread /usr/src/debug/nspr-4.13.1/pr/src/pthreads/../../../nspr/pr/src/pthreads/ptthread.c:457:0 SUMMARY: AddressSanitizer: heap-use-after-free (/opt/dirsrv/lib/dirsrv/plugins/libback-ldbm.so+0x70d70) Shadow bytes around the buggy address: 0x0c06800c0550: fa fa fd fd fd fd fa fa fd fd fd fa fa fa fd fd 0x0c06800c0560: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c06800c0570: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c06800c0580: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c06800c0590: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa =>0x0c06800c05a0: fd fd fd fd fa fa fd fd fd fa fa fa[fd]fd fd fd 0x0c06800c05b0: fa fa fd fd fd fd fa fa fd fd fd fa fa fa fd fd 0x0c06800c05c0: fd fa fa fa fd fd fd fd fa fa fd fd fd fd fa fa 0x0c06800c05d0: fd fd fd fd fa fa fd fd fd fa fa fa fd fd fd fa 0x0c06800c05e0: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c06800c05f0: fd fa fa fa 00 00 02 fa fa fa fd fd fd fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==25879==ABORTING }}} It looks like when you free this, the attrinfo is re-used. I'm assuming the issue arises when a ai_dblayer is set in attrinfo a, but then when we call dblayer_close_indexes, we are not freeing the associated attrinfo's, so they get reused when dblayer_get_index_file is called next. This causes a new fault to be observed. {{{ int dblayer_get_index_file(backend *be, struct attrinfo *a, DB** ppDB, int open_flags) .... PR_AtomicIncrement(&a->ai_dblayer_count); if (a->ai_dblayer && ((dblayer_handle*)(a->ai_dblayer))->dblayer_dbp) { // this is line 3072 /* This means that the pointer is valid, so we should return it. */ *ppDB = ((dblayer_handle*)(a->ai_dblayer))->dblayer_dbp; .... }}} Patch attached which handles this.

Looks good to me. Thanks!

commit 753f7c5bbee286dab2a2321a7eb7e46d3df5e652
Writing objects: 100% (17/17), 2.53 KiB | 0 bytes/s, done.
Total 17 (delta 13), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
4d448d3..9917640 master -> master

fa54d0c..f4b2a54 389-ds-base-1.2.11 -> 389-ds-base-1.2.11
commit f4b2a54
Author: Noriko Hosoi nhosoi@redhat.com
Date: Wed Sep 28 15:28:28 2016 -0700

2dd62a7..85f4cd6 389-ds-base-1.3.4 -> 389-ds-base-1.3.4
commit 85f4cd6

60a4fa0..6c39c95 389-ds-base-1.3.5 -> 389-ds-base-1.3.5
commit 6c39c95

Metadata Update from @nhosoi:
- Issue assigned to firstyear
- Issue set to the milestone: 1.3.6 backlog

3 years ago

Login to comment on this ticket.

Metadata