#48891 ns-slapd crashes during the shutdown after adding attribute with a matching rule
Closed: Fixed None Opened 3 years ago by nhosoi.

Description of problem:
Crash is observed in automated tests in indexes test suite (test case
indexes3).

How reproducible:
always

Additional info:
core_backtrace:
:{   "signal": 11
:,   "executable": "/usr/sbin/ns-slapd"
:,   "stacktrace":
:      [ {   "crash_thread": true
:        ,   "frames":
:              [ {   "address": 140696621491566
:                ,   "build_id": "32976b8813b497dac812200e859d7455f9058c61"
:                ,   "build_id_offset": 623982
:                ,   "function_name": "objset_first_obj"
:                ,   "file_name": "/usr/lib64/dirsrv/libslapd.so.0.0.0"
:                }
:              , {   "address": 140696424276440
:                ,   "build_id": "b3e230bd8e99fd9c94e6287dd6528ae60312f0fd"
:                ,   "build_id_offset": 143832
:                ,   "function_name": "dblayer_terminate"
:                ,   "file_name": "/usr/lib64/dirsrv/plugins/libback-ldbm.so"
:                }
:              , {   "address": 140696424263570
:                ,   "build_id": "b3e230bd8e99fd9c94e6287dd6528ae60312f0fd"
:                ,   "build_id_offset": 130962
:                ,   "function_name": "ldbm_back_cleanup"
:                ,   "file_name": "/usr/lib64/dirsrv/plugins/libback-ldbm.so"
:                }
:              , {   "address": 140696621209691
:                ,   "build_id": "32976b8813b497dac812200e859d7455f9058c61"
:                ,   "build_id_offset": 342107
:                ,   "function_name": "be_cleanupall"
:                ,   "file_name": "/usr/lib64/dirsrv/libslapd.so.0.0.0"
:                }
:              , {   "address": 140696626657880
:                ,   "build_id": "0e11d0e385de2114fcd2fc170b8e6c6000d9977a"
:                ,   "build_id_offset": 133720
:                ,   "function_name": "slapd_daemon"
:                ,   "file_name": "/usr/sbin/ns-slapd"
:                }
:              , {   "address": 140696626598147
:                ,   "build_id": "0e11d0e385de2114fcd2fc170b8e6c6000d9977a"
:                ,   "build_id_offset": 73987
:                ,   "function_name": "main"
:                ,   "file_name": "/usr/sbin/ns-slapd"
:                } ]
:        } ]
:}

A possible test case is to create two suffixes. Then both backends will refer the same be_database (see bz).
Then at shutdown plugin_free will be called for each backends that will do a double free.

This is likely related to the change https://fedorahosted.org/389/ticket/48872, that call plugin_free for each backends.

I'm happy for this to be reverted, I will keep this test in mind when I come back to this. The intent was to prevent a memory leak on shutdown.

Ack from me.

git '''merge''' ticket_48891
Updating 250a49d..7b8fc84
Fast-forward
dirsrvtests/tests/tickets/ticket48891_test.py | 175 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ldap/servers/slapd/backend_manager.c | 1 -
ldap/servers/slapd/daemon.c | 2 +-
ldap/servers/slapd/plugin.c | 7 +---
ldap/servers/slapd/proto-slap.h | 1 -
5 files changed, 178 insertions(+), 8 deletions(-)
create mode 100644 dirsrvtests/tests/tickets/ticket48891_test.py

git '''push''' origin master
Counting objects: 13, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (13/13), 3.29 KiB | 0 bytes/s, done.
Total 13 (delta 10), reused 1 (delta 1)
To ssh://git.fedorahosted.org/git/389/ds.git
250a49d..7b8fc84 master -> master

commit 7b8fc84
Author: Thierry Bordaz tbordaz@redhat.com
Date: Tue Jun 21 10:28:03 2016 +0200

The bug fix https://fedorahosted.org/389/ticket/48872 only exists in 1.3.5 branch.
Closing the ticket

My build from the master branch shows Invalid read.
{{{
==26679== Invalid read of size 8
==26679== at 0x4C696CC: be_cleanupall (backend_manager.c:318)
==26679== by 0x128340: slapd_daemon (daemon.c:1345)
==26679== by 0x13198E: main (main.c:1143)
==26679== Address 0x11673ac0 is 304 bytes inside a block of size 696 free'd
==26679== at 0x4A07D6A: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==26679== by 0x4C6B916: slapi_ch_free (ch_malloc.c:292)
==26679== by 0x4CD6739: plugin_free (plugin.c:2732)
==26679== by 0x4CD4C96: plugin_closeall (plugin.c:1959)
==26679== by 0x1282E6: slapd_daemon (daemon.c:1321)
==26679== by 0x13198E: main (main.c:1143)
==26679==
==26679== Invalid read of size 8
==26679== at 0x4C697A8: be_cleanupall (backend_manager.c:325)
==26679== by 0x128340: slapd_daemon (daemon.c:1345)
==26679== by 0x13198E: main (main.c:1143)
==26679== Address 0x11673ac0 is 304 bytes inside a block of size 696 free'd
==26679== at 0x4A07D6A: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==26679== by 0x4C6B916: slapi_ch_free (ch_malloc.c:292)
==26679== by 0x4CD6739: plugin_free (plugin.c:2732)
==26679== by 0x4CD4C96: plugin_closeall (plugin.c:1959)
==26679== by 0x1282E6: slapd_daemon (daemon.c:1321)
==26679== by 0x13198E: main (main.c:1143)
==26679==
==26679== Invalid read of size 8
==26679== at 0x4CCA458: slapi_pblock_get (pblock.c:442)
==26679== by 0x127B377A: ldbm_back_cleanup (cleanup.c:24)
==26679== by 0x4C697BF: be_cleanupall (backend_manager.c:325)
==26679== by 0x128340: slapd_daemon (daemon.c:1345)
==26679== by 0x13198E: main (main.c:1143)
==26679== Address 0x11673990 is 0 bytes inside a block of size 696 free'd
==26679== at 0x4A07D6A: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==26679== by 0x4C6B916: slapi_ch_free (ch_malloc.c:292)
==26679== by 0x4CD6739: plugin_free (plugin.c:2732)
==26679== by 0x4CD4C96: plugin_closeall (plugin.c:1959)
==26679== by 0x1282E6: slapd_daemon (daemon.c:1321)
==26679== by 0x13198E: main (main.c:1143)
}}}
The patch 0001-Ticket-48891-ns-slapd-crashes-during-the-shutdown-af.patch​ eliminated some changes made by the fix in #48872. But it seems we still have to skip the backend release in plugin_closeall... By resurrecting this code, valgrind stops complaining the invalid read...
{{{
diff --git a/ldap/servers/slapd/daemon.c b/ldap/servers/slapd/daemon.c
index 862f6e7..90d1f7d 100644
--- a/ldap/servers/slapd/daemon.c
+++ b/ldap/servers/slapd/daemon.c
@@ -1318,7 +1318,7 @@ void slapd_daemon( daemon_ports_t *ports )
uniqueIDGenCleanup ();
}

  • plugin_closeall( 1 / Close Backends /, 1 / Close Globals /);
  • plugin_closeall( 0 / Close Backends /, 1 / Close Globals /);
    if ( ! in_referral_mode ) {
            /* Close SNMP collator after the plugins closed...
    

    diff --git a/ldap/servers/slapd/plugin.c b/ldap/servers/slapd/plugin.c
    index d377b33..cc92943 100644
    --- a/ldap/servers/slapd/plugin.c
    +++ b/ldap/servers/slapd/plugin.c
    @@ -1955,6 +1955,10 @@ plugin_closeall(int close_backends, int close_globals)
    iterp = dep_plugin_entries;
    while (iterp) {
    nextp = iterp->next;
    + if (!close_backends && (iterp->plugin->plg_type == SLAPI_PLUGIN_DATABASE)) {
    + iterp = nextp;
    + continue;
    + }
    slapi_entry_free(iterp->e);
    plugin_free(iterp->plugin);
    slapi_ch_free((void **)&iterp);
    }}}
    Let me reopen this ticket and bug.

Replying to [comment:12 mreynolds]:

ack
+1
note: I also verified valgrind no longer reports Invalid read.
Thanks!

Noriko, Mark thanks for this review and for your tests !

'''git merge ticket_48891'''
Updating 10949b7..5f7bcdc
Fast-forward
ldap/servers/slapd/daemon.c | 1 +
ldap/servers/slapd/plugin.c | 47 +++++++++++++++++++++++++++++++++++------------
ldap/servers/slapd/proto-slap.h | 1 +
3 files changed, 37 insertions(+), 12 deletions(-)

'''git push origin master'''
Counting objects: 8, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 1.41 KiB | 0 bytes/s, done.
Total 8 (delta 6), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
10949b7..5f7bcdc master -> master

commit 5f7bcdc
Author: Thierry Bordaz tbordaz@redhat.com
Date: Thu Jun 23 15:48:58 2016 +0200

Metadata Update from @tbordaz:
- Issue assigned to tbordaz
- Issue set to the milestone: 1.3.5.9

2 years ago

Login to comment on this ticket.

Metadata