#50709 Several memory leaks reported by Valgrind for 389-ds 1.3.9.1-10.
Closed: fixed a month ago by tbordaz. Opened 3 months ago by tbordaz.

Ticket was cloned from Red Hat Bugzilla (product Red Hat Enterprise Linux 7): Bug 1769418

Description of problem:

A customer is experiencing a high memory usage with RHDS.
The memory consumption starts from a few GB and can go up to 100 GB after a
week.

Version-Release number of selected component (if applicable):

$ cat etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
$
$ grep 389-ds-base installed-rpms
389-ds-base-1.3.9.1-10.el7.x86_64                           Mon Aug 26 14:13:40
2019
389-ds-base-debuginfo-1.3.9.1-10.el7.x86_64                 Tue Nov  5 10:20:03
2019
389-ds-base-libs-1.3.9.1-10.el7.x86_64                      Mon Aug 26 14:13:26
2019
$

How reproducible:

Always at customer site.

Steps to Reproduce:

1. Restart the RHDS instance.
The sum of the configured caches is about 1.3 GB:
$ grep total errors | tail -1
[05/Nov/2019:11:32:52.600935718 +0000] - NOTICE - ldbm_back_start - total cache
size: 1316466524 B;
$

2. The memory usage will grow significantly:
Eg:
$ head -1 ./ps
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
$
$ grep ns-slapd ./ps
dirsrv    3696 79.9 93.6 64044164 61602916 ?   Ssl  Oct15 18237:22
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-XXX -i /var/run/dirsrv/slapd-XXX.pid
$


Actual results:

High memory usage indicating memory leaks.

Expected results:

The memory usage should not go that high.
Need to fix the memory leaks.

Additional info:

From the Valgrind output:

* The leak summary:

Direct leak of 112 byte(s) in 1 object(s) allocated from:
    #0 0x7f83d893cc58 in __interceptor_malloc (/lib64/libasan.so.5+0x10dc58)
    #1 0x7f83d859b028 in slapi_ch_malloc (/usr/lib64/dirsrv/libslapd.so.0+0xdb028)
    #2 0x7f83d4267069 in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:265
    #3 0x7f83d40a3ba9 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.3.20191112git15789e895.so+0x46ba9)
    #4 0x7f83d40a0e9c in LASIpEval lib/libaccess/lasip.cpp:496
    #5 0x7f83d40a5168 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    #6 0x7f83d40a6ef2 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    #7 0x7f83d40a7f2d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.3.20191112git15789e895.so+0x4af2d)
    #8 0x7f83d4242a00 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    #9 0x7f83d424be3e in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x22e3e)
    #10 0x7f83d427e2a0 in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    #11 0x7f83d86803c9 in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c03c9)
    #12 0x7f83d86810bd in slapi_access_allowed (/usr/lib64/dirsrv/libslapd.so.0+0x1c10bd)
    #13 0x7f83d424ed11 in acl_check_mods (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x25d11)
    #14 0x7f83d8680632 in plugin_call_acl_mods_access (/usr/lib64/dirsrv/libslapd.so.0+0x1c0632)
    #15 0x7f83d1948699 in ldbm_back_modify ldap/servers/slapd/back-ldbm/ldbm_modify.c:616
    #16 0x7f83d863c019 in op_shared_modify ldap/servers/slapd/modify.c:1021
    #17 0x7f83d863fa93 in do_modify (/usr/lib64/dirsrv/libslapd.so.0+0x17fa93)
    #18 0x561c3cf403f9 in connection_dispatch_operation ldap/servers/slapd/connection.c:636
    #19 0x561c3cf403f9 in connection_threadmain ldap/servers/slapd/connection.c:1765
    #20 0x7f83d8287868  (/lib64/libnspr4.so+0x2a868)

Metadata Update from @tbordaz:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1769418

3 months ago

Metadata Update from @tbordaz:
- Issue assigned to tbordaz

3 months ago

I'm sorry, but I don't understand how the leak in acl relates to the memory leak reported by the customer. The customer leaks were in ssl_Recv(), so how does the acl leak fit in

Metadata Update from @lkrispen:
- Assignee reset
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None

3 months ago

@lkrispen the valgrind output contains several signatures. One of them is related to DS_LASIpGetter. It is not the only one but account for one third of the overall leaks.

thanks, I hadn't looked at the full report, sorry for the noise

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.4.0 (was: 0.0 NEEDS_TRIAGE)

3 months ago

8a604aa..463d6b7 master
19b24c8..8fbe976 389-ds-base-1.4.1
881a53f..e141883 389-ds-base-1.4.0

Metadata Update from @tbordaz:
- Issue assigned to tbordaz

2 months ago

Metadata Update from @tbordaz:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

2 months ago

This fix causes a crash in ASAN:

py.test ./paged_results_test.py::test_search_dns_ip_aci

ASAN output:

==1009==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000079f60 at pc 0x7fd3d80cdd7d bp 0x7fd39cef5b20 sp 0x7fd39cef5b10
READ of size 4 at 0x60b000079f60 thread T13
    #0 0x7fd3d80cdd7c in slapi_pblock_set (/usr/lib64/dirsrv/libslapd.so.0+0x1b1d7c)
    #1 0x7fd3c94ed3cd in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:308
    #2 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    #3 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    #4 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    #5 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    #6 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    #7 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    #8 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    #9 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    #10 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    #11 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    #12 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    #13 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    #14 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    #15 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    #16 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    #17 0x7fd3d80afc64 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x193c64)
    #18 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    #19 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    #20 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    #21 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)
    #22 0x7fd3d527f2dd in start_thread (/lib64/libpthread.so.0+0x82dd)
    #23 0x7fd3d4a13e82 in __GI___clone (/lib64/libc.so.6+0xfbe82)

0x60b000079f60 is located 0 bytes inside of 112-byte region [0x60b000079f60,0x60b000079fd0)
freed by thread T13 here:
    #0 0x7fd3d857b7b0 in __interceptor_free (/lib64/libasan.so.5+0xef7b0)
    #1 0x7fd3d7ff7a6c in slapi_ch_free (/usr/lib64/dirsrv/libslapd.so.0+0xdba6c)
    #2 0x7fd3d80c5830 in slapi_pblock_set (/usr/lib64/dirsrv/libslapd.so.0+0x1a9830)
    #3 0x7fd3c94ed3cd in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:308
    #4 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    #5 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    #6 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    #7 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    #8 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    #9 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    #10 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    #11 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    #12 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    #13 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    #14 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    #15 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    #16 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    #17 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    #18 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    #19 0x7fd3d80afc64 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x193c64)
    #20 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    #21 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    #22 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    #23 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)

previously allocated by thread T20 here:
    #0 0x7fd3d857bb78 in __interceptor_malloc (/lib64/libasan.so.5+0xefb78)
    #1 0x7fd3d7ff7277 in slapi_ch_malloc (/usr/lib64/dirsrv/libslapd.so.0+0xdb277)
    #2 0x7fd3c94ed499 in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:269
    #3 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    #4 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    #5 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    #6 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    #7 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    #8 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    #9 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    #10 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    #11 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    #12 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    #13 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    #14 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    #15 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    #16 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    #17 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    #18 0x7fd3d80b0191 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x194191)
    #19 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    #20 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    #21 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    #22 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)

Metadata Update from @mreynolds:
- Issue status updated to: Open (was: Closed)

a month ago

New PR for the heap-user-after-free regression
https://pagure.io/389-ds-base/pull-request/50833

Please cherry-pick to 1.4.2 & 1.4.1, thanks!

New PR for the heap-user-after-free regression
https://pagure.io/389-ds-base/pull-request/50833

Please cherry-pick to 1.4.2 & 1.4.1, thanks!

And 1.3.10 for customer hot fix...

77327b009...058f4da master
b747be0..b05c86d 389-ds-base-1.4.2
44d9b73..1d748c5 389-ds-base-1.4.1
ea0af42..9fd5623 389-ds-base-1.4.0

77327b0...058f4da master
b747be0..b05c86d 389-ds-base-1.4.2
44d9b73..1d748c5 389-ds-base-1.4.1
ea0af42..9fd5623 389-ds-base-1.4.0

Don't forget 1.3.10 ;-)

bcb1e24..f1d48ed 389-ds-base-1.3.10 (thanks @vashirov for the asan build !)

Metadata Update from @tbordaz:
- Issue set to the milestone: 1.3.10 (was: 1.4.0)

a month ago

Metadata Update from @tbordaz:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

a month ago

Unfortunately the backport in 1.3.10 is invalid :( and triggers a hang
Working on a fix

Metadata Update from @tbordaz:
- Issue status updated to: Open (was: Closed)

a month ago

fixing 1.3.10 backport: f1d48ed..dab015b 389-ds-base-1.3.10

Metadata Update from @tbordaz:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

a month ago

Login to comment on this ticket.

Metadata