#50709 Several memory leaks reported by Valgrind for 389-ds 1.3.9.1-10.
Closed: wontfix 4 years ago by tbordaz. Opened 4 years ago by tbordaz.

Ticket was cloned from Red Hat Bugzilla (product Red Hat Enterprise Linux 7): Bug 1769418

Description of problem:

A customer is experiencing a high memory usage with RHDS.
The memory consumption starts from a few GB and can go up to 100 GB after a
week.

Version-Release number of selected component (if applicable):

$ cat etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
$
$ grep 389-ds-base installed-rpms
389-ds-base-1.3.9.1-10.el7.x86_64                           Mon Aug 26 14:13:40
2019
389-ds-base-debuginfo-1.3.9.1-10.el7.x86_64                 Tue Nov  5 10:20:03
2019
389-ds-base-libs-1.3.9.1-10.el7.x86_64                      Mon Aug 26 14:13:26
2019
$

How reproducible:

Always at customer site.

Steps to Reproduce:

1. Restart the RHDS instance.
The sum of the configured caches is about 1.3 GB:
$ grep total errors | tail -1
[05/Nov/2019:11:32:52.600935718 +0000] - NOTICE - ldbm_back_start - total cache
size: 1316466524 B;
$

2. The memory usage will grow significantly:
Eg:
$ head -1 ./ps
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
$
$ grep ns-slapd ./ps
dirsrv    3696 79.9 93.6 64044164 61602916 ?   Ssl  Oct15 18237:22
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-XXX -i /var/run/dirsrv/slapd-XXX.pid
$


Actual results:

High memory usage indicating memory leaks.

Expected results:

The memory usage should not go that high.
Need to fix the memory leaks.

Additional info:

From the Valgrind output:

* The leak summary:

Direct leak of 112 byte(s) in 1 object(s) allocated from:
    #0 0x7f83d893cc58 in __interceptor_malloc (/lib64/libasan.so.5+0x10dc58)
    #1 0x7f83d859b028 in slapi_ch_malloc (/usr/lib64/dirsrv/libslapd.so.0+0xdb028)
    #2 0x7f83d4267069 in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:265
    #3 0x7f83d40a3ba9 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.3.20191112git15789e895.so+0x46ba9)
    #4 0x7f83d40a0e9c in LASIpEval lib/libaccess/lasip.cpp:496
    #5 0x7f83d40a5168 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    #6 0x7f83d40a6ef2 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    #7 0x7f83d40a7f2d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.3.20191112git15789e895.so+0x4af2d)
    #8 0x7f83d4242a00 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    #9 0x7f83d424be3e in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x22e3e)
    #10 0x7f83d427e2a0 in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    #11 0x7f83d86803c9 in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c03c9)
    #12 0x7f83d86810bd in slapi_access_allowed (/usr/lib64/dirsrv/libslapd.so.0+0x1c10bd)
    #13 0x7f83d424ed11 in acl_check_mods (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x25d11)
    #14 0x7f83d8680632 in plugin_call_acl_mods_access (/usr/lib64/dirsrv/libslapd.so.0+0x1c0632)
    #15 0x7f83d1948699 in ldbm_back_modify ldap/servers/slapd/back-ldbm/ldbm_modify.c:616
    #16 0x7f83d863c019 in op_shared_modify ldap/servers/slapd/modify.c:1021
    #17 0x7f83d863fa93 in do_modify (/usr/lib64/dirsrv/libslapd.so.0+0x17fa93)
    #18 0x561c3cf403f9 in connection_dispatch_operation ldap/servers/slapd/connection.c:636
    #19 0x561c3cf403f9 in connection_threadmain ldap/servers/slapd/connection.c:1765
    #20 0x7f83d8287868  (/lib64/libnspr4.so+0x2a868)

Metadata Update from @tbordaz:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1769418

4 years ago

Metadata Update from @tbordaz:
- Issue assigned to tbordaz

4 years ago

I'm sorry, but I don't understand how the leak in acl relates to the memory leak reported by the customer. The customer leaks were in ssl_Recv(), so how does the acl leak fit in

Metadata Update from @lkrispen:
- Assignee reset
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None

4 years ago

@lkrispen the valgrind output contains several signatures. One of them is related to DS_LASIpGetter. It is not the only one but account for one third of the overall leaks.

thanks, I hadn't looked at the full report, sorry for the noise

Metadata Update from @mreynolds:
- Issue set to the milestone: 1.4.0 (was: 0.0 NEEDS_TRIAGE)

4 years ago

8a604aa..463d6b7 master
19b24c8..8fbe976 389-ds-base-1.4.1
881a53f..e141883 389-ds-base-1.4.0

Metadata Update from @tbordaz:
- Issue assigned to tbordaz

4 years ago

Metadata Update from @tbordaz:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

4 years ago

This fix causes a crash in ASAN:

py.test ./paged_results_test.py::test_search_dns_ip_aci

ASAN output:

==1009==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000079f60 at pc 0x7fd3d80cdd7d bp 0x7fd39cef5b20 sp 0x7fd39cef5b10
READ of size 4 at 0x60b000079f60 thread T13
    #0 0x7fd3d80cdd7c in slapi_pblock_set (/usr/lib64/dirsrv/libslapd.so.0+0x1b1d7c)
    #1 0x7fd3c94ed3cd in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:308
    #2 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    #3 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    #4 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    #5 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    #6 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    #7 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    #8 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    #9 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    #10 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    #11 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    #12 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    #13 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    #14 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    #15 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    #16 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    #17 0x7fd3d80afc64 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x193c64)
    #18 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    #19 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    #20 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    #21 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)
    #22 0x7fd3d527f2dd in start_thread (/lib64/libpthread.so.0+0x82dd)
    #23 0x7fd3d4a13e82 in __GI___clone (/lib64/libc.so.6+0xfbe82)

0x60b000079f60 is located 0 bytes inside of 112-byte region [0x60b000079f60,0x60b000079fd0)
freed by thread T13 here:
    #0 0x7fd3d857b7b0 in __interceptor_free (/lib64/libasan.so.5+0xef7b0)
    #1 0x7fd3d7ff7a6c in slapi_ch_free (/usr/lib64/dirsrv/libslapd.so.0+0xdba6c)
    #2 0x7fd3d80c5830 in slapi_pblock_set (/usr/lib64/dirsrv/libslapd.so.0+0x1a9830)
    #3 0x7fd3c94ed3cd in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:308
    #4 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    #5 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    #6 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    #7 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    #8 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    #9 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    #10 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    #11 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    #12 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    #13 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    #14 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    #15 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    #16 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    #17 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    #18 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    #19 0x7fd3d80afc64 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x193c64)
    #20 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    #21 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    #22 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    #23 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)

previously allocated by thread T20 here:
    #0 0x7fd3d857bb78 in __interceptor_malloc (/lib64/libasan.so.5+0xefb78)
    #1 0x7fd3d7ff7277 in slapi_ch_malloc (/usr/lib64/dirsrv/libslapd.so.0+0xdb277)
    #2 0x7fd3c94ed499 in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:269
    #3 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    #4 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    #5 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    #6 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    #7 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    #8 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    #9 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    #10 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    #11 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    #12 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    #13 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    #14 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    #15 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    #16 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    #17 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    #18 0x7fd3d80b0191 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x194191)
    #19 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    #20 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    #21 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    #22 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)

Metadata Update from @mreynolds:
- Issue status updated to: Open (was: Closed)

4 years ago

New PR for the heap-user-after-free regression
https://pagure.io/389-ds-base/pull-request/50833

Please cherry-pick to 1.4.2 & 1.4.1, thanks!

New PR for the heap-user-after-free regression
https://pagure.io/389-ds-base/pull-request/50833

Please cherry-pick to 1.4.2 & 1.4.1, thanks!

And 1.3.10 for customer hot fix...

77327b009...058f4da master
b747be0..b05c86d 389-ds-base-1.4.2
44d9b73..1d748c5 389-ds-base-1.4.1
ea0af42..9fd5623 389-ds-base-1.4.0

77327b0...058f4da master
b747be0..b05c86d 389-ds-base-1.4.2
44d9b73..1d748c5 389-ds-base-1.4.1
ea0af42..9fd5623 389-ds-base-1.4.0

Don't forget 1.3.10 ;-)

bcb1e24..f1d48ed 389-ds-base-1.3.10 (thanks @vashirov for the asan build !)

Metadata Update from @tbordaz:
- Issue set to the milestone: 1.3.10 (was: 1.4.0)

4 years ago

Metadata Update from @tbordaz:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Unfortunately the backport in 1.3.10 is invalid :( and triggers a hang
Working on a fix

Metadata Update from @tbordaz:
- Issue status updated to: Open (was: Closed)

4 years ago

fixing 1.3.10 backport: f1d48ed..dab015b 389-ds-base-1.3.10

Metadata Update from @tbordaz:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

4 years ago

389-ds-base is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in 389-ds-base's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/389ds/389-ds-base/issues/3764

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.

Metadata Update from @spichugi:
- Issue close_status updated to: wontfix (was: fixed)

3 years ago

Login to comment on this ticket.

Metadata