Ticket was cloned from Red Hat Bugzilla (product Red Hat Enterprise Linux 7): Bug 1769418
Description of problem: A customer is experiencing a high memory usage with RHDS. The memory consumption starts from a few GB and can go up to 100 GB after a week. Version-Release number of selected component (if applicable): $ cat etc/redhat-release Red Hat Enterprise Linux Server release 7.7 (Maipo) $ $ grep 389-ds-base installed-rpms 389-ds-base-1.3.9.1-10.el7.x86_64 Mon Aug 26 14:13:40 2019 389-ds-base-debuginfo-1.3.9.1-10.el7.x86_64 Tue Nov 5 10:20:03 2019 389-ds-base-libs-1.3.9.1-10.el7.x86_64 Mon Aug 26 14:13:26 2019 $ How reproducible: Always at customer site. Steps to Reproduce: 1. Restart the RHDS instance. The sum of the configured caches is about 1.3 GB: $ grep total errors | tail -1 [05/Nov/2019:11:32:52.600935718 +0000] - NOTICE - ldbm_back_start - total cache size: 1316466524 B; $ 2. The memory usage will grow significantly: Eg: $ head -1 ./ps USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND $ $ grep ns-slapd ./ps dirsrv 3696 79.9 93.6 64044164 61602916 ? Ssl Oct15 18237:22 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-XXX -i /var/run/dirsrv/slapd-XXX.pid $ Actual results: High memory usage indicating memory leaks. Expected results: The memory usage should not go that high. Need to fix the memory leaks. Additional info: From the Valgrind output: * The leak summary: Direct leak of 112 byte(s) in 1 object(s) allocated from: #0 0x7f83d893cc58 in __interceptor_malloc (/lib64/libasan.so.5+0x10dc58) #1 0x7f83d859b028 in slapi_ch_malloc (/usr/lib64/dirsrv/libslapd.so.0+0xdb028) #2 0x7f83d4267069 in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:265 #3 0x7f83d40a3ba9 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.3.20191112git15789e895.so+0x46ba9) #4 0x7f83d40a0e9c in LASIpEval lib/libaccess/lasip.cpp:496 #5 0x7f83d40a5168 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215 #6 0x7f83d40a6ef2 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752 #7 0x7f83d40a7f2d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.3.20191112git15789e895.so+0x4af2d) #8 0x7f83d4242a00 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289 #9 0x7f83d424be3e in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x22e3e) #10 0x7f83d427e2a0 in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371 #11 0x7f83d86803c9 in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c03c9) #12 0x7f83d86810bd in slapi_access_allowed (/usr/lib64/dirsrv/libslapd.so.0+0x1c10bd) #13 0x7f83d424ed11 in acl_check_mods (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x25d11) #14 0x7f83d8680632 in plugin_call_acl_mods_access (/usr/lib64/dirsrv/libslapd.so.0+0x1c0632) #15 0x7f83d1948699 in ldbm_back_modify ldap/servers/slapd/back-ldbm/ldbm_modify.c:616 #16 0x7f83d863c019 in op_shared_modify ldap/servers/slapd/modify.c:1021 #17 0x7f83d863fa93 in do_modify (/usr/lib64/dirsrv/libslapd.so.0+0x17fa93) #18 0x561c3cf403f9 in connection_dispatch_operation ldap/servers/slapd/connection.c:636 #19 0x561c3cf403f9 in connection_threadmain ldap/servers/slapd/connection.c:1765 #20 0x7f83d8287868 (/lib64/libnspr4.so+0x2a868)
Metadata Update from @tbordaz: - Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1769418
Metadata Update from @tbordaz: - Issue assigned to tbordaz
I'm sorry, but I don't understand how the leak in acl relates to the memory leak reported by the customer. The customer leaks were in ssl_Recv(), so how does the acl leak fit in
Metadata Update from @lkrispen: - Assignee reset - Custom field origin adjusted to None - Custom field reviewstatus adjusted to None
@lkrispen the valgrind output contains several signatures. One of them is related to DS_LASIpGetter. It is not the only one but account for one third of the overall leaks.
thanks, I hadn't looked at the full report, sorry for the noise
PR https://pagure.io/389-ds-base/pull-request/50710
Metadata Update from @mreynolds: - Issue set to the milestone: 1.4.0 (was: 0.0 NEEDS_TRIAGE)
8a604aa..463d6b7 master 19b24c8..8fbe976 389-ds-base-1.4.1 881a53f..e141883 389-ds-base-1.4.0
Metadata Update from @tbordaz: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
This fix causes a crash in ASAN:
py.test ./paged_results_test.py::test_search_dns_ip_aci
ASAN output:
==1009==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000079f60 at pc 0x7fd3d80cdd7d bp 0x7fd39cef5b20 sp 0x7fd39cef5b10 READ of size 4 at 0x60b000079f60 thread T13 #0 0x7fd3d80cdd7c in slapi_pblock_set (/usr/lib64/dirsrv/libslapd.so.0+0x1b1d7c) #1 0x7fd3c94ed3cd in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:308 #2 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128) #3 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496 #4 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215 #5 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752 #6 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d) #7 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289 #8 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce) #9 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371 #10 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a) #11 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956 #12 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817 #13 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa) #14 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713 #15 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307 #16 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660 #17 0x7fd3d80afc64 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x193c64) #18 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376 #19 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662 #20 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767 #21 0x7fd3d58e4567 (/lib64/libnspr4.so+0x2b567) #22 0x7fd3d527f2dd in start_thread (/lib64/libpthread.so.0+0x82dd) #23 0x7fd3d4a13e82 in __GI___clone (/lib64/libc.so.6+0xfbe82) 0x60b000079f60 is located 0 bytes inside of 112-byte region [0x60b000079f60,0x60b000079fd0) freed by thread T13 here: #0 0x7fd3d857b7b0 in __interceptor_free (/lib64/libasan.so.5+0xef7b0) #1 0x7fd3d7ff7a6c in slapi_ch_free (/usr/lib64/dirsrv/libslapd.so.0+0xdba6c) #2 0x7fd3d80c5830 in slapi_pblock_set (/usr/lib64/dirsrv/libslapd.so.0+0x1a9830) #3 0x7fd3c94ed3cd in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:308 #4 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128) #5 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496 #6 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215 #7 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752 #8 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d) #9 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289 #10 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce) #11 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371 #12 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a) #13 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956 #14 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817 #15 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa) #16 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713 #17 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307 #18 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660 #19 0x7fd3d80afc64 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x193c64) #20 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376 #21 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662 #22 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767 #23 0x7fd3d58e4567 (/lib64/libnspr4.so+0x2b567) previously allocated by thread T20 here: #0 0x7fd3d857bb78 in __interceptor_malloc (/lib64/libasan.so.5+0xefb78) #1 0x7fd3d7ff7277 in slapi_ch_malloc (/usr/lib64/dirsrv/libslapd.so.0+0xdb277) #2 0x7fd3c94ed499 in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:269 #3 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128) #4 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496 #5 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215 #6 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752 #7 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d) #8 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289 #9 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce) #10 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371 #11 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a) #12 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956 #13 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817 #14 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa) #15 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713 #16 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307 #17 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660 #18 0x7fd3d80b0191 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x194191) #19 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376 #20 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662 #21 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767 #22 0x7fd3d58e4567 (/lib64/libnspr4.so+0x2b567)
Metadata Update from @mreynolds: - Issue status updated to: Open (was: Closed)
New PR for the heap-user-after-free regression
https://pagure.io/389-ds-base/pull-request/50833
New PR for the heap-user-after-free regression https://pagure.io/389-ds-base/pull-request/50833
Please cherry-pick to 1.4.2 & 1.4.1, thanks!
New PR for the heap-user-after-free regression https://pagure.io/389-ds-base/pull-request/50833 Please cherry-pick to 1.4.2 & 1.4.1, thanks!
And 1.3.10 for customer hot fix...
77327b009...058f4da master b747be0..b05c86d 389-ds-base-1.4.2 44d9b73..1d748c5 389-ds-base-1.4.1 ea0af42..9fd5623 389-ds-base-1.4.0
77327b0...058f4da master b747be0..b05c86d 389-ds-base-1.4.2 44d9b73..1d748c5 389-ds-base-1.4.1 ea0af42..9fd5623 389-ds-base-1.4.0
Don't forget 1.3.10 ;-)
bcb1e24..f1d48ed 389-ds-base-1.3.10 (thanks @vashirov for the asan build !)
Metadata Update from @tbordaz: - Issue set to the milestone: 1.3.10 (was: 1.4.0)
Unfortunately the backport in 1.3.10 is invalid :( and triggers a hang Working on a fix
Metadata Update from @tbordaz: - Issue status updated to: Open (was: Closed)
fixing 1.3.10 backport: f1d48ed..dab015b 389-ds-base-1.3.10
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/3764
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: fixed)
Login to comment on this ticket.