Another deadlock occurred involving memberof and retrocl, but what is specific to this case is that the sequence of calls memberof--> retrocl was triggered by the deletion of a tombstone.
tombstones are entries which are deleted, plugins have been called in the delete op, and purging is an internal db operation only.
For reference the two threads:
#0 0x00007f9832a7a705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f98330d02c3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007f98291f7132 in dblayer_lock_backend (be=<optimized out>) at ldap/servers/slapd/back-ldbm/dblayer.c:3946 #3 0x00007f98291fbf36 in dblayer_txn_begin (be=0x7f98367397e0, parent_txn=0x7f95cd065af0, txn=txn@entry=0x7f95dfff1370) at ldap/servers/slapd/back-ldbm/dblayer.c:3668 #4 0x00007f982921bb67 in ldbm_back_add (pb=0x7f95ccb1d430) at ldap/servers/slapd/back-ldbm/ldbm_add.c:272 #5 0x00007f9834c8bf1a in op_shared_add (pb=pb@entry=0x7f95ccb1d430) at ldap/servers/slapd/add.c:735 #6 0x00007f9834c8c773 in add_internal_pb (pb=pb@entry=0x7f95ccb1d430) at ldap/servers/slapd/add.c:434 #7 0x00007f9834c8d453 in slapi_add_internal_pb (pb=pb@entry=0x7f95ccb1d430) at ldap/servers/slapd/add.c:356 #8 0x00007f9827aa632d in write_replog_db (newsuperior=0x0, modrdn_mods=0x0, newrdn=0x0, log_e=0x0, curtime=1457532345, flag=0, log_m=0x7f95cd50deb0, dn=0x7f95cc7760a0 "uid=ccrgst74,cn=users,cn=accounts,dc=cbls,dc=ccr,dc=buffalo,dc=edu", optype=<optimized out>, pb=0x7f95cd45d8d0) at ldap/servers/plugins/retrocl/retrocl_po.c:377 #9 retrocl_postob (pb=0x7f95cd45d8d0, optype=<optimized out>) at ldap/servers/plugins/retrocl/retrocl_po.c:682 #10 0x00007f9834ce5280 in plugin_call_func (list=0x7f9836516b80, operation=operation@entry=561, pb=pb@entry=0x7f95cd45d8d0, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1952 #11 0x00007f9834ce54d8 in plugin_call_list (pb=0x7f95cd45d8d0, operation=561, list=<optimized out>) at ldap/servers/slapd/plugin.c:1886 #12 plugin_call_plugins (pb=pb@entry=0x7f95cd45d8d0, whichfunction=whichfunction@entry=561) at ldap/servers/slapd/plugin.c:459 #13 0x00007f9829236be9 in ldbm_back_modify (pb=<optimized out>) at ldap/servers/slapd/back-ldbm/ldbm_modify.c:847 #14 0x00007f9834cd20e1 in op_shared_modify (pb=pb@entry=0x7f95cd45d8d0, pw_change=pw_change@entry=0, old_pw=0x0) at ldap/servers/slapd/modify.c:1086 #15 0x00007f9834cd2ba4 in modify_internal_pb (pb=pb@entry=0x7f95cd45d8d0) at ldap/servers/slapd/modify.c:631 #16 0x00007f9834cd36a3 in slapi_modify_internal_pb (pb=pb@entry=0x7f95cd45d8d0) at ldap/servers/slapd/modify.c:486 #17 0x00007f98289116f1 in memberof_fix_memberof_callback (e=<optimized out>, callback_data=callback_data@entry=0x7f95dfff6030) at ldap/servers/plugins/memberof/memberof.c:2902 #18 0x00007f98289121d5 in memberof_modop_one_replace_r (pb=pb@entry=0x7f95cd5ee4b0, config=config@entry=0x7f95dfff6030, mod_op=mod_op@entry=1, group_sdn=group_sdn@entry=0x7f95cd35fad0, op_this_sdn=op_this_sdn@entry=0x7f95cd35fad0, replace_with_sdn=replace_with_sdn@entry=0x0, op_to_sdn=0x7f95cc862bc0, stack=stack@entry=0x0) at ldap/servers/plugins/memberof/memberof.c:1674 #19 0x00007f98289123c5 in memberof_modop_one_r (stack=0x0, op_to_sdn=<optimized out>, op_this_sdn=0x7f95cd35fad0, group_sdn=0x7f95cd35fad0, mod_op=1, config=0x7f95dfff6030, pb=0x7f95cd5ee4b0) at ldap/servers/plugins/memberof/memberof.c:1417 #20 memberof_mod_attr_list_r (pb=pb@entry=0x7f95cd5ee4b0, config=config@entry=0x7f95dfff6030, mod=mod@entry=1, group_sdn=group_sdn@entry=0x7f95cd35fad0, op_this_sdn=op_this_sdn@entry=0x7f95cd35fad0, attr=0x7f95cd28ec80, stack=stack@entry=0x0) at ldap/servers/plugins/memberof/memberof.c:1919 #21 0x00007f98289143e6 in memberof_mod_attr_list (attr=<optimized out>, group_sdn=0x7f95cd35fad0, mod=1, config=0x7f95dfff6030, pb=0x7f95cd5ee4b0) at ldap/servers/plugins/memberof/memberof.c:1858 #22 memberof_del_attr_list (attr=<optimized out>, group_sdn=0x7f95cd35fad0, config=0x7f95dfff6030, pb=0x7f95cd5ee4b0) at ldap/servers/plugins/memberof/memberof.c:1957 #23 memberof_postop_del (pb=0x7f95cd5ee4b0) at ldap/servers/plugins/memberof/memberof.c:579 #24 0x00007f9834ce5280 in plugin_call_func (list=0x7f98364f8f20, operation=operation@entry=563, pb=pb@entry=0x7f95cd5ee4b0, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1952 #25 0x00007f9834ce54d8 in plugin_call_list (pb=0x7f95cd5ee4b0, operation=563, list=<optimized out>) at ldap/servers/slapd/plugin.c:1886 #26 plugin_call_plugins (pb=pb@entry=0x7f95cd5ee4b0, whichfunction=whichfunction@entry=563) at ldap/servers/slapd/plugin.c:459 #27 0x00007f982922942e in ldbm_back_delete (pb=0x7f95cd5ee4b0) at ldap/servers/slapd/back-ldbm/ldbm_delete.c:1226 #28 0x00007f9834c99190 in op_shared_delete (pb=pb@entry=0x7f95cd5ee4b0) at ldap/servers/slapd/delete.c:364 #29 0x00007f9834c99342 in delete_internal_pb (pb=pb@entry=0x7f95cd5ee4b0) at ldap/servers/slapd/delete.c:242 #30 0x00007f9834c995f3 in slapi_delete_internal_pb (pb=pb@entry=0x7f95cd5ee4b0) at ldap/servers/slapd/delete.c:185 #31 0x00007f9828f76af8 in _delete_tombstone (tombstone_dn=0x7f944fde94b0 "nsuniqueid=d48e5121-cf5311e5-9a208f7e-e0b1a377,ipaUniqueID=ec5e3106-cf53-11e5-b8e8-a0369f577818,cn=hbac,dc=cbls,dc=ccr,dc=buffalo,dc=edu", uniqueid=0x7f95bee65750 "d48e5121-cf5311e5-9a208f7e-e0b1a377", ext_op_flags=0) at ldap/servers/plugins/replication/repl5_replica.c:2932 #32 0x00007f9828f79362 in process_reap_entry (entry=0x7f944f540430, cb_data=0x7f95dfffecc0) at ldap/servers/plugins/replication/repl5_replica.c:2993 #33 0x00007f9834cf9a7d in send_ldap_search_entry_ext (pb=pb@entry=0x7f95cd2ee540, e=<optimized out>, ectrls=ectrls@entry=0x0, attrs=0x7f95cc647490, attrsonly=0, send_result=send_result@entry=0, nentries=nentries@entry=0, urls=urls@entry=0x0) at ldap/servers/slapd/result.c:1544 #34 0x00007f9834cfa2bc in send_ldap_search_entry (pb=pb@entry=0x7f95cd2ee540, e=<optimized out>, ectrls=ectrls@entry=0x0, attrs=<optimized out>, attrsonly=<optimized out>) at ldap/servers/slapd/result.c:1084 #35 0x00007f9834cd76d3 in iterate (pb=pb@entry=0x7f95cd2ee540, pnentries=pnentries@entry=0x7f95dfff88c8, pagesize=pagesize@entry=-1, pr_statp=pr_statp@entry=0x7f95dfff8844, be=0x7f95cd2ee540, send_result=1) at ldap/servers/slapd/opshared.c:1485 #36 0x00007f9834cd787a in send_results_ext (pb=pb@entry=0x7f95cd2ee540, nentries=nentries@entry=0x7f95dfff88c8, pagesize=-1, pr_stat=pr_stat@entry=0x7f95dfff8844, send_result=1) at ldap/servers/slapd/opshared.c:1724 #37 0x00007f9834cd91c1 in op_shared_search (pb=pb@entry=0x7f95cd2ee540, send_result=send_result@entry=1) at ldap/servers/slapd/opshared.c:881 #38 0x00007f9834ce91de in search_internal_callback_pb (pb=pb@entry=0x7f95cd2ee540, callback_data=callback_data@entry=0x7f95dfffecc0, prc=prc@entry=0x7f9828f75be0 <get_reap_result>, psec=psec@entry=0x7f9828f79260 <process_reap_entry>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:812 #39 0x00007f9834ce9759 in slapi_search_internal_callback_pb (pb=pb@entry=0x7f95cd2ee540, callback_data=callback_data@entry=0x7f95dfffecc0, prc=prc@entry=0x7f9828f75be0 <get_reap_result>, psec=psec@entry=0x7f9828f79260 <process_reap_entry>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:593 #40 0x00007f9828f7b7b1 in _replica_reap_tombstones (arg=0x7f983bbdd820) at ldap/servers/plugins/replication/repl5_replica.c:3123 #41 0x00007f98330d57bb in _pt_root () from /lib64/libnspr4.so #42 0x00007f9832a76df5 in start_thread () from /lib64/libpthread.so.0 #43 0x00007f98327a41ad in clone () from /lib64/libc.so.6 #0 0x00007f9832a7a705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f98330d02c3 in PR_EnterMonitor () from /lib64/libnspr4.so #2 0x00007f982891431a in memberof_postop_del (pb=0x7f9505362290) at ldap/servers/plugins/memberof/memberof.c:554 #3 0x00007f9834ce5280 in plugin_call_func (list=0x7f98364f8f20, operation=operation@entry=563, pb=pb@entry=0x7f9505362290, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:1952 #4 0x00007f9834ce54d8 in plugin_call_list (pb=0x7f9505362290, operation=563, list=<optimized out>) at ldap/servers/slapd/plugin.c:1886 #5 plugin_call_plugins (pb=pb@entry=0x7f9505362290, whichfunction=whichfunction@entry=563) at ldap/servers/slapd/plugin.c:459 #6 0x00007f982922942e in ldbm_back_delete (pb=0x7f9505362290) at ldap/servers/slapd/back-ldbm/ldbm_delete.c:1226 #7 0x00007f9834c99190 in op_shared_delete (pb=pb@entry=0x7f9505362290) at ldap/servers/slapd/delete.c:364 #8 0x00007f9834c99342 in delete_internal_pb (pb=pb@entry=0x7f9505362290) at ldap/servers/slapd/delete.c:242 #9 0x00007f9834c995f3 in slapi_delete_internal_pb (pb=pb@entry=0x7f9505362290) at ldap/servers/slapd/delete.c:185 #10 0x00007f9827aa6afe in delete_changerecord (cnum=cnum@entry=17901598) at ldap/servers/plugins/retrocl/retrocl_trim.c:117 #11 0x00007f9827aa6dd1 in trim_changelog () at ldap/servers/plugins/retrocl/retrocl_trim.c:316 #12 changelog_trim_thread_fn (arg=<optimized out>) at ldap/servers/plugins/retrocl/retrocl_trim.c:359 #13 0x00007f98330d57bb in _pt_root () from /lib64/libnspr4.so #14 0x00007f9832a76df5 in start_thread () from /lib64/libpthread.so.0 #15 0x00007f98327a41ad in clone () from /lib64/libc.so.6
Nice one !!
Few remarks: it should not be a frequent deadlock as it requires tombstone_purging + retroCL trimming.
Something that is not clear to me: The thread doing the tombstone_purging, deletes a tombstone. Then we call membeof that actually does update an entry. I do not understand why a tombstone is still memberOf of some other entry.
Also the trimming of the retroCL calls memberof. If 'cn=changelog' is excluded from memberof scope and the version has https://fedorahosted.org/389/ticket/47931, then we could "workaround" this issue on the trimming side.
if the tombstone entry is a group, memberof might try to remove the memberof attribute from users listed in this group
Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1316742
attachment 0001-Ticket-48759-no-plugin-calls-in-tombstone-purging.patch
attachment 0001-Ticket-48759-no-plugin-calls-in-tombstone-purging.2.patch
attachment 0001-add-testcase-for-ticket-48759.patch
Looks good to me.
commits to master: commit 5d975ea commit 93f7d82
commits to 1.3.4: commit 54b200b commit c6e146b
Metadata Update from @lkrispen: - Issue assigned to lkrispen - Issue set to the milestone: 1.3.4.10
389-ds-base is moving from Pagure to Github. This means that new issues and pull requests will be accepted only in 389-ds-base's github repository.
This issue has been cloned to Github and is available here: - https://github.com/389ds/389-ds-base/issues/1819
If you want to receive further updates on the issue, please navigate to the github issue and click on subscribe button.
subscribe
Thank you for understanding. We apologize for all inconvenience.
Metadata Update from @spichugi: - Issue close_status updated to: wontfix (was: Fixed)
Login to comment on this ticket.