#6821 Deadlock between topology and schema-compat plugins
Closed: fixed 7 years ago Opened 7 years ago by tbordaz.

On Fedora 25
389-ds-base-1.3.5.15-1
freeipa 4.5.git build

The deadlock occurs 20sec after DS startup. A schedule event (topology plugin) creates a 'iparepltoposegment' entry (thread 36):
ADD "cn=<vm-1>-to-<vm-2>,cn=domain,cn=topology,cn=ipa,cn=etc,<suffix>"

at the same time a new index (numsubordinates) on IPACA is created (thread 35):
cn=numsubordinates,cn=index,cn=ipaca,cn=ldbm database,cn=plugins,cn=config
objectclass: top
objectclass: nsIndex
cn: numsubordinates
nsSystemIndex: true
nsIndexType: pres

The problem is that thread 36 hold DB locks before attempting to acquire schema-compat map lock (to find groups owning the index).

At the same time thread 35 that holds schema-compat map lock does an internal search to retrieve groups owning the segment.
-b "cn=groups,cn=accounts,<suffix>" "member=cn=numsubordinates,cn=index,cn=ipaca,cn=ldbm database,cn=plugins,cn=config"

So the deadlock comes from locks taken in the opposite order.


Details about the deadlock threads

    30 dd=42 locks held 0    write locks 0    pid/thread 23284/140663341352704 (0x7feebc7e8700) flags 0    priority 100           
    30 READ          1 WAIT    userRoot/objectclass.db   page          7
8000006e dd= 1 locks held 41   write locks 34   pid/thread 23284/140663332960000   (0x7feebbfe7700) flags 0    priority 100
8000006e WRITE         1 HELD    userRoot/entryrdn.db      page         27
8000006e WRITE         2 HELD    userRoot/objectclass.db   page          7

Thread 36 (Thread 0x7feebbfe7700 (LWP 23321)):
#0  0x00007feef0e777f8 in futex_wait (private=<optimized out>, expected=0, futex_word=0x5653ed5549ec) at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1  futex_wait_simple (private=<optimized out>, expected=0, futex_word=0x5653ed5549ec) at ../sysdeps/nptl/futex-internal.h:135
#2  __pthread_rwlock_wrlock_slow (rwlock=0x5653ed5549e0) at pthread_rwlock_wrlock.c:67
#3  0x00007feef0e779d8 in __GI___pthread_rwlock_wrlock (rwlock=<optimized out>) at pthread_rwlock_wrlock.c:124
#4  0x00007feef3338aea in slapi_rwlock_wrlock (rwlock=<optimized out>) at ldap/servers/slapd/slapi2nspr.c:238
#5  0x00007feee3e65688 in wrap_rwlock_wrlock (rwlock=<optimized out>) at wrap.c:328
#6  0x00007feee3e6498c in map_wrlock () at map.c:1230
#7  0x00007feee3e55fbe in backend_shr_add_cb (pb=0x7feebbfe6a70) at back-shr.c:1925
#8  backend_shr_internal_post_add_cb (pb=0x7feebbfe6a70) at back-shr.c:1981
#9  0x00007feee3e56197 in backend_shr_betxn_post_add_cb (pb=0x7feebbfe6a70) at back-shr.c:1965
#10 0x00007feef3314fd9 in plugin_call_func (list=0x5653ed556f50, operation=operation@entry=560, pb=pb@entry=0x7feebbfe6a70, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049
#11 0x00007feef3315244 in plugin_call_list (pb=0x7feebbfe6a70, operation=560, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993
#12 plugin_call_plugins (pb=pb@entry=0x7feebbfe6a70, whichfunction=whichfunction@entry=560) at ldap/servers/slapd/plugin.c:445
#13 0x00007feee5e0b6d7 in ldbm_back_add (pb=0x7feebbfe6a70) at ldap/servers/slapd/back-ldbm/ldbm_add.c:1150
#14 0x00007feef32b7f49 in op_shared_add (pb=pb@entry=0x7feebbfe6a70) at ldap/servers/slapd/add.c:699
#15 0x00007feef32b9228 in do_add (pb=pb@entry=0x7feebbfe6a70) at ldap/servers/slapd/add.c:226
#16 0x00005653eccfe310 in connection_dispatch_operation (pb=0x7feebbfe6a70, op=0x5653ed8b64a0, conn=0x7feed4705b78) at ldap/servers/slapd/connection.c:612
#17 connection_threadmain () at ldap/servers/slapd/connection.c:1759
#18 0x00007feef14d55bc in _pt_root (arg=0x5653ed6610f0) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#19 0x00007feef0e726ca in start_thread (arg=0x7feebbfe7700) at pthread_create.c:333
#20 0x00007feef0bacf7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105




Thread 35 (Thread 0x7feebc7e8700 (LWP 23320)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007feeea281743 in __db_pthread_mutex_condwait (env=0x5653ed576f70, mutex=140664220648470, timespec=0x0, mutexp=<optimized out>) at ../../src/mutex/mut_pthread.c:321
#2  __db_hybrid_mutex_suspend (env=env@entry=0x5653ed576f70, mutex=mutex@entry=2332, timespec=timespec@entry=0x0, exclusive=exclusive@entry=1) at ../../src/mutex/mut_pthread.c:577
#3  0x00007feeea280b1f in __db_tas_mutex_lock_int (nowait=0, timeout=0, mutex=2332, env=0x5653ed576f70) at ../../src/mutex/mut_tas.c:255
#4  __db_tas_mutex_lock (env=env@entry=0x5653ed576f70, mutex=2332, timeout=timeout@entry=0) at ../../src/mutex/mut_tas.c:286
#5  0x00007feeea32b5f0 in __lock_get_internal (lt=lt@entry=0x5653ed57c9b0, sh_locker=sh_locker@entry=0x7feedffb09c8, flags=flags@entry=0, obj=<optimized out>, lock_mode=<optimized out>, timeout=timeout@entry=0, lock=<optimized out>) at ../../src/lock/lock.c:989
#6  0x00007feeea32c112 in __lock_get (env=env@entry=0x5653ed576f70, locker=0x7feedffb09c8, flags=0, obj=obj@entry=0x5653ed574a40, lock_mode=lock_mode@entry=DB_LOCK_READ, lock=lock@entry=0x7feebc7d6830) at ../../src/lock/lock.c:469
#7  0x00007feeea357fbf in __db_lget (dbc=dbc@entry=0x5653ed574950, action=action@entry=0, pgno=7, mode=mode@entry=DB_LOCK_READ, lkflags=lkflags@entry=0, lockp=lockp@entry=0x7feebc7d6830) at ../../src/db/db_meta.c:1257
#8  0x00007feeea29e9e1 in __bam_search (dbc=dbc@entry=0x5653ed574950, root_pgno=1, root_pgno@entry=0, key=key@entry=0x7feebc7d6b00, flags=1409, slevel=slevel@entry=1, recnop=recnop@entry=0x0, exactp=0x7feebc7d6964) at ../../src/btree/bt_search.c:723
#9  0x00007feeea28963b in __bamc_search (dbc=dbc@entry=0x5653ed574950, root_pgno=root_pgno@entry=0, key=0x7feebc7d6b00, flags=<optimized out>, exactp=<optimized out>) at ../../src/btree/bt_cursor.c:2804
#10 0x00007feeea28b79f in __bamc_get (dbc=0x5653ed574950, key=<optimized out>, data=<optimized out>, flags=<optimized out>, pgnop=0x7feebc7d69f4) at ../../src/btree/bt_cursor.c:1099
#11 0x00007feeea344aa3 in __dbc_iget (dbc=0x5653ed575cf0, key=0x7feebc7d6b00, data=0x7feebc7d6b30, flags=26) at ../../src/db/db_cam.c:952
#12 0x00007feeea34533d in __dbc_get (dbc=dbc@entry=0x5653ed575cf0, key=key@entry=0x7feebc7d6b00, data=data@entry=0x7feebc7d6b30, flags=flags@entry=2074) at ../../src/db/db_cam.c:770
#13 0x00007feeea353d46 in __dbc_get_pp (dbc=0x5653ed575cf0, key=0x7feebc7d6b00, data=0x7feebc7d6b30, flags=2074) at ../../src/db/db_iface.c:2361
#14 0x00007feee5df6272 in idl_new_fetch (be=0x5653ed63cd00, db=0x5653ed70fab0, inkey=0x7feebc7d8c90, txn=0x0, a=0x5653ed70b2d0, flag_err=0x7feebc7df9ac, allidslimit=4000) at ldap/servers/slapd/back-ldbm/idl_new.c:202
#15 0x00007feee5df5f15 in idl_fetch_ext (be=be@entry=0x5653ed63cd00, db=<optimized out>, key=key@entry=0x7feebc7d8c90, txn=txn@entry=0x0, a=<optimized out>, err=err@entry=0x7feebc7df9ac, allidslimit=4000) at ldap/servers/slapd/back-ldbm/idl_shim.c:101
#16 0x00007feee5e04d70 in index_read_ext_allids (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, type=type@entry=0x7fee5c0897b0 "objectclass", indextype=indextype@entry=0x7feee5e4d13f "eq", val=<optimized out>, txn=txn@entry=0x7feebc7dcf70, err=0x7feebc7df9ac, unindexed=0x7feebc7dcf64, allidslimit=4000) at ldap/servers/slapd/back-ldbm/index.c:1028
#17 0x00007feee5deecbd in keys2idl (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, type=0x7fee5c0897b0 "objectclass", indextype=indextype@entry=0x7feee5e4d13f "eq", ivals=0x7feebc7dd050, err=err@entry=0x7feebc7df9ac, unindexed=0x7feebc7dcf64, txn=0x7feebc7dcf70, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:986
#18 0x00007feee5def55a in ava_candidates (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, f=f@entry=0x7fee5c08a320, ftype=<optimized out>, err=0x7feebc7df9ac, allidslimit=4000, range=0, nextf=0x0) at ldap/servers/slapd/back-ldbm/filterindex.c:288
#19 0x00007feee5defb19 in filter_candidates_ext (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, base=base@entry=0x7fee5c081930 "cn=groups,cn=accounts,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", f=f@entry=0x7fee5c08a320, nextf=nextf@entry=0x0, range=range@entry=0, err=0x7feebc7df9ac, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:111
#20 0x00007feee5df0acf in list_candidates (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, base=base@entry=0x7fee5c081930 "cn=groups,cn=accounts,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", flist=flist@entry=0x7fee5c088c60, ftype=161, err=0x7feebc7df9ac, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:817
#21 0x00007feee5defa8a in filter_candidates_ext (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, base=base@entry=0x7fee5c081930 "cn=groups,cn=accounts,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", f=f@entry=0x7fee5c088c60, nextf=nextf@entry=0x0, range=range@entry=0, err=0x7feebc7df9ac, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:144
#22 0x00007feee5df0acf in list_candidates (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, base=base@entry=0x7fee5c081930 "cn=groups,cn=accounts,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", flist=flist@entry=0x7fee5c0811c0, ftype=160, err=0x7feebc7df9ac, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:817
#23 0x00007feee5defa8a in filter_candidates_ext (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, base=base@entry=0x7fee5c081930 "cn=groups,cn=accounts,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", f=0x7fee5c0811c0, nextf=nextf@entry=0x0, range=range@entry=0, err=0x7feebc7df9ac, allidslimit=4000) at ldap/servers/slapd/back-ldbm/filterindex.c:144
#24 0x00007feee5df109f in filter_candidates (pb=pb@entry=0x7fee5c08ad70, be=be@entry=0x5653ed63cd00, base=base@entry=0x7fee5c081930 "cn=groups,cn=accounts,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", f=<optimized out>, nextf=nextf@entry=0x0, range=range@entry=0, err=0x7feebc7df9ac) at ldap/servers/slapd/back-ldbm/filterindex.c:175
#25 0x00007feee5e2db2b in onelevel_candidates (err=0x7feebc7df9ac, lookup_returned_allidsp=0x7feebc7df99c, managedsait=<optimized out>, filter=<optimized out>, e=0x5653ed8ad3b0, base=0x7fee5c081930 "cn=groups,cn=accounts,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", be=0x5653ed63cd00, pb=0x7fee5c08ad70) at ldap/servers/slapd/back-ldbm/ldbm_search.c:1143
#26 build_candidate_list (candidates=0x7feebc7df9d8, lookup_returned_allidsp=0x7feebc7df99c, scope=1, base=0x7fee5c081930 "cn=groups,cn=accounts,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", e=<optimized out>, be=0x5653ed63cd00, pb=0x7fee5c08ad70) at ldap/servers/slapd/back-ldbm/ldbm_search.c:1010
#27 ldbm_back_search (pb=0x7fee5c08ad70) at ldap/servers/slapd/back-ldbm/ldbm_search.c:657
#28 0x00007feef3309650 in op_shared_search (pb=pb@entry=0x7fee5c08ad70, send_result=send_result@entry=1) at ldap/servers/slapd/opshared.c:807
#29 0x00007feef331a23e in search_internal_callback_pb (pb=pb@entry=0x7fee5c08ad70, callback_data=callback_data@entry=0x7feebc7e4340, prc=prc@entry=0x0, psec=psec@entry=0x7feee3e55480 <backend_shr_note_entry_sdn_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:783
#30 0x00007feef331a771 in slapi_search_internal_callback_pb (pb=pb@entry=0x7fee5c08ad70, callback_data=callback_data@entry=0x7feebc7e4340, prc=prc@entry=0x0, psec=psec@entry=0x7feee3e55480 <backend_shr_note_entry_sdn_cb>, prec=prec@entry=0x0) at ldap/servers/slapd/plugin_internal_op.c:564
#31 0x00007feee3e56ea9 in backend_shr_update_references_cb (group=0x7fee50007f50 "cn=compat,dc=dom-184,dc=abc,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com", set=0x7fee5000a980 "cn=groups", flag=<optimized out>, backend_data=0x7fee50006b80, cbdata_ptr=0x7feebc7e4430) at back-shr.c:1539
#32 0x00007feee3e6344f in map_data_foreach_map (state=state@entry=0x5653ed555530, domain_name=domain_name@entry=0x0, fn=fn@entry=0x7feee3e56550 <backend_shr_update_references_cb>, cbdata=cbdata@entry=0x7feebc7e4430) at map.c:347
#33 0x00007feee3e54e6b in backend_shr_update_references (state=0x5653ed555530, pb=pb@entry=0x7fee5c005150, e=<optimized out>, mods=mods@entry=0x0, modlist=modlist@entry=0x0) at back-shr.c:1825
#34 0x00007feee3e5603b in backend_shr_add_cb (pb=0x7fee5c005150) at back-shr.c:1953
#35 backend_shr_internal_post_add_cb (pb=0x7fee5c005150) at back-shr.c:1981
#36 0x00007feee3e56197 in backend_shr_betxn_post_add_cb (pb=0x7fee5c005150) at back-shr.c:1965
#37 0x00007feef3314fd9 in plugin_call_func (list=0x5653ed556f50, operation=operation@entry=560, pb=pb@entry=0x7fee5c005150, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049
#38 0x00007feef3315244 in plugin_call_list (pb=0x7fee5c005150, operation=560, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993
#39 plugin_call_plugins (pb=pb@entry=0x7fee5c005150, whichfunction=whichfunction@entry=560) at ldap/servers/slapd/plugin.c:445
#40 0x00007feef32cf94a in dse_add (pb=0x7fee5c005150) at ldap/servers/slapd/dse.c:2403
#41 0x00007feef32b7f49 in op_shared_add (pb=pb@entry=0x7fee5c005150) at ldap/servers/slapd/add.c:699
#42 0x00007feef32b8723 in add_internal_pb (pb=pb@entry=0x7fee5c005150) at ldap/servers/slapd/add.c:408
#43 0x00007feef32b942e in slapi_add_internal_pb (pb=pb@entry=0x7fee5c005150) at ldap/servers/slapd/add.c:330
#44 0x00007feee5e14e12 in ldbm_config_add_dse_entries (li=li@entry=0x5653ed519220, entries=entries@entry=0x7feebc7e6f00, string1=string1@entry=0x7fee5c087c80 "dn: cn=numsubordinates,cn=index,cn=ipaca,cn=ldbm database,cn=plugins,cn=config\nobjectclass: top\nobjectclass: nsIndex\ncn: numsubordinates\nnsSystemIndex: true\nnsIndexType: pres\n", string2=string2@entry=0x0, string3=string3@entry=0x0, flags=flags@entry=1) at ldap/servers/slapd/back-ldbm/ldbm_config.c:73
#45 0x00007feee5e14f21 in ldbm_config_add_dse_entry (li=li@entry=0x5653ed519220, entry=entry@entry=0x7fee5c087c80 "dn: cn=numsubordinates,cn=index,cn=ipaca,cn=ldbm database,cn=plugins,cn=config\nobjectclass: top\nobjectclass: nsIndex\ncn: numsubordinates\nnsSystemIndex: true\nnsIndexType: pres\n", flags=flags@entry=1) at ldap/servers/slapd/back-ldbm/ldbm_config.c:94
#46 0x00007feee5e22448 in ldbm_instance_config_add_index_entry (inst=inst@entry=0x7fee5c004840, e=e@entry=0x7fee5c0851d0, flags=flags@entry=1) at ldap/servers/slapd/back-ldbm/ldbm_index_config.c:325
#47 0x00007feee5e09465 in ldbm_instance_create_default_indexes (be=be@entry=0x7fee5c004fc0) at ldap/servers/slapd/back-ldbm/instance.c:212
#48 0x00007feee5e23d39 in ldbm_instance_generate (li=li@entry=0x5653ed519220, instance_name=<optimized out>, ret_be=ret_be@entry=0x7feebc7e7008) at ldap/servers/slapd/back-ldbm/ldbm_instance_config.c:866
#49 0x00007feee5e23f84 in ldbm_instance_postadd_instance_entry_callback (pb=<optimized out>, entryBefore=<optimized out>, entryAfter=<optimized out>, returncode=<optimized out>, returntext=<optimized out>, arg=0x5653ed519220) at ldap/servers/slapd/back-ldbm/ldbm_instance_config.c:903
#50 0x00007feef32cce09 in dse_call_callback (pb=pb@entry=0x7feebc7e7a70, operation=operation@entry=16, flags=flags@entry=2, entryBefore=0x7fee5c000fa0, entryAfter=entryAfter@entry=0x0, returncode=0x7feebc7e70c4, returntext=0x7feebc7e7120 "", pdse=<optimized out>) at ldap/servers/slapd/dse.c:2634
#51 0x00007feef32cfec1 in dse_add (pb=0x7feebc7e7a70) at ldap/servers/slapd/dse.c:2384
#52 0x00007feef32b7f49 in op_shared_add (pb=pb@entry=0x7feebc7e7a70) at ldap/servers/slapd/add.c:699
#53 0x00007feef32b9228 in do_add (pb=pb@entry=0x7feebc7e7a70) at ldap/servers/slapd/add.c:226
#54 0x00005653eccfe310 in connection_dispatch_operation (pb=0x7feebc7e7a70, op=0x5653ed9ab950, conn=0x7feed4705a10) at ldap/servers/slapd/connection.c:612
#55 connection_threadmain () at ldap/servers/slapd/connection.c:1759
#56 0x00007feef14d55bc in _pt_root (arg=0x5653ed660de0) at ../../../nspr/pr/src/pthreads/ptthread.c:216
#57 0x00007feef0e726ca in start_thread (arg=0x7feebc7e8700) at pthread_create.c:333
#58 0x00007feef0bacf7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Note that /usr/share/ipa/updates/10-schema_compat.update, restrict schema compat to "cn=Schema Compatibility,cn=plugins,cn=config" and ignore updates under 'dna' configuration.
dn: cn=sudoers,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-restrict-subtree: $SUFFIX
add: schema-compat-restrict-subtree: cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=dna,cn=ipa,cn=etc,$SUFFIX

 dn: cn=ng,cn=Schema Compatibility,cn=plugins,cn=config
 add: schema-compat-restrict-subtree: $SUFFIX
 add: schema-compat-restrict-subtree: cn=Schema Compatibility,cn=plugins,cn=config
 add: schema-compat-ignore-subtree: cn=dna,cn=ipa,cn=etc,$SUFFIX

 dn: cn=computers, cn=Schema Compatibility, cn=plugins, cn=config
 add: schema-compat-restrict-subtree: $SUFFIX
 add: schema-compat-restrict-subtree: cn=Schema Compatibility,cn=plugins,cn=config
 add: schema-compat-ignore-subtree: cn=dna,cn=ipa,cn=etc,$SUFFIX

 dn: cn=users,cn=Schema Compatibility,cn=plugins,cn=config
 add: schema-compat-restrict-subtree: $SUFFIX
 add: schema-compat-restrict-subtree: cn=Schema Compatibility,cn=plugins,cn=config
 add: schema-compat-ignore-subtree: cn=dna,cn=ipa,cn=etc,$SUFFIX

 dn: cn=groups,cn=Schema Compatibility,cn=plugins,cn=config
 add: schema-compat-restrict-subtree: $SUFFIX
 add: schema-compat-restrict-subtree: cn=Schema Compatibility,cn=plugins,cn=config
 add: schema-compat-ignore-subtree: cn=dna,cn=ipa,cn=etc,$SUFFIX

But in dse.ldif, those restrict/ignore were missing.
This is like the update script was not applied !

The install (replica-install) was looking like:
2017-03-24T04:45:08Z DEBUG [1/4]: stopping ntpd
2017-03-24T04:45:08Z DEBUG [2/4]: writing configuration
...
2017-03-24T04:45:34Z DEBUG [41/45]: enabling compatibility plugin
2017-03-24T04:45:37Z DEBUG [42/45]: activating sidgen plugin
2017-03-24T04:45:37Z DEBUG [43/45]: activating extdom plugin
2017-03-24T04:45:37Z DEBUG [44/45]: tuning directory server
2017-03-24T04:45:39Z DEBUG [45/45]: configuring directory to start on boot
...
2017-03-24T04:45:56Z DEBUG Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
2017-03-24T04:45:56Z DEBUG [1/28]: creating certificate server user
2017-03-24T04:45:56Z DEBUG [2/28]: creating certificate server db

The hang occurred before 'Applying LDAP updates'. So 'schema-compat.update' was not applied and schema-compat caught any req (without restrict/ignore).

-> first idea is to move 'enabling compatibility plugin' outside of 'Configuring directory server (dirsrv)' and at the end of the install

-> second idea is too add those definitions 
dn: cn=sudoers,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

dn: cn=ng,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

dn: cn=computers, cn=Schema Compatibility, cn=plugins, cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

dn: cn=users,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

dn: cn=groups,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

The install (replica-install) was looking like:
2017-03-24T04:45:08Z DEBUG [1/4]: stopping ntpd
2017-03-24T04:45:08Z DEBUG [2/4]: writing configuration
...
2017-03-24T04:45:34Z DEBUG [41/45]: enabling compatibility plugin
2017-03-24T04:45:37Z DEBUG [42/45]: activating sidgen plugin
2017-03-24T04:45:37Z DEBUG [43/45]: activating extdom plugin
2017-03-24T04:45:37Z DEBUG [44/45]: tuning directory server
2017-03-24T04:45:39Z DEBUG [45/45]: configuring directory to start on boot
...
2017-03-24T04:45:56Z DEBUG Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
2017-03-24T04:45:56Z DEBUG [1/28]: creating certificate server user
2017-03-24T04:45:56Z DEBUG [2/28]: creating certificate server db

The hang occurred before 'Applying LDAP updates'. So 'schema-compat.update' was not applied and schema-compat caught any req (without restrict/ignore).

-> first idea is to move 'enabling compatibility plugin' outside of 'Configuring directory server (dirsrv)' and at the end of the install

-> second idea is too add those definitions 
dn: cn=sudoers,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

dn: cn=ng,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

dn: cn=computers, cn=Schema Compatibility, cn=plugins, cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

dn: cn=users,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

dn: cn=groups,cn=Schema Compatibility,cn=plugins,cn=config
add: schema-compat-ignore-subtree: cn=topology,cn=ipa,cn=etc,$SUFFIX

I think we should remove "2017-03-24T04:45:34Z DEBUG [41/45]: enabling compatibility plugin" step from general directory server instance setup and move it into explicit configuration after everything is done -- at least after applying updates.

We also need to make sure we restrict cn=config watching to Schema Compat configuration subtree.

looking into this I have two commments:
1] in the problem description in thread 36 I don't see any involvement of the topology plugin, the stack:

#15 0x00007feef32b9228 in do_add (pb=pb@entry=0x7feebbfe6a70) at ldap/servers/slapd/add.c:226
#16 0x00005653eccfe310 in connection_dispatch_operation (pb=0x7feebbfe6a70, op=0x5653ed8b64a0, conn=0x7feed4705b78) at ldap/servers/slapd/connection.c:612

looks like an external ADD operation

2] I agree with Thierry, that the cn=topology subtree should be removed from the scope of the compat plugin

Metadata Update from @pvoborni:
- Issue priority set to: blocker
- Issue set to the milestone: FreeIPA 4.5.1

7 years ago

Metadata Update from @stlaz:
- Issue assigned to stlaz

7 years ago

master:

  • 0c0af8c compat-manage: behave the same for all users
  • ddbbb1c Move the compat plugin setup at the end of install
  • 6456159 compat: ignore cn=topology,cn=ipa,cn=etc subtree
  • 68c8ddf compat plugin: Update link to slapi-nis project

ipa-4-5:

  • 4fa7718 compat-manage: behave the same for all users
  • 7364c13 Move the compat plugin setup at the end of install
  • e691877 compat: ignore cn=topology,cn=ipa,cn=etc subtree
  • efe0960 compat plugin: Update link to slapi-nis project

Metadata Update from @mbasti:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

7 years ago

Metadata Update from @pvoborni:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1444947

7 years ago

Metadata Update from @pvoborni:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1444947

7 years ago

Login to comment on this ticket.

Metadata