Replica and original master installed with DNS and CA.
freeipa-server-3.3.90GIT278c87c-0.fc19.x86_64
Reproduction:
With working replication (new user is replicated):
$ ipa-replica-manage del vm-175.idm.lab.eng.brq.redhat.com Deleting a master is irreversible. To reconnect to the remote master you will need to prepare a new replica file and re-install. Continue to delete? [no]: yes No RUV records found.
$ ipa-replica-manage del vm-175.idm.lab.eng.brq.redhat.com -cf No RUV records found.
With replica dead (turned off)
$ ipa-replica-manage del vm-175.idm.lab.eng.brq.redhat.com -cf Connection to 'vm-175.idm.lab.eng.brq.redhat.com' failed: Forcing removal of vm-175.idm.lab.eng.brq.redhat.com Skipping calculation to determine if one or more masters would be orphaned. No RUV records found.
Replication is still working after running those commands.
Can you attach the output of these searches:
$ ldapsearch -x -W -D 'cn=directory manager' -b $BASEDN -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv
$ ldapsearch -x -W -D 'cn=directory manager' -b 'cn=mapping tree,cn=config'
$ldapsearch -x -W -D 'cn=directory manager' -b 'dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com' -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com> with scope oneLevel # filter: (&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone)) # requesting: nsds50ruv # # search result search: 2 result: 0 Success # numResponses: 1 $ ldapsearch -x -W -D 'cn=directory manager' -b 'cn=mapping tree,cn=config' Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=mapping tree,cn=config> with scope subtree # filter: (objectclass=*) # requesting: ALL # # mapping tree, config dn: cn=mapping tree,cn=config objectClass: top objectClass: extensibleObject cn: mapping tree # dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom, mapping tree, config dn: cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom,cn=m apping tree,cn=config objectClass: top objectClass: extensibleObject objectClass: nsMappingTree cn: dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com cn: "dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com" nsslapd-state: backend nsslapd-backend: userRoot nsslapd-referral: ldap://vm-175.idm.lab.eng.brq.redhat.com:389/dc%3Didm%2Cdc%3 Dlab%2Cdc%3Deng%2Cdc%3Dbrq%2Cdc%3Dredhat%2Cdc%3Dcom # o\3Dipaca, mapping tree, config dn: cn=o\3Dipaca,cn=mapping tree,cn=config objectClass: top objectClass: extensibleObject objectClass: nsMappingTree cn: o=ipaca nsslapd-backend: ipaca nsslapd-state: Backend nsslapd-referral: ldap://vm-175.idm.lab.eng.brq.redhat.com:389/o%3Dipaca # replica, dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom , mapping tree, config dn: cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc \3Dcom,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 objectClass: top objectClass: nsds5replica objectClass: extensibleobject nsDS5ReplicaType: 3 nsDS5ReplicaRoot: dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com nsds5ReplicaLegacyConsumer: off nsDS5ReplicaId: 4 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDN: krbprincipalname=ldap/vm-175.idm.lab.eng.brq.redhat.com@ID M.LAB.ENG.BRQ.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=eng,dc=brq, dc=redhat,dc=com nsState:: BAAAAAAAAAAHVhdSAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAA== nsDS5ReplicaName: 83087503-0be611e3-bc16f0d2-68492a22 nsds5ReplicaChangeCount: 98 nsds5replicareapactive: 0 # replica, o\3Dipaca, mapping tree, config dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-vm-175.idm.lab.eng .brq.redhat.com-pki-tomcat,ou=csusers,cn=config cn: replica nsDS5ReplicaId: 96 nsDS5Flags: 1 nsState:: YAAAAAAAAAB9XxdSAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAA== nsDS5ReplicaName: a6cbbb07-0be611e3-bc16f0d2-68492a22 nsds5ReplicaChangeCount: 15 nsds5replicareapactive: 0 # meTovm-175.idm.lab.eng.brq.redhat.com, replica, dc\3Didm\2Cdc\3Dlab\2Cdc\3D eng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom, mapping tree, config dn: cn=meTovm-175.idm.lab.eng.brq.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab \2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping tree,cn=config cn: meTovm-175.idm.lab.eng.brq.redhat.com objectClass: nsds5replicationagreement objectClass: top nsDS5ReplicaTransportInfo: LDAP description: me to vm-175.idm.lab.eng.brq.redhat.com nsDS5ReplicaRoot: dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com nsDS5ReplicaHost: vm-175.idm.lab.eng.brq.redhat.com nsds5replicaTimeout: 120 nsDS5ReplicaPort: 389 nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicaBindMethod: SASL/GSSAPI nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 20130823123607Z nsds5replicaLastUpdateEnd: 20130823123607Z nsds5replicaChangesSentSinceStartup:: NDo0MC8wIA== nsds5replicaLastUpdateStatus: 0 Replica acquired successfully: Incremental upd ate succeeded nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 20130823112433Z nsds5replicaLastInitEnd: 20130823112435Z nsds5replicaLastInitStatus: 0 Total update succeeded # search result search: 2 result: 0 Success # numResponses: 7 # numEntries: 6
Wow, I don't see any RUV data at all. So I think there are things to do here:
Figure out why there is no RUV data and what it means/implies.
We should not prevent deleting a replica because there is no RUV found.
Adding Mark and Rich to comment on the RUV part.
In regards to:
$ldapsearch -x -W -D 'cn=directory manager' -b 'dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com' -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv
This should at least be returning an entry if replication is enabled. Is there anything under: o=ipaca?
I do not see an issue with deleting a replica if no RUV information is found, as long as you know it is in fact a replica.
Yes, there is.
$ ldapsearch -x -W -D 'cn=directory manager' -b 'o=ipaca' -s one <snip> # numResponses: 10 # numEntries: 9 $ ldapsearch -x -W -D 'cn=directory manager' -b 'o=ipaca' -s subtree <snip> # numResponses: 76 # numEntries: 75
Sorry I wanted you to run:
$ldapsearch -x -W -D 'cn=directory manager' -b 'o=ipaca' -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv
Nothing there.
ldapsearch -x -W -D 'cn=directory manager' -b 'o=ipaca' -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv Enter LDAP Password: # extended LDIF # # LDAPv3 # base <o=ipaca> with scope oneLevel # filter: (&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone)) # requesting: nsds50ruv # # search result search: 2 result: 0 Success # numResponses: 1
Sorry I'm not sure how this can happen. Once replication is enabled, the root tombstone entry is created. Even is you remove replication, that entry still exists. From the outside looking in, it looks like replication was never set up, except that you can see the config entries. But without any ruv data I don't know how replication could still be working.
If you restart the DS, are there any errors, and is the RUV data still missing? Also, what version of DS is this?
Latest in updates: 389-ds-base-1.3.1.6-1.fc19.x86_64
I don't see any errors after restart. RUV data still missing.
$ sudo systemctl restart dirsrv@IDM-LAB-ENG-BRQ-REDHAT-COM.service $ sudo tail -255 /var/log/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/errors <snip> [23/Aug/2013:18:53:00 +0200] - slapd shutting down - signaling operation threads [23/Aug/2013:18:53:00 +0200] - slapd shutting down - closing down internal subsystems and plugins [23/Aug/2013:18:53:00 +0200] - Waiting for 4 database threads to stop [23/Aug/2013:18:53:01 +0200] - All database threads now stopped [23/Aug/2013:18:53:01 +0200] - slapd stopped. [23/Aug/2013:18:53:02 +0200] - 389-Directory/1.3.1.6 B2013.213.2156 starting up [23/Aug/2013:18:53:02 +0200] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com [23/Aug/2013:18:53:02 +0200] schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com [23/Aug/2013:18:53:02 +0200] schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com [23/Aug/2013:18:53:02 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com--no CoS Templates found, which should be added before the CoS Definition. [23/Aug/2013:18:53:02 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com--no CoS Templates found, which should be added before the CoS Definition. [23/Aug/2013:18:53:02 +0200] - slapd started. Listening on All Interfaces port 389 for LDAP requests [23/Aug/2013:18:53:02 +0200] - Listening on All Interfaces port 636 for LDAPS requests [23/Aug/2013:18:53:02 +0200] - Listening on /var/run/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM.socket for LDAPI requests
NSMMReplicationPlugin errors, note that replica was shutdown once:
$ sudo grep NSMMReplicationPlugin /var/log/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/errors [23/Aug/2013:13:24:33 +0200] NSMMReplicationPlugin - agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica. [23/Aug/2013:13:24:33 +0200] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389)". [23/Aug/2013:13:24:37 +0200] NSMMReplicationPlugin - Finished total update of replica "agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389)". Sent 245 entries. [23/Aug/2013:13:25:39 +0200] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389)". [23/Aug/2013:13:25:43 +0200] NSMMReplicationPlugin - Finished total update of replica "agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389)". Sent 70 entries. [23/Aug/2013:13:26:21 +0200] NSMMReplicationPlugin - agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389): Unable to receive the response for a startReplication extended operation to consumer (Can't contact LDAP server). Will retry later. [23/Aug/2013:13:26:24 +0200] NSMMReplicationPlugin - agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389): Replication bind with SIMPLE auth resumed [23/Aug/2013:13:42:31 +0200] NSMMReplicationPlugin - agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) () [23/Aug/2013:13:53:02 +0200] NSMMReplicationPlugin - agmt_delete: begin [23/Aug/2013:13:53:02 +0200] NSMMReplicationPlugin - Warning: incremental protocol for replica "agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389)" did not shut down properly. [23/Aug/2013:18:37:11 +0200] NSMMReplicationPlugin - agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389): Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () [23/Aug/2013:18:37:14 +0200] NSMMReplicationPlugin - agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389): Replication bind with GSSAPI auth resumed
I looked into the replica box, and it seems to be working fine. The problem appears to be with your scope. Remove the "-s one" from the search:
ldapsearch -xLLL -W -D 'cn=directory manager' -b 'dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com' "(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))" dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com objectClass: top objectClass: nsTombstone objectClass: extensibleobject nsds50ruv: {replicageneration} 52174670000000040000 nsds50ruv: {replica 3 ldap://vm-175.idm.lab.eng.brq.redhat.com:389} 52174678000000030000 5217b27b000100030000 nsds50ruv: {replica 4 ldap://vm-161.idm.lab.eng.brq.redhat.com:389} 521746cb000000040000 5217938d000200040000 dc: idm
In the repl agreements the RUV data is there as well:
dn: cn=meTovm-161.idm.lab.eng.brq.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping tree,cn=config cn: meTovm-161.idm.lab.eng.brq.redhat.com objectClass: nsds5replicationagreement objectClass: top nsDS5ReplicaTransportInfo: LDAP description: me to vm-161.idm.lab.eng.brq.redhat.com nsDS5ReplicaRoot: dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com nsDS5ReplicaHost: vm-161.idm.lab.eng.brq.redhat.com nsds5replicaTimeout: 120 nsDS5ReplicaPort: 389 nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicaBindMethod: SASL/GSSAPI creatorsName: cn=directory manager modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=config createTimestamp: 20130823112433Z modifyTimestamp: 20130823114107Z nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount nsds50ruv: {replicageneration} 52174670000000040000 nsds50ruv: {replica 4 ldap://vm-161.idm.lab.eng.brq.redhat.com:389} 521746cb000000040000 52174767000600040000 nsds50ruv: {replica 3 ldap://vm-175.idm.lab.eng.brq.redhat.com:389} 52174678000000030000 52174772000000030000 nsruvReplicaLastModified: {replica 4 ldap://vm-161.idm.lab.eng.brq.redhat.com:389} 00000000 nsruvReplicaLastModified: {replica 3 ldap://vm-175.idm.lab.eng.brq.redhat.com:389} 00000000 nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName internalModifyTimestamp
Everything appears normal on the DS side. However, I'm not sure what/where "ipa-replica-manage del" is looking for the RUVs. Maybe it too is using a scoped search?
Yes, ipa-replica-manage uses a scoped search.
What I don't understand is how this entry was missed as it seems to be within scope.
Replying to [comment:11 rcritten]:
Yes, ipa-replica-manage uses a scoped search. What I don't understand is how this entry was missed as it seems to be within scope.
This appears to be a design limitation. Doing a one level tree search uses the parentid index, but since this is a tombstone entry it is not indexed, and it does not contain the parentid operational attribute, and hence not picked up for scope "one,base" searches. In fact, any search for a tombstone that is "base,one" scoped will fail to return anything. "subtree" scoped searches are not internally modified, just "one" and "base" level searches have their filters modified internally(where its adds a filter component for the parentid).
There are cases when looking up special entries where the search scope is not intuitive. Like for root DSE searches, you have to use "-b base" or else the entry is not returned, and for tombstones use must use a subtree scoped search "-s sub".
Ticket has been cloned to Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1001662
master: f312d72[[BR]] ipa-3-3: 93b314d
Linked to Bugzilla bug: https://bugzilla.redhat.com/show_bug.cgi?id=1005448 (Red Hat Enterprise Linux 6)
Metadata Update from @pvoborni: - Issue assigned to pvoborni - Issue set to the milestone: FreeIPA 3.3.x - 2013/09 (bug fixing)
Login to comment on this ticket.