#3876 Unable to remove replica
Closed: Fixed None Opened 10 years ago by pvoborni.

Replica and original master installed with DNS and CA.

freeipa-server-3.3.90GIT278c87c-0.fc19.x86_64

Reproduction:

  • With working replication (new user is replicated):

    $ ipa-replica-manage del vm-175.idm.lab.eng.brq.redhat.com
    Deleting a master is irreversible.
    To reconnect to the remote master you will need to prepare a new replica file
    and re-install.
    Continue to delete? [no]: yes
    No RUV records found.

    $ ipa-replica-manage del vm-175.idm.lab.eng.brq.redhat.com -cf
    No RUV records found.

  • With replica dead (turned off)

    $ ipa-replica-manage del vm-175.idm.lab.eng.brq.redhat.com -cf
    Connection to 'vm-175.idm.lab.eng.brq.redhat.com' failed:
    Forcing removal of vm-175.idm.lab.eng.brq.redhat.com
    Skipping calculation to determine if one or more masters would be orphaned.
    No RUV records found.

Replication is still working after running those commands.


Can you attach the output of these searches:

$ ldapsearch -x -W -D 'cn=directory manager' -b $BASEDN -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv

$ ldapsearch -x -W -D 'cn=directory manager' -b 'cn=mapping tree,cn=config'

$ldapsearch -x -W -D 'cn=directory manager' -b 'dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com' -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com> with scope oneLevel
# filter: (&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))
# requesting: nsds50ruv 
#

# search result
search: 2
result: 0 Success

# numResponses: 1



$ ldapsearch -x -W -D 'cn=directory manager' -b 'cn=mapping tree,cn=config'
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <cn=mapping tree,cn=config> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# mapping tree, config
dn: cn=mapping tree,cn=config
objectClass: top
objectClass: extensibleObject
cn: mapping tree

# dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom, mapping
  tree, config
dn: cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom,cn=m
 apping tree,cn=config
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree
cn: dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com
cn: "dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com"
nsslapd-state: backend
nsslapd-backend: userRoot
nsslapd-referral: ldap://vm-175.idm.lab.eng.brq.redhat.com:389/dc%3Didm%2Cdc%3
 Dlab%2Cdc%3Deng%2Cdc%3Dbrq%2Cdc%3Dredhat%2Cdc%3Dcom

# o\3Dipaca, mapping tree, config
dn: cn=o\3Dipaca,cn=mapping tree,cn=config
objectClass: top
objectClass: extensibleObject
objectClass: nsMappingTree
cn: o=ipaca
nsslapd-backend: ipaca
nsslapd-state: Backend
nsslapd-referral: ldap://vm-175.idm.lab.eng.brq.redhat.com:389/o%3Dipaca

# replica, dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom
 , mapping tree, config
dn: cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc
 \3Dcom,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
objectClass: top
objectClass: nsds5replica
objectClass: extensibleobject
nsDS5ReplicaType: 3
nsDS5ReplicaRoot: dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com
nsds5ReplicaLegacyConsumer: off
nsDS5ReplicaId: 4
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaBindDN: krbprincipalname=ldap/vm-175.idm.lab.eng.brq.redhat.com@ID
 M.LAB.ENG.BRQ.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=eng,dc=brq,
 dc=redhat,dc=com
nsState:: BAAAAAAAAAAHVhdSAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAA==
nsDS5ReplicaName: 83087503-0be611e3-bc16f0d2-68492a22
nsds5ReplicaChangeCount: 98
nsds5replicareapactive: 0

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
objectClass: top
objectClass: nsDS5Replica
objectClass: extensibleobject
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaType: 3
nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-vm-175.idm.lab.eng
 .brq.redhat.com-pki-tomcat,ou=csusers,cn=config
cn: replica
nsDS5ReplicaId: 96
nsDS5Flags: 1
nsState:: YAAAAAAAAAB9XxdSAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAA==
nsDS5ReplicaName: a6cbbb07-0be611e3-bc16f0d2-68492a22
nsds5ReplicaChangeCount: 15
nsds5replicareapactive: 0

# meTovm-175.idm.lab.eng.brq.redhat.com, replica, dc\3Didm\2Cdc\3Dlab\2Cdc\3D
 eng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom, mapping tree, config
dn: cn=meTovm-175.idm.lab.eng.brq.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab
 \2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping tree,cn=config
cn: meTovm-175.idm.lab.eng.brq.redhat.com
objectClass: nsds5replicationagreement
objectClass: top
nsDS5ReplicaTransportInfo: LDAP
description: me to vm-175.idm.lab.eng.brq.redhat.com
nsDS5ReplicaRoot: dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com
nsDS5ReplicaHost: vm-175.idm.lab.eng.brq.redhat.com
nsds5replicaTimeout: 120
nsDS5ReplicaPort: 389
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
  entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
 uccessfulauth krblastfailedauth krbloginfailedcount
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 20130823123607Z
nsds5replicaLastUpdateEnd: 20130823123607Z
nsds5replicaChangesSentSinceStartup:: NDo0MC8wIA==
nsds5replicaLastUpdateStatus: 0 Replica acquired successfully: Incremental upd
 ate succeeded
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 20130823112433Z
nsds5replicaLastInitEnd: 20130823112435Z
nsds5replicaLastInitStatus: 0 Total update succeeded

# search result
search: 2
result: 0 Success

# numResponses: 7
# numEntries: 6

Wow, I don't see any RUV data at all. So I think there are things to do here:

  • Figure out why there is no RUV data and what it means/implies.

  • We should not prevent deleting a replica because there is no RUV found.

Adding Mark and Rich to comment on the RUV part.

In regards to:

$ldapsearch -x -W -D 'cn=directory manager' -b 'dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com' -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv

This should at least be returning an entry if replication is enabled. Is there anything under: o=ipaca?

I do not see an issue with deleting a replica if no RUV information is found, as long as you know it is in fact a replica.

Yes, there is.

$ ldapsearch -x -W -D 'cn=directory manager' -b 'o=ipaca' -s one
<snip>
# numResponses: 10
# numEntries: 9


$ ldapsearch -x -W -D 'cn=directory manager' -b 'o=ipaca' -s subtree
<snip>
# numResponses: 76
# numEntries: 75

Sorry I wanted you to run:

$ldapsearch -x -W -D 'cn=directory manager' -b 'o=ipaca' -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv

Nothing there.

ldapsearch -x -W -D 'cn=directory manager' -b 'o=ipaca' -s one '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv 
Enter LDAP Password: 
# extended LDIF
#
# LDAPv3
# base <o=ipaca> with scope oneLevel
# filter: (&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))
# requesting: nsds50ruv 
#

# search result
search: 2
result: 0 Success

# numResponses: 1

Sorry I'm not sure how this can happen. Once replication is enabled, the root tombstone entry is created. Even is you remove replication, that entry still exists. From the outside looking in, it looks like replication was never set up, except that you can see the config entries. But without any ruv data I don't know how replication could still be working.

If you restart the DS, are there any errors, and is the RUV data still missing? Also, what version of DS is this?

Latest in updates: 389-ds-base-1.3.1.6-1.fc19.x86_64

I don't see any errors after restart. RUV data still missing.

$ sudo systemctl restart dirsrv@IDM-LAB-ENG-BRQ-REDHAT-COM.service 
$ sudo tail -255 /var/log/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/errors
<snip>
[23/Aug/2013:18:53:00 +0200] - slapd shutting down - signaling operation threads
[23/Aug/2013:18:53:00 +0200] - slapd shutting down - closing down internal subsystems and plugins
[23/Aug/2013:18:53:00 +0200] - Waiting for 4 database threads to stop
[23/Aug/2013:18:53:01 +0200] - All database threads now stopped
[23/Aug/2013:18:53:01 +0200] - slapd stopped.
[23/Aug/2013:18:53:02 +0200] - 389-Directory/1.3.1.6 B2013.213.2156 starting up
[23/Aug/2013:18:53:02 +0200] schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com
[23/Aug/2013:18:53:02 +0200] schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com
[23/Aug/2013:18:53:02 +0200] schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com
[23/Aug/2013:18:53:02 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[23/Aug/2013:18:53:02 +0200] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[23/Aug/2013:18:53:02 +0200] - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[23/Aug/2013:18:53:02 +0200] - Listening on All Interfaces port 636 for LDAPS requests
[23/Aug/2013:18:53:02 +0200] - Listening on /var/run/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM.socket for LDAPI requests

NSMMReplicationPlugin errors, note that replica was shutdown once:

$ sudo grep NSMMReplicationPlugin /var/log/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/errors 
[23/Aug/2013:13:24:33 +0200] NSMMReplicationPlugin - agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
[23/Aug/2013:13:24:33 +0200] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389)".
[23/Aug/2013:13:24:37 +0200] NSMMReplicationPlugin - Finished total update of replica "agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389)". Sent 245 entries.
[23/Aug/2013:13:25:39 +0200] NSMMReplicationPlugin - Beginning total update of replica "agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389)".
[23/Aug/2013:13:25:43 +0200] NSMMReplicationPlugin - Finished total update of replica "agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389)". Sent 70 entries.
[23/Aug/2013:13:26:21 +0200] NSMMReplicationPlugin - agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389): Unable to receive the response for a startReplication extended operation to consumer (Can't contact LDAP server). Will retry later.
[23/Aug/2013:13:26:24 +0200] NSMMReplicationPlugin - agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389): Replication bind with SIMPLE auth resumed
[23/Aug/2013:13:42:31 +0200] NSMMReplicationPlugin - agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ()
[23/Aug/2013:13:53:02 +0200] NSMMReplicationPlugin - agmt_delete: begin
[23/Aug/2013:13:53:02 +0200] NSMMReplicationPlugin - Warning: incremental protocol for replica "agmt="cn=masterAgreement1-vm-175.idm.lab.eng.brq.redhat.com-pki-tomcat" (vm-175:389)" did not shut down properly.
[23/Aug/2013:18:37:11 +0200] NSMMReplicationPlugin - agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389): Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[23/Aug/2013:18:37:14 +0200] NSMMReplicationPlugin - agmt="cn=meTovm-175.idm.lab.eng.brq.redhat.com" (vm-175:389): Replication bind with GSSAPI auth resumed

I looked into the replica box, and it seems to be working fine. The problem appears to be with your scope. Remove the "-s one" from the search:

ldapsearch -xLLL -W -D 'cn=directory manager' -b 'dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com' "(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))"
dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 52174670000000040000
nsds50ruv: {replica 3 ldap://vm-175.idm.lab.eng.brq.redhat.com:389} 52174678000000030000 5217b27b000100030000
nsds50ruv: {replica 4 ldap://vm-161.idm.lab.eng.brq.redhat.com:389} 521746cb000000040000 5217938d000200040000
dc: idm

In the repl agreements the RUV data is there as well:

dn: cn=meTovm-161.idm.lab.eng.brq.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Deng\2Cdc\3Dbrq\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping tree,cn=config
cn: meTovm-161.idm.lab.eng.brq.redhat.com
objectClass: nsds5replicationagreement
objectClass: top
nsDS5ReplicaTransportInfo: LDAP
description: me to vm-161.idm.lab.eng.brq.redhat.com
nsDS5ReplicaRoot: dc=idm,dc=lab,dc=eng,dc=brq,dc=redhat,dc=com
nsDS5ReplicaHost: vm-161.idm.lab.eng.brq.redhat.com
nsds5replicaTimeout: 120
nsDS5ReplicaPort: 389
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsDS5ReplicaBindMethod: SASL/GSSAPI
creatorsName: cn=directory manager
modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=config
createTimestamp: 20130823112433Z
modifyTimestamp: 20130823114107Z
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
 uccessfulauth krblastfailedauth krbloginfailedcount
nsds50ruv: {replicageneration} 52174670000000040000
nsds50ruv: {replica 4 ldap://vm-161.idm.lab.eng.brq.redhat.com:389} 521746cb000000040000 52174767000600040000
nsds50ruv: {replica 3 ldap://vm-175.idm.lab.eng.brq.redhat.com:389} 52174678000000030000 52174772000000030000
nsruvReplicaLastModified: {replica 4 ldap://vm-161.idm.lab.eng.brq.redhat.com:389} 00000000
nsruvReplicaLastModified: {replica 3 ldap://vm-175.idm.lab.eng.brq.redhat.com:389} 00000000
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName internalModifyTimestamp

Everything appears normal on the DS side. However, I'm not sure what/where "ipa-replica-manage del" is looking for the RUVs. Maybe it too is using a scoped search?

Yes, ipa-replica-manage uses a scoped search.

What I don't understand is how this entry was missed as it seems to be within scope.

Replying to [comment:11 rcritten]:

Yes, ipa-replica-manage uses a scoped search.

What I don't understand is how this entry was missed as it seems to be within scope.

This appears to be a design limitation. Doing a one level tree search uses the parentid index, but since this is a tombstone entry it is not indexed, and it does not contain the parentid operational attribute, and hence not picked up for scope "one,base" searches. In fact, any search for a tombstone that is "base,one" scoped will fail to return anything. "subtree" scoped searches are not internally modified, just "one" and "base" level searches have their filters modified internally(where its adds a filter component for the parentid).

There are cases when looking up special entries where the search scope is not intuitive. Like for root DSE searches, you have to use "-b base" or else the entry is not returned, and for tombstones use must use a subtree scoped search "-s sub".

Metadata Update from @pvoborni:
- Issue assigned to pvoborni
- Issue set to the milestone: FreeIPA 3.3.x - 2013/09 (bug fixing)

7 years ago

Login to comment on this ticket.

Metadata