From the ipa-restore logs, the restore operation happens between 2022-01-18T09:25:16Z and 2022-01-18T09:27:48Z, and dsctl ldif2db is executed between 2022-01-18T09:25:24Z and 2022-01-18T09:25:28Z to restore the userRootDB.
The 389ds server is then restarted and detects issues with the replication:

[18/Jan/2022:09:25:48.621846204 +0000] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTomaster.ipa.test" (master:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.

The replication never resumes.

The test scenario could be challenged: according to the doc "Recovering from data loss with IdM backups", the other servers should be re-initialized from the restored server.

The replication failing is expected. As part of restoration all replication agreements are disabled. The other servers need to be re-initialized (or the server re-installed) so they all have the same restored state. Restore really is the last resort.

I'm perplexed how this was ever successful. Restoration is just a load from LDIF and I don't see any code that would change the server state.

I can't reproduce this manually either. My ldapsearch of the tree post-restore shows things like:

dn: cn=KDC,cn=replica.example.test,cn=masters,cn=ipa,cn=etc,dc=example,dc=test
cn: KDC
ipaConfigString: startOrder 10
ipaConfigString: configuredService
ipaConfigString: kdcProxyEnabled
ipaConfigString: pkinitEnabled
ipaConfigString: enabledService
ipaConfigString: hiddenService
objectClass: nsContainer
objectClass: ipaConfigObject
objectClass: top

I think I'll try a PR to instrument this a bit to see if I can figure out what is going on in current systems and what changed.

Need more testing but as far as I can tell ipaConfigString prior to uninstall and restore contains hiddenService and not enabledService. After restore it contains both, so something must be adding it. Unfortunately the PR that is associated with this test gives no indication at all that I can find where this is being done. I can see the reasoning though. A restore leaves you with one server and obviously you don't want it hidden.

I think this is a bug in 389-ds that is now maybe fixed since the test is failing against 389-nightly.

I picked a random service out of the many to troubleshoot, landing on KDC.

The entry prior to backup contains:

dn: cn=KDC,cn=replica.example.test,cn=masters,cn=ipa,cn=etc,dc=example,dc=test
cn: KDC
ipaConfigString: startOrder 10
ipaConfigString: configuredService
ipaConfigString: kdcProxyEnabled
ipaConfigString: pkinitEnabled
ipaConfigString: hiddenService
objectClass: nsContainer
objectClass: ipaConfigObject
objectClass: top

Looking at the LDIF backup there is:

# entry-id: 514
dn: cn=KDC,cn=replica.example.test,cn=masters,cn=ipa,cn=etc,dc=example,dc=test
modifyTimestamp;adcsn-61f0725d000000030003;vucsn-61f0725d000000030003: 2022012
 5215749Z
modifiersName;adcsn-61f0725d000000030002;vucsn-61f0725d000000030002: uid=admin
 ,cn=users,cn=accounts,dc=example,dc=test
objectClass;vucsn-61f06e96000000030000: nsContainer
objectClass;vucsn-61f06e96000000030000: ipaConfigObject
objectClass;vucsn-61f06e96000000030000: top
cn;vucsn-61f06e96000000030000;mdcsn-61f06e96000000030000: KDC
ipaConfigString;vucsn-61f06e96000000030000: startOrder 10
ipaConfigString;vucsn-61f06eb3000000030000: kdcProxyEnabled
ipaConfigString;vucsn-61f06fb3000000030000: pkinitEnabled
ipaConfigString;vucsn-61f0725d000000030001: hiddenService
ipaConfigString;vucsn-61f06e96000000030000;vdcsn-61f06fef000000030000;deleted:
 configuredService
ipaConfigString;vucsn-61f06fef000000030001;vdcsn-61f0725d000000030000;deleted:
 enabledService
creatorsName;vucsn-61f06e96000000030000: cn=Directory Manager
createTimestamp;vucsn-61f06e96000000030000: 20220125214142Z
nsUniqueId: 8dd4ca9b-7e2711ec-9dd58ace-11c1cbb2

I can only assume that the deleted values for configuredService and enabledServer are due to my making the entry hidden.

So I uninstall IPA and restore and the entry looks like:

# KDC, replica.example.test, masters, ipa, etc, example.test
dn: cn=KDC,cn=replica.example.test,cn=masters,cn=ipa,cn=etc,dc=example,dc=test
cn: KDC
ipaConfigString: startOrder 10
ipaConfigString: configuredService
ipaConfigString: kdcProxyEnabled
ipaConfigString: pkinitEnabled
ipaConfigString: enabledService
ipaConfigString: hiddenService
objectClass: nsContainer
objectClass: ipaConfigObject
objectClass: top

So the deleted values are also being restored, it seems. This is on F33 with 389-ds-base-1.4.4.17. So quite old, but we're comparing old behavior and new behavior.

I'll see if can make the PR test an updated F35 system with 389-nightly and see if the behavior is different.

I reproduced this in my test PR https://github.com/freeipa/freeipa/pull/6153

If we want a restored server to no longer be hidden when are going to need to add code to do so, or modify the test. The original behavior in IPA was a side-effect of 389-ds.

I filed a bug against 389-ds for RHEL 8: https://bugzilla.redhat.com/show_bug.cgi?id=2050439

I closed the PR.

In team meeting we decided to force a hidden server to be made visible on restoration. There is no use-case for a single server IPA Installation to be hidden (and is prevented to be done under normal circumstances).

master:

• 50241b3 ipa-restore: Mark a restored server as enabled

ipa-4-9:

• ab9e7da ipa-restore: Mark a restored server as enabled