Moving to next month iteration.

The chain of events which caused the failure goes like this:

1) CA is installed, creates a ds instance (pki)

2) DS in installed, calls is_ds_running, but does not qualify which instance, return true

3) Stores running key with value TRUE in sysrestore.state for DS instance

4) at conclusion there are 2 DS instnaces (pki & ipa)

Then on uninstall:

1) CA is uninstalled, pki instance is removed.

2) DS uninstall begins, gets running value from sysrestore, which is TRUE, saves this value

3) DS instance is uninstalled, no dirsrv instances remain (pki and ipa are now gone)

4) /etc/sysconfig/dirsrv is restored from sysrestore

5) dirsrv is started because the running flag was TRUE, this causes failure, there are no dirsrv instances.

The root cause of the problem is the running flag for dirsrv is wrong, it's picking the fact the PKI ds instance is running which is started earlier in the install sequence. It needs to test if a specific instance of dirsrv is running, not any dirsrv instance.

There are two fixes needed:

in dsinstance.py:

    def __create_instance(self):
        self.backup_state("running", is_ds_running())
        self.backup_state("serverid", self.serverid)

it should be:

    def __create_instance(self):
        self.backup_state("running", is_ds_running(self.serverid))
        self.backup_state("serverid", self.serverid)

In other words the is_ds_running() needs to be passed the instance name.

The other fix is to catch the error and report it instead of spewing a backtrace.

After thinking about this I realized the above proposed fix is incorrect. The purpose of saving the running flag is restart a service we may have taken ownership of during the IPA epoch after it's configuration has been restored at the conclusion of the IPA epoch. The proposed fix is silly, it's asking if the IPA ds instance is running and should we restart it after uninstalling IPA. The answer to the question "Is the IPA DS instance running should be no (it's a serious error if it is during install). Therefore it returns False and we don't attempt to restart dirsrv. This does in fact solve the backtrace for the wrong reasons.

The fundamental problem is we've muddled concepts. We want to restore services to their prior state after uninstall. Most services have a one-to-one mapping between the service and the service instance, very few services allow arbitrary instances. The existing logic is modelled after the common case of services having only a single eponymous instance.

However for services which can have multiple instances the above logic does not hold. Here's why, we create our own instances of a service (e.g. dirsrv or pki), those instances are independent of any other instance which may be started/stopped by an initscript or systemd. Since those instances of the service belong uniquely to IPA there is no point in asking if we should restart them after IPA is uninstalled, the answer is trivially NO, after IPA is uninstalled those instances should not exist.

However, other instances of the service may indeed exist, it's not our business to start or stop them, they're independent of IPA. For services which support multiple instances we should not be capturing their prior state, we should not start or stop them (e.g. all instances of the service), nor should we restore their state. The reason why is because the configuration and state of other instances are wholly independent of IPA. This is markedly different than single-only instance services where we take complete ownership of the service and modify it for IPA's usage. In those cases we do need to save/restore.

I believe the real fix is to treat services supporting multiple instances a bit differently. Do not save or restore they're state and ALWAYS only operate on the instance(s) we own, never on the collection of all instances on the system. Unless we want to guarantee no other instances run simultaneously with IPA, but even that would require different handling.

patch submitted

[PATCH 71] improve handling of ds instances during uninstall

master: af0ac84

ipa-2-2: ad678d1