I have been investigating this issue when testing RHEL and have an idea how to fix - I can take it.

But we will definitely need to fix it in 3.3.x branch.

attachment
freeipa-mkosek-461-update-dogtag-9-database-during-replica-installation.patch

Patch freeipa-mkosek-461-update-dogtag-9-database-during-replica-installation.patch sent for review

Starting review

master:

• b3c2197 Update Dogtag 9 database during replica installation

ipa-3-3:

• 9bc032f Update Dogtag 9 database during replica installation

The fix almost works perfectly but I am reopening because it misses a critical part.
The script must restart the pki-tomcat instance or the changes will not have effect.
I just spent hours trying to create a clone and it was always failing wth authorization errors.

As soon as I restarted pki-tomcatd on the master, I was able to successfully create a CA replica without any issue.

The patch does restart the pki-tomcat on the installed replica. Not sure what can we do about master it replicates from, we do not have control over it in ipa-replica-install/ipa-ca-install.

AFAIR this update is performed also at RPM upgrade time, it's at that time that pki-tomcatd need to be restarted.
It is probably a good idea to always restart the CA after an rpm update of freeipa in case something changes anyway.

Ah, I think I finally understood your scenario:

1. You had existing FreeIPA server based on Dogtag 9
2. You installed FreeIPA Dogtag10 based replica with some FreeIPA version before this fix.
3. You tried to install other FreeIPA Dogtag10 based replica which failed due to the access error
4. You updated replica FreeIPA version to the version with this patch applied, replica creation still did not work due to the access error unless you restarted the VM.

Correct?

If this is correct, I am not convinced this is something we need to fix as only very limited number of deployments is affected, all new migrations do restart the PKI so we are covered there. Also, workaround is very easy.

A potential fix would not be easy as we cannot restart PKI based on *.update file contents, we would need to turn this into update plugin.

Normally, we restart PKI when any change is done, but we mostly do such changes only in ipa-upgradeconfig, you can check in:

grep ca_restart install/tools/ipa-upgradeconfig

We had another discussion with simo outside of this ticket.

Something I still wondering about is why the IPA services did not restart on the server upgrade?

We have following snippet in our spec file:

%posttrans server
# This must be run in posttrans so that updates from previous
# execution that may no longer be shipped are not applied.
/usr/sbin/ipa-ldap-updater --upgrade --quiet >/dev/null || :
/usr/sbin/ipa-upgradeconfig --quiet >/dev/null || :

# Restart IPA processes. This must be also run in postrans so that plugins
# and software is in consistent state
python2 -c "import sys; from ipaserver.install import installutils; sys.exit(0 if installutils.          is_ipa_configured() else 1);" > /dev/null 2>&1
# NOTE: systemd specific section
if [  $? -eq 0 ]; then
    /bin/systemctl try-restart ipa.service >/dev/null 2>&1 || :
fi
# END

Simo, does it mean that the Python call does evaluate as "1" in your case so that IPA services are not restarted?

Replying to [comment:11 mkosek]:

We had another discussion with simo outside of this ticket.

Something I still wondering about is why the IPA services did not restart on the server upgrade?

We have following snippet in our spec file:

{{{
%posttrans server

This must be run in posttrans so that updates from previous

execution that may no longer be shipped are not applied.

/usr/sbin/ipa-ldap-updater --upgrade --quiet >/dev/null || :
/usr/sbin/ipa-upgradeconfig --quiet >/dev/null || :

Restart IPA processes. This must be also run in postrans so that plugins

and software is in consistent state

python2 -c "import sys; from ipaserver.install import installutils; sys.exit(0 if installutils. is_ipa_configured() else 1);" > /dev/null 2>&1

NOTE: systemd specific section

if [ $? -eq 0 ]; then
/bin/systemctl try-restart ipa.service >/dev/null 2>&1 || :
fi

END

}}}

Simo, does it mean that the Python call does evaluate as "1" in your case so that IPA services are not restarted?

I've just run the command manually and it returns 0 on my system.

So it should have restarted it ... but apparently didn't

And I just ry systemctl try-restart ipa.service on my machine and it did nothing.

I would try a more robust approach, can we check if ipa.service is enabled and in that case forcibly run
systemctl stop ipa.service
systemctl start ipa.service

My system was saying that the ipa service is dead which caused try-restart to fail and do nothing :-(

Would you rather like to change it to "systemctl restart ipa.service" then? This would try to restart it whether it is running or not. Checking if IPA server is enabled is indeed a good idea:

# systemctl is-enabled ipa.service ; echo $?
enabled
0

Speaking of the failure in your case, it seems that systemd thought that IPA service is not running. I wonder why. ipa service is a oneshot type of systemd service so I think systemd assumes whether ipa is running or not based on ipactl start result.

It makes me wonder why systemd thought the service was dead. I think that try-restart is the "correct" way to do it and doing anything else is just trying to work around systemd bugs (or bugs in the way we report status).

This was discussed on the meeting and we will workaround systemd bug for the short term in 4.0. For the long term, we would need refactoring of our systemd unit files.

This is a quick fix we should do in 4.0, thus moving to month milestone.

We should also consider backporting to ipa-3-3 as this blocks #4213.

master:

• bd1df14 Fix ipa.service restart

ipa-3-3:

• fcb2cd4 Fix ipa.service restart