#6756 interrupted ipa-server-install results in failing DS instance removal
Opened 5 years ago by mbabinsk. Modified 2 years ago

When user interrupts server/replica installation at some random point (by Ctrl+C'ing e.g. during CA instance creating as shown in the example), the user is informed about the DS instance cleanup taking place:

<SNIP>
 [45/47]: activating extdom plugin
  [46/47]: tuning directory server
  [47/47]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds
  [1/30]: creating certificate server user
  [2/30]: configuring certificate server instance
^C  [error] KeyboardInterrupt: 

Cleaning up...
Removing configuration for IPA-TEST instance
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

However, subsequent uninstallation fails to unconfigure DS instance, see error on the console:

[root@master1 ~]# ipa-server-install --uninstall -U

WARNING: Failed to connect to Directory Server to find information about
replication agreements. Uninstallation will continue despite the possible
existing replication agreements.

If this server is the last instance of CA,
KRA, or DNSSEC master, uninstallation may result in data loss.
Shutting down all IPA services
Unconfiguring ntpd
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Unconfiguring CA
Unconfiguring directory server
ipa         : ERROR    Instance removal failed.
ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
ipa         : ERROR    Unable to restart DS instance IPA-TEST: Command '/bin/systemctl restart dirsrv@IPA-TEST.service' returned non-zero exit status 1
Removing IPA client configuration
Removing Kerberos service principals from /etc/krb5.keytab
Disabling client Kerberos and LDAP configurations
Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted
nscd daemon is not installed, skip configuration
nslcd daemon is not installed, skip configuration
Client uninstall complete.
The ipa-client-install command was successful

and the uninstall log:

2017-03-13T13:31:18Z DEBUG Removing DS instance IPA-TEST
2017-03-13T13:31:18Z DEBUG Starting external process
2017-03-13T13:31:18Z DEBUG args=/usr/sbin/remove-ds.pl -i slapd-IPA-TEST
2017-03-13T13:31:18Z DEBUG Process finished, return code=1
2017-03-13T13:31:18Z DEBUG stdout=
2017-03-13T13:31:18Z DEBUG stderr=The following errors occurred during removal:
Error: could not find directory server configuration directory 'slapd-IPA-TEST'.  Error: No such file or directory
Error: could not remove directory server IPA-TEST

2017-03-13T13:31:18Z DEBUG '/usr/sbin/remove-ds.pl' failed. Attempting to force removal
2017-03-13T13:31:18Z DEBUG Forcing instance removal
2017-03-13T13:31:18Z DEBUG Starting external process
2017-03-13T13:31:18Z DEBUG args=/usr/sbin/remove-ds.pl -i slapd-IPA-TEST -f
2017-03-13T13:31:18Z DEBUG Process finished, return code=1
2017-03-13T13:31:18Z DEBUG stdout=
2017-03-13T13:31:18Z DEBUG stderr=The following errors occurred during removal:
Error: could not find directory server configuration directory 'slapd-IPA-TEST'.  Error: No such file or directory
Error: could not remove directory server IPA-TEST

2017-03-13T13:31:18Z ERROR Instance removal failed.
2017-03-13T13:31:18Z ERROR Failed to remove DS instance. You may need to remove instance data manually

This of course breaks subsequent attempts to re-install server/replica due to DS leftovers:

Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 1 minute
  [1/47]: creating directory server user
  [2/47]: creating directory server instance
  [error] RuntimeError: failed to create DS instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmpwlemx5' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    failed to create DS instance Command '/usr/sbin/setup-ds.pl --silent --logfile - -f /tmp/tmpwlemx5' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

When inspecting the FS after uninstall the following files/directory were left over from failed DS instance removal:

[root@master1 ~]# find / -name *IPA-TEST* -print
/etc/dirsrv/slapd-IPA-TEST.removed
/etc/dirsrv/slapd-IPA-TEST
/etc/systemd/system/dirsrv.target.wants/dirsrv@IPA-TEST.service
/var/lib/dirsrv/scripts-IPA-TEST
/run/slapd-IPA-TEST.socket

Removing /etc/dirsrv/slapd-IPA-TEST directory enabled me to reinstall IPA server again.

Relevant versions:

[root@master1 ~]# rpm -q freeipa-server
freeipa-server-4.4.90.dev201703130946+gitb3f5f3f-0.fc25.x86_64
[root@master1 ~]# rpm -q 389-ds-base
389-ds-base-1.3.5.15-1.fc25.x86_64

From DS pov, "could not find directory server configuration directory 'slapd-IPA-TEST'. Error: No such file or directory" means
either
/etc/sysconfig/dirsrv-IPA-TEST
or
/etc/dirsrv/slapd-IPA-TEST

do not exist

Hardening of dsremove.pl and setupds.pl is a long term goal of 389-ds.
I will be achieve with rewriting those tools over lib389: https://pagure.io/lib389/issue/8

Metadata Update from @pvoborni:
- Custom field external_tracker adjusted to https://pagure.io/lib389/issue/8
- Issue priority set to: major
- Issue set to the milestone: FreeIPA 4.7
- Issue tagged with: bug, tracker

5 years ago

Metadata Update from @rcritten:
- Issue set to the milestone: FreeIPA 4.7.1 (was: FreeIPA 4.7)

4 years ago

FreeIPA 4.7 has been released, moving to FreeIPA 4.7.1 milestone

This is still reproducible. Re-running ipa-server-install --uninstall a second time cleared things up for me.

Metadata Update from @rcritten:
- Issue priority set to: minor (was: normal)

3 years ago

On Fedora 33, FreeIPA 4.8.10-6, uninstall fails:


File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 431, in runner
exc_handler(exc_info)
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(exc_info)
File "/usr/lib/python3.9/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(
exc_info)
File "/usr/lib/python3.9/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 655, in _configure
next(executor)
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 518, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(exc_info)
File "/usr/lib/python3.9/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 515, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(
exc_info)
File "/usr/lib/python3.9/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.9/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.9/site-packages/ipapython/install/common.py", line 73, in _uninstall
for unused in self._uninstaller(self.parent):
File "/usr/lib/python3.9/site-packages/ipaclient/install/client.py", line 3879, in main
uninstall(self)
File "/usr/lib/python3.9/site-packages/ipaclient/install/client.py", line 3275, in uninstall
ipa_db = certdb.NSSDatabase(paths.IPA_NSSDB_DIR)
File "/usr/lib/python3.9/site-packages/ipapython/certdb.py", line 267, in __init

raise ValueError(

2020-11-30T21:57:25Z DEBUG The ipa-client-install command failed, exception: ValueError: NSS is built without support of the legacy database(DBM)
2020-11-30T21:57:25Z ERROR NSS is built without support of the legacy database(DBM)
2020-11-30T21:57:25Z ERROR The ipa-client-install command failed. See /var/log/ipaclient-uninstall.log for more information

Login to comment on this ticket.

Metadata