#6176 Updating of dns system records rapidly slowdown uninstallation
Closed: fixed 3 years ago Opened 4 years ago by mbasti.

In case that server is uninstalled using ipa-server-install --uninstall, there is unwanted update of DNS records executed.

This is caused by following:
1. ipa-server-install --uninstall
1. internally calls ipa server-del
1. [server-del] server principals are disabled
1. [server-del] replication stops working
1. [server-del] named died
1. [server-del] cleanup of IPA system records +++
1. unconfiguring services

+++ this is bad named is dead already, so it does not provide any records, changes are probably not replicated anymore, and update plugins only do timeouts t=<number of CA servers> * 120sec. Also updated DNS records are not replicated back to topology thus admin must manually run ipa dns-update-system-records on different replica.

This is somehow chicken egg problem with this use case, we need update DNS records before replication is broken, but removing DNS records may break replication too and removing replica from topology is incomplete and needs --cleanup.

Not sure but we may hit this issue too with

[thisserver ~]# ipa server-del thisserver

Solution may be to redirect call of server-del to another replica.

[ipa.ipatests.test_integration.host.Host.replica3.ParamikoTransport] RUN ['ipa-server-install', '--uninstall', '-U', '--ignore-topology-disconnect', '--ignore-last-of-role']
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] RUN ['ipa-server-install', '--uninstall', '-U', '--ignore-topology-disconnect', '--ignore-last-of-role']
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Updating DNS system records
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for replica3.ipa.test. 1 failed: The DNS operation timed out after 30.0013668537 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for replica3.ipa.test. 1 failed: The DNS operation timed out after 30.0007688999 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for replica3.ipa.test. 1 failed: The DNS operation timed out after 30.0009179115 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for replica3.ipa.test. 1 failed: The DNS operation timed out after 30.0009920597 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    unable to resolve host name replica3.ipa.test. to IP address, ipa-ca DNS record will be incomplete
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for master.ipa.test. 1 failed: The DNS operation timed out after 30.0015940666 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for master.ipa.test. 1 failed: The DNS operation timed out after 30.0007860661 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for master.ipa.test. 1 failed: The DNS operation timed out after 30.0015718937 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for master.ipa.test. 1 failed: The DNS operation timed out after 30.0010399818 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    unable to resolve host name master.ipa.test. to IP address, ipa-ca DNS record will be incomplete
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for replica2.ipa.test. 1 failed: The DNS operation timed out after 30.0017671585 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for replica2.ipa.test. 1 failed: The DNS operation timed out after 30.001762867 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for replica2.ipa.test. 1 failed: The DNS operation timed out after 30.0012130737 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    DNS query for replica2.ipa.test. 1 failed: The DNS operation timed out after 30.0008029938 seconds
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] ipa         : ERROR    unable to resolve host name replica2.ipa.test. to IP address, ipa-ca DNS record will be incomplete
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] --------------------------------------
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Deleted IPA server "replica3.ipa.test"
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] --------------------------------------
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Shutting down all IPA services
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring ntpd
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Configuring certmonger to stop tracking system certificates for KRA
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Configuring certmonger to stop tracking system certificates for CA
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring CA
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring web server
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring krb5kdc
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring kadmin
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring directory server
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring ipa-custodia
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring ipa_memcached
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring ipa-otpd
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Removing Kerberos service principals from /etc/krb5.keytab
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Disabling client Kerberos and LDAP configurations
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd/sssd.conf.deleted
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Restoring client configuration files
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Unconfiguring the NIS domain.
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] nscd daemon is not installed, skip configuration
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] nslcd daemon is not installed, skip configuration
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Systemwide CA database updated.
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Client uninstall complete.
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Removing IPA client configuration
[ipa.ipatests.test_integration.host.Host.replica3.cmd32] Exit code: 0

Also note that installer is trying to add ipa-ca record for replica3, that is being uninstalled, so there is some data inconsistency.

ipa         : ERROR    DNS query for replica3.ipa.test. 1 failed: The DNS operation timed out after 30.0013668537 seconds
ipa         : ERROR    DNS query for replica3.ipa.test. 1 failed: The DNS operation timed out after 30.0007688999 seconds
ipa         : ERROR    DNS query for replica3.ipa.test. 1 failed: The DNS operation timed out after 30.0009179115 seconds
ipa         : ERROR    DNS query for replica3.ipa.test. 1 failed: The DNS operation timed out after 30.0009920597 seconds
ipa         : ERROR    unable to resolve host name replica3.ipa.test. to IP address, ipa-ca DNS record will be incomplete

I am not sure if these things are related, but:[[BR]]
When I uninstall replica with the common {{{ipa-server-install --uninstall}}}, the corresponding topology segment stays on master and other replicas. When I afterwards run ipa-replica-manage-del on master, it says:

<replica_hostname>: server not found

Which means that the server-dell call is executed during replica uninstallation, but the replication agreement is not removed.
It turns out that the only way to correctly remove replica from topology is to run ipa-replica-manage-del.

This affects uninstallation if server-del is not run in advance. Given that IPA Guide still mentions the old method which runs it in advance the fix /proper investiagtion may be done later.

This issue also affects uninstallation after an unsuccessful installation. It takes several minutes more to uninstall:

# ipa-server-install --uninstall  --unattended
Updating DNS system records
ipa         : ERROR    unable to resolve host name master.ipa.example. to IP address, ipa-ca DNS record will be incomplete
---------------------------------------
Deleted IPA server "master.ipa.example"
---------------------------------------
Shutting down all IPA services

/etc/resolv.conf points to an upstream DNS server. master.ipa.example is in /etc/hosts.

Metadata Update from @mbasti:
- Issue assigned to someone
- Issue set to the milestone: FreeIPA 4.5 backlog

3 years ago

This causes significant slowdown of the PR CI. The multihost fixture uninstalls the server during teardown and it's affected by this issue. Moving to current milestone.

Metadata Update from @tkrizek:
- Issue close_status updated to: None
- Issue set to the milestone: FreeIPA 4.5.2 (was: FreeIPA 4.5 backlog)

3 years ago

Metadata Update from @tkrizek:
- Issue set to the milestone: FreeIPA 4.5.3 (was: FreeIPA 4.5.2)

3 years ago

Metadata Update from @tkrizek:
- Issue set to the milestone: FreeIPA 4.5.4 (was: FreeIPA 4.5.3)

3 years ago
3 years ago
3 years ago

master:

  • dffddbd DNS update: reduce timeout for CA records

ipa-4-5:

  • 2b3b94f DNS update: reduce timeout for CA records

Metadata Update from @tkrizek:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata