#6205 The ipa-server-upgrade command failed when named-pkcs11 does not happen to run during dnf upgrade
Closed: Fixed None Opened 3 years ago by pspacek.

Ticket was cloned from Red Hat Bugzilla (product Fedora): Bug 1365669

Description of problem:

When freeipa-server-4.3.1-1.fc24.x86_64 gets upgraded to
freeipa-server-4.3.2-1.fc24.x86_64 and named-pkcs11 service is not running in
the posttrans (?), error

DNS query for ipa.example.test. A failed: The DNS operation timed out after
30.0009489059 seconds
Upgrade failed with The DNS operation timed out after 30.0009489059 seconds
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
('IPA upgrade failed.', 1)
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
information

is shown.

Version-Release number of selected component (if applicable):

freeipa-server-4.3.2-1.fc24.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Install Freeipa 4.3.1, configure it with ipa-server-install --setup-dns.
2. Run dnf upgrade --downloadonly --enablerepo updates-testing freeipa-server.
3. Run systemctl stop named-pkcs11
4. Run dnf upgrade --enablerepo updates-testing freeipa-server

Actual results:

  Upgrading   : freeipa-common-4.3.2-1.fc24.noarch           1/20
  Upgrading   : freeipa-client-common-4.3.2-1.fc24.noarch    2/20
  Upgrading   : freeipa-server-common-4.3.2-1.fc24.noarch    3/20
  Upgrading   : python2-ipalib-4.3.2-1.fc24.noarch           4/20
  Upgrading   : python2-ipaclient-4.3.2-1.fc24.noarch        5/20
  Upgrading   : freeipa-client-4.3.2-1.fc24.x86_64           6/20
Could not load host key: /etc/ssh/ssh_host_dsa_key
  Upgrading   : python2-ipaserver-4.3.2-1.fc24.noarch        7/20
  Upgrading   : freeipa-admintools-4.3.2-1.fc24.noarch       8/20
  Upgrading   : freeipa-server-4.3.2-1.fc24.x86_64           9/20
  Upgrading   : freeipa-server-dns-4.3.2-1.fc24.noarch      10/20
  Cleanup     : freeipa-server-dns-4.3.1-1.fc24.noarch      11/20
  Cleanup     : freeipa-server-4.3.1-1.fc24.x86_64          12/20
  Cleanup     : python2-ipaserver-4.3.1-1.fc24.noarch       13/20
  Cleanup     : freeipa-client-4.3.1-1.fc24.x86_64          14/20
  Cleanup     : python2-ipaclient-4.3.1-1.fc24.noarch       15/20
  Cleanup     : freeipa-admintools-4.3.1-1.fc24.noarch      16/20
  Cleanup     : python2-ipalib-4.3.1-1.fc24.noarch          17/20
  Cleanup     : freeipa-server-common-4.3.1-1.fc24.noarch   18/20
  Cleanup     : freeipa-client-common-4.3.1-1.fc24.noarch   19/20
  Cleanup     : freeipa-common-4.3.1-1.fc24.noarch          20/20
DNS query for ipa.example.test. A failed: The DNS operation timed out after
30.0009489059 seconds
Upgrade failed with The DNS operation timed out after 30.0009489059 seconds
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
('IPA upgrade failed.', 1)
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
information
  Verifying   : freeipa-server-4.3.2-1.fc24.x86_64           1/20
  Verifying   : freeipa-admintools-4.3.2-1.fc24.noarch       2/20

Expected results:

No error.

Additional info:

The /var/log/ipaupgrade.log ends with

2016-08-09T19:37:05Z DEBUG dnsconfig_show(rights=False, all=True, raw=False,
version=u'2.164')
2016-08-09T19:37:05Z DEBUG Executing upgrade plugin:
update_dnsforward_emptyzones
2016-08-09T19:37:05Z DEBUG raw: update_dnsforward_emptyzones
2016-08-09T19:37:05Z DEBUG raw: dnsconfig_show(all=True, version=u'2.164')
2016-08-09T19:37:05Z DEBUG dnsconfig_show(rights=False, all=True, raw=False,
version=u'2.164')
2016-08-09T19:37:05Z DEBUG Updating forwarding policies in LDAP to avoid
conflicts with automatic empty zones
2016-08-09T19:37:05Z DEBUG raw: dnsconfig_mod(ipadnsversion=2,
version=u'2.164')
2016-08-09T19:37:05Z DEBUG dnsconfig_mod(ipadnsversion=2, rights=False,
all=False, raw=False, version=u'2.164')
2016-08-09T19:37:05Z DEBUG raw: dnsforwardzone_find(None, all=True, raw=True,
version=u'2.164')
2016-08-09T19:37:05Z DEBUG dnsforwardzone_find(None, all=True, raw=True,
version=u'2.164', pkey_only=False)
2016-08-09T19:37:35Z ERROR DNS query for ipa.example.test. A failed: The DNS
operation timed out after 30.0009489059 seconds
2016-08-09T19:37:35Z DEBUG Destroyed connection context.ldap2_140607877043344
2016-08-09T19:37:35Z ERROR Upgrade failed with The DNS operation timed out
after 30.0009489059 seconds
2016-08-09T19:37:35Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py",
line 216, in __upgrade
    self.modified = (ld.update(self.files) or self.modified)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line
918, in update
    self._run_updates(all_updates)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line
890, in _run_updates
    self._run_update_plugin(update['plugin'])
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line
863, in _run_update_plugin
    restart_ds, updates = self.api.Updater[plugin_name]()
  File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 1404, in
__call__
    return self.execute(**options)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/plugins/dns.py",
line 492, in execute
    if dnsutil.has_empty_zone_addresses(self.api.env.host):
  File "/usr/lib/python2.7/site-packages/ipapython/dnsutil.py", line 278, in
has_empty_zone_addresses
    ip_addresses = resolve_ip_addresses(hostname)
  File "/usr/lib/python2.7/site-packages/ipapython/dnsutil.py", line 328, in
resolve_ip_addresses
    rrsets = resolve_rrsets(fqdn, ['A', 'AAAA'])
  File "/usr/lib/python2.7/site-packages/ipapython/dnsutil.py", line 305, in
resolve_rrsets
    answer = dns.resolver.query(fqdn, rdtype)
  File "/usr/lib/python2.7/site-packages/dns/resolver.py", line 1029, in query
    raise_on_no_answer, source_port)
  File "/usr/lib/python2.7/site-packages/dns/resolver.py", line 949, in query
    timeout = self._compute_timeout(start)
  File "/usr/lib/python2.7/site-packages/dns/resolver.py", line 770, in
_compute_timeout
    raise Timeout(timeout=duration)
Timeout: The DNS operation timed out after 30.0009489059 seconds

2016-08-09T19:37:35Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
447, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
437, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py",
line 224, in __upgrade
    raise RuntimeError(e)
RuntimeError: The DNS operation timed out after 30.0009489059 seconds

2016-08-09T19:37:35Z DEBUG   [error] RuntimeError: The DNS operation timed out
after 30.0009489059 seconds
2016-08-09T19:37:35Z DEBUG   [cleanup]: stopping directory server
2016-08-09T19:37:35Z DEBUG Starting external process
2016-08-09T19:37:35Z DEBUG args=/bin/systemctl stop dirsrv@EXAMPLE-TEST.service
2016-08-09T19:37:37Z DEBUG Process finished, return code=0
2016-08-09T19:37:37Z DEBUG stdout=
2016-08-09T19:37:37Z DEBUG stderr=
2016-08-09T19:37:37Z DEBUG   duration: 1 seconds
2016-08-09T19:37:37Z DEBUG   [cleanup]: restoring configuration
2016-08-09T19:37:37Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2016-08-09T19:37:37Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2016-08-09T19:37:37Z DEBUG Saving StateFile to
'/var/lib/ipa/sysrestore/sysrestore.state'
2016-08-09T19:37:37Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2016-08-09T19:37:37Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2016-08-09T19:37:37Z DEBUG Saving StateFile to
'/var/lib/ipa/sysrestore/sysrestore.state'
2016-08-09T19:37:37Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2016-08-09T19:37:37Z DEBUG   duration: 0 seconds
2016-08-09T19:37:37Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2016-08-09T19:37:37Z DEBUG   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
    return_value = self.run()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 50, in run
    raise admintool.ScriptError(str(e))

2016-08-09T19:37:37Z DEBUG The ipa-server-upgrade command failed, exception:
ScriptError: ('IPA upgrade failed.', 1)
2016-08-09T19:37:37Z ERROR ('IPA upgrade failed.', 1)
2016-08-09T19:37:37Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information

master:

  • f2fe357 DNS server upgrade: do not fail when DNS server did not respond

ipa-4-3:

  • 27534f8 DNS server upgrade: do not fail when DNS server did not respond

Upgrading from 4.3.1-1.fc24 to 4.3.2-2.fc24 still complains:

  Upgrading   : freeipa-common-4.3.2-2.fc24.noarch                         1/20 
  Upgrading   : freeipa-client-common-4.3.2-2.fc24.noarch                  2/20 
  Upgrading   : freeipa-server-common-4.3.2-2.fc24.noarch                  3/20 
  Upgrading   : python2-ipalib-4.3.2-2.fc24.noarch                         4/20 
  Upgrading   : python2-ipaclient-4.3.2-2.fc24.noarch                      5/20 
  Upgrading   : freeipa-client-4.3.2-2.fc24.x86_64                         6/20 
Could not load host key: /etc/ssh/ssh_host_dsa_key
  Upgrading   : python2-ipaserver-4.3.2-2.fc24.noarch                      7/20 
  Upgrading   : freeipa-admintools-4.3.2-2.fc24.noarch                     8/20 
  Upgrading   : freeipa-server-4.3.2-2.fc24.x86_64                         9/20 
  Upgrading   : freeipa-server-dns-4.3.2-2.fc24.noarch                    10/20 
  Cleanup     : freeipa-server-dns-4.3.1-1.fc24.noarch                    11/20 
  Cleanup     : freeipa-server-4.3.1-1.fc24.x86_64                        12/20 
  Cleanup     : python2-ipaserver-4.3.1-1.fc24.noarch                     13/20 
  Cleanup     : freeipa-client-4.3.1-1.fc24.x86_64                        14/20 
  Cleanup     : python2-ipaclient-4.3.1-1.fc24.noarch                     15/20 
  Cleanup     : freeipa-admintools-4.3.1-1.fc24.noarch                    16/20 
  Cleanup     : python2-ipalib-4.3.1-1.fc24.noarch                        17/20 
  Cleanup     : freeipa-server-common-4.3.1-1.fc24.noarch                 18/20 
  Cleanup     : freeipa-client-common-4.3.1-1.fc24.noarch                 19/20 
  Cleanup     : freeipa-common-4.3.1-1.fc24.noarch                        20/20
MARK-LWD-LOOP -- 2016-09-07 04:15:09 --

DNS query for ipa.example.test. A failed: The DNS operation timed out after 30.0014810562 seconds
Skipping update of global DNS forwarder in LDAP: Unable to determine if local server is using an IP address belonging to an automatic empty zone. Consider changing forwarding policy to "only". DNS exception: The DNS operation timed out after 30.0014810562 seconds
DNS query for ipa.example.test. A failed: The DNS operation timed out after 30.0013720989 seconds
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
Timeout: The DNS operation timed out after 30.0013720989 seconds
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
  Verifying   : freeipa-server-4.3.2-2.fc24.x86_64                         1/20 
  Verifying   : freeipa-admintools-4.3.2-2.fc24.noarch                     2/20 
  Verifying   : freeipa-client-4.3.2-2.fc24.x86_64                         3/20 
  Verifying   : freeipa-common-4.3.2-2.fc24.noarch                         4/20

and the log shows

2016-09-07T08:16:44Z INFO [Checking global forwarding policy in named.conf to avoid conflicts with automatic empty zones]
2016-09-07T08:16:44Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2016-09-07T08:16:44Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2016-09-07T08:16:44Z DEBUG Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state'
2016-09-07T08:17:14Z ERROR DNS query for ipa.example.test. A failed: The DNS operation timed out after 30.0013720989 seconds
2016-09-07T08:17:14Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2016-09-07T08:17:14Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1781, in upgrade
    upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1671, in upgrade_configuration
    named_update_global_forwarder_policy(),
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 817, in named_update_global_forwarder_policy
    if not dnsutil.has_empty_zone_addresses(api.env.host):
  File "/usr/lib/python2.7/site-packages/ipapython/dnsutil.py", line 278, in has_empty_zone_addresses
    ip_addresses = resolve_ip_addresses(hostname)
  File "/usr/lib/python2.7/site-packages/ipapython/dnsutil.py", line 328, in resolve_ip_addresses
    rrsets = resolve_rrsets(fqdn, ['A', 'AAAA'])
  File "/usr/lib/python2.7/site-packages/ipapython/dnsutil.py", line 305, in resolve_rrsets
    answer = dns.resolver.query(fqdn, rdtype)
  File "/usr/lib/python2.7/site-packages/dns/resolver.py", line 1029, in query
    raise_on_no_answer, source_port)
  File "/usr/lib/python2.7/site-packages/dns/resolver.py", line 858, in query
    timeout = self._compute_timeout(start)
  File "/usr/lib/python2.7/site-packages/dns/resolver.py", line 770, in _compute_timeout
    raise Timeout(timeout=duration)

2016-09-07T08:17:14Z DEBUG The ipa-server-upgrade command failed, exception: Timeout: The DNS operation timed out after 30.0013720989 seconds
2016-09-07T08:17:14Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
Timeout: The DNS operation timed out after 30.0013720989 seconds
2016-09-07T08:17:14Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
+ code=0

ipa-4-3:

  • 2d011b9 Start named during configuration upgrade.
  • 93756dc Catch DNS exceptions during emptyzones named.conf upgrade

ipa-4-4:

  • afeb4bd Start named during configuration upgrade.
  • e39cc53 Catch DNS exceptions during emptyzones named.conf upgrade

master:

  • 22fd6f0 Start named during configuration upgrade.
  • 271a4f0 Catch DNS exceptions during emptyzones named.conf upgrade

Metadata Update from @pspacek:
- Issue assigned to mbasti
- Issue set to the milestone: FreeIPA 4.3.3

3 years ago

Login to comment on this ticket.

Metadata