As admin , I want remove the first master server with KRA installed to complete the fail over the new master.
Recently, Hard disk drive on the first master failed ( not recoverable), I was able to switch CA renewal to a new replica master server. And while trying to remove the old master from replication topology i encounter the following error
root@tobor-new:280 # ipa-replica-manage del first-master.sample.com -v --force ipa: WARNING: Lookup failed: Preferred host new-master.sample.com does not provide KRA. ipa: INFO: Starting new HTTPS connection (1): first-master.sample.com ipa: INFO: Starting new HTTPS connection (2): first-master.sample.com ('Connection aborted.', error(111, 'Connection refused'))
This error render me from removing first-master.sample.com and also causing new replication to fail trying Search DNS from first-master.sample.com dns server .
is anyone aware of this issue?
Error trying to remove first master. root@new-master:381 # ipa server-del first-master.sample.com --ignore-last-of-role --force Removing first-master.sample.com from replication topology, please wait... ipa: ERROR: an internal error has occurred
root@first-master:280 # ipa-replica-manage del first-master.sample.com -v --force ipa: WARNING: Lookup failed: Preferred host new-master.sample.com does not provide KRA. ipa: INFO: Starting new HTTPS connection (1): first-master.sample.com ipa: INFO: Starting new HTTPS connection (2): first-master.sample.com ('Connection aborted.', error(111, 'Connection refused'))
(what do you expect to happen)
$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
package freeipa-server is not installed package freeipa-client is not installed ipa-server-4.6.6-11.el7.centos.x86_64 ipa-client-4.6.6-11.el7.centos.x86_64 389-ds-base-1.3.10.1-5.el7.x86_64 pki-ca-10.5.17-6.el7.noarch krb5-server-1.15.1-46.el7.x86_64
Log file locations: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/config-files-logs.html Troubleshooting guide: https://www.freeipa.org/page/Troubleshooting
Hi @khoaitaybeo86 Can you provide the versions that you are using: rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
and also the domain-level of your topology: kinit admin ipa domainlevel-get
kinit admin
ipa domainlevel-get
If the domain level is 1, the correct command to remove a server is ipa server-del as described in Linux Domain Identity, Authentication, and Policy Guide and it provides the option --ignore-last-of-role to force removal even if the server to be removed is the only one providing a specific service.
ipa server-del
--ignore-last-of-role
@frenaud Thanks, I have updated my original post with more information.
ipa server-del --ignore-last-of-role also gave me internal error.
For internal errors look to the Apache error log: /var/log/httpd/error_log
@rcritten Thanks for pointing out the error_log, Looks like IPA tried to connect the corrupted first-master.svceng.com then thrown error while not able to connect.
[Sun Jul 05 22:38:01.233321 2020] [:error] [pid 57586] ipa: WARNING: Lookup failed: Preferred host tobor-new.svceng.com does not provide KRA. [Sun Jul 05 22:38:01.245575 2020] [:error] [pid 57586] ipa: INFO: Starting new HTTPS connection (1): first-master.sample.com [Sun Jul 05 22:38:04.264266 2020] [:error] [pid 57586] ipa: INFO: Starting new HTTPS connection (2): first-master.sample.com [Sun Jul 05 22:38:07.272380 2020] [:error] [pid 57586] ipa: ERROR: non-public: ConnectionError: ('Connection aborted.', error(111, 'Connection refused')) [Sun Jul 05 22:38:07.272405 2020] [:error] [pid 57586] Traceback (most recent call last): [Sun Jul 05 22:38:07.272408 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 369, in wsgi_execute [Sun Jul 05 22:38:07.272410 2020] [:error] [pid 57586] result = command(args, options) [Sun Jul 05 22:38:07.272413 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 450, in call [Sun Jul 05 22:38:07.272415 2020] [:error] [pid 57586] return self.__do_call(*args, options) [Sun Jul 05 22:38:07.272418 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 478, in __do_call [Sun Jul 05 22:38:07.272420 2020] [:error] [pid 57586] ret = self.run(args, options) [Sun Jul 05 22:38:07.272423 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 800, in run [Sun Jul 05 22:38:07.272425 2020] [:error] [pid 57586] return self.execute(*args, options) [Sun Jul 05 22:38:07.272428 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/baseldap.py", line 1590, in execute [Sun Jul 05 22:38:07.272430 2020] [:error] [pid 57586] delete_entry(pkey) [Sun Jul 05 22:38:07.272433 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/baseldap.py", line 1541, in delete_entry [Sun Jul 05 22:38:07.272435 2020] [:error] [pid 57586] dn = callback(self, ldap, dn, nkeys, options) [Sun Jul 05 22:38:07.272438 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 751, in pre_callback [Sun Jul 05 22:38:07.272440 2020] [:error] [pid 57586] pkey, ignore_last_of_role=options.get('ignore_last_of_role', False) [Sun Jul 05 22:38:07.272443 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 510, in _ensure_last_of_role [Sun Jul 05 22:38:07.272445 2020] [:error] [pid 57586] vault_config = self.api.Command.vaultconfig_show()['result'] [Sun Jul 05 22:38:07.272448 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 450, in call [Sun Jul 05 22:38:07.272450 2020] [:error] [pid 57586] return self.__do_call(*args, options) [Sun Jul 05 22:38:07.272452 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 478, in __do_call [Sun Jul 05 22:38:07.272463 2020] [:error] [pid 57586] ret = self.run(args, options) [Sun Jul 05 22:38:07.272465 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 800, in run [Sun Jul 05 22:38:07.272467 2020] [:error] [pid 57586] return self.execute(*args, options) [Sun Jul 05 22:38:07.272469 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/ipaserver/plugins/vault.py", line 1001, in execute [Sun Jul 05 22:38:07.272472 2020] [:error] [pid 57586] transport_cert = kra_client.system_certs.get_transport_cert() [Sun Jul 05 22:38:07.272474 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/pki/init.py", line 431, in handler [Sun Jul 05 22:38:07.272476 2020] [:error] [pid 57586] return fn_call(inst, args, kwargs) [Sun Jul 05 22:38:07.272478 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/pki/systemcert.py", line 54, in get_transport_cert [Sun Jul 05 22:38:07.272480 2020] [:error] [pid 57586] response = self.connection.get(url, self.headers) [Sun Jul 05 22:38:07.272482 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/pki/client.py", line 46, in wrapper [Sun Jul 05 22:38:07.272484 2020] [:error] [pid 57586] return func(self, *args, kwargs) [Sun Jul 05 22:38:07.272486 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/pki/client.py", line 159, in get [Sun Jul 05 22:38:07.272488 2020] [:error] [pid 57586] data=payload) [Sun Jul 05 22:38:07.272490 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 498, in get [Sun Jul 05 22:38:07.272492 2020] [:error] [pid 57586] return self.request('GET', url, kwargs) [Sun Jul 05 22:38:07.272494 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request [Sun Jul 05 22:38:07.272496 2020] [:error] [pid 57586] resp = self.send(prep, send_kwargs) [Sun Jul 05 22:38:07.272499 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send [Sun Jul 05 22:38:07.272501 2020] [:error] [pid 57586] r = adapter.send(request, *kwargs) [Sun Jul 05 22:38:07.272503 2020] [:error] [pid 57586] File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send [Sun Jul 05 22:38:07.272505 2020] [:error] [pid 57586] raise ConnectionError(err, request=request) [Sun Jul 05 22:38:07.272507 2020] [:error] [pid 57586] ConnectionError: ('Connection aborted.', error(111, 'Connection refused')) [Sun Jul 05 22:38:07.272759 2020] [:error] [pid 57586] ipa: INFO: [jsonserver_session] admin@SVCENG.COM: server_del/1([u'first-master.sample.com'], version=u'2.231'): InternalError
Apologies for the extreme delay, are you still experiencing this problem?
Metadata Update from @rcritten: - Issue assigned to rcritten
I can reproduce this.
It's failing trying to determine if the last KRA server will be removed because it can't contact the last KRA server.
I think the solution is to depend on the server roles to determine where a KRA is configured.
https://github.com/freeipa/freeipa/pull/5908
master:
Metadata Update from @rcritten: - Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1985069
Issue linked to Bugzilla: Bug 1985069
Metadata Update from @rcritten: - Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1985069 https://bugzilla.redhat.com/show_bug.cgi?id=1985072 (was: https://bugzilla.redhat.com/show_bug.cgi?id=1985069)
ipa-4-9:
Metadata Update from @rcritten: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Metadata Update from @rcritten: - Custom field affects_doc adjusted to on - Custom field knownissue adjusted to on - Issue status updated to: Open (was: Closed)
The previous fix includes server_server=hostname, in the role-find. This limits the search to only the current server. We're trying to see if there are any KRA servers not whether the current server has one.
server_server=hostname,
https://github.com/freeipa/freeipa/pull/6101
Metadata Update from @frenaud: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Metadata Update from @rcritten: - Custom field changelog adjusted to The KRA role search was too narrow resulting in false positives when trying to delete a server with a KRA, resulting in an error that the last KRA was being removed when this was not the case.
Metadata Update from @abbra: - Custom field knownissue reset (from on)
Login to comment on this ticket.