freeipa

Clone
Source Code
GIT

#7318 Cannot uninstall ipaserver after fresh install - {'desc': "Can't contact LDAP server", 'errno': 111, 'info': 'Connection refused'}
Closed: fixed 7 years ago Opened 7 years ago by mreznik.

Closed: fixed
Reopen Issue

Cannot uninstall ipaserver after fresh install. Getting:

[root@master freeipa]# ipa-server-install --uninstall -U

WARNING: Failed to connect to Directory Server to find information about
replication agreements. Uninstallation will continue despite the possible
existing replication agreements.

If this server is the last instance of CA,
KRA, or DNSSEC master, uninstallation may result in data loss.
Shutting down all IPA services
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
ipapython.install.common: ERROR    {'desc': "Can't contact LDAP server", 'errno': 111, 'info': 'Connection refused'}

Steps to Reproduce:

  1. ipa-server-install
  2. ipa-server-install --uninstall

Actual behavior:

Server cannot be uninstalled. Even after several attempts.

Expected behavior:

Server is uninstalled fine.

Version/Release/Distribution:

Fedora 27

[root@master freeipa]# rpm -q freeipa-server freeipa-client 389-ds-base pki-ca krb5-server
freeipa-server-4.6.90.dev201712120625+git7fbbf6689-0.fc27.x86_64
freeipa-client-4.6.90.dev201712120625+git7fbbf6689-0.fc27.x86_64
389-ds-base-1.3.7.8-1.fc27.x86_64
pki-ca-10.5.1-2.fc27.noarch
krb5-server-1.15.2-4.fc27.x86_64

Traceback from ipaserver-uninstall.log:

2017-12-12T07:09:04Z DEBUG stderr=
2017-12-12T07:09:04Z DEBUG Starting external process
2017-12-12T07:09:04Z DEBUG args=/bin/systemctl stop ipa-custodia.service
2017-12-12T07:09:04Z DEBUG Process finished, return code=0
2017-12-12T07:09:04Z DEBUG stdout=
2017-12-12T07:09:04Z DEBUG stderr=
2017-12-12T07:09:04Z DEBUG Starting external process
2017-12-12T07:09:04Z DEBUG args=/bin/systemctl disable ipa-custodia.service
2017-12-12T07:09:04Z DEBUG Process finished, return code=0
2017-12-12T07:09:04Z DEBUG stdout=
2017-12-12T07:09:04Z DEBUG stderr=
2017-12-12T07:09:04Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 94, in _handle_execute_exception
    super(Continuous, self)._handle_execute_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 459, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 420, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 417, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 654, in _configure
    next(executor)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 430, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 459, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 517, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 514, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 420, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 417, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 74, in _uninstall
    for _nothing in self._uninstaller(self.parent):
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py", line 589, in main
    uninstall(self)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/install.py", line 250, in decorated
    func(installer)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/install.py", line 1083, in uninstall
    custodiainstance.CustodiaInstance().uninstall()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/custodiainstance.py", line 73, in uninstall
    keystore.remove_server_keys()
  File "/usr/lib/python3.6/site-packages/ipaserver/secrets/kem.py", line 241, in remove_server_keys
    self.remove_keys('host')
  File "/usr/lib/python3.6/site-packages/ipaserver/secrets/kem.py", line 248, in remove_keys
    ldapconn.del_key(KEY_USAGE_SIG, principal)
  File "/usr/lib/python3.6/site-packages/ipaserver/secrets/kem.py", line 169, in del_key
    dn = self._get_dn(usage, principal)
  File "/usr/lib/python3.6/site-packages/ipaserver/secrets/kem.py", line 133, in _get_dn
    return DN(('cn', name), service_rdn, self.keysbase)
  File "/usr/lib/python3.6/site-packages/ipaserver/secrets/kem.py", line 40, in keysbase
    return '%s,%s' % (IPA_REL_BASE_DN, self.basedn)
  File "/usr/lib/python3.6/site-packages/ipaserver/secrets/common.py", line 24, in basedn
    conn = self.connect()
  File "/usr/lib/python3.6/site-packages/ipaserver/secrets/common.py", line 38, in connect
    conn.sasl_interactive_bind_s('', auth_tokens)
  File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 431, in sasl_interactive_bind_s
    return self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
  File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 298, in _ldap_call
    reraise(exc_type, exc_value, exc_traceback)
  File "/usr/lib64/python3.6/site-packages/ldap/compat.py", line 43, in reraise
    raise exc_value
  File "/usr/lib64/python3.6/site-packages/ldap/ldapobject.py", line 282, in _ldap_call
    result = func(*args,**kwargs)
ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server", 'errno': 111, 'info': 'Connection refused'}

2017-12-12T07:09:04Z ERROR {'desc': "Can't contact LDAP server", 'errno': 111, 'info': 'Connection refused'}
2017-12-12T07:09:04Z INFO The ipa-server-install command was successful

Also majority of upstream tests are affected.

Probably caused by:

https://bodhi.fedoraproject.org/updates/FEDORA-2017-ab83b12bb0

Will try to reproduce one more time as /var/log/dirsrv was already cleared.

Attached access/errors logs after installation.

@mreznik I have difficutly to match uninstall logs timestamp with DS logs.
Now something looks weird:

2017-12-12T07:08:59Z DEBUG args=/usr/sbin/remove-ds.pl -i slapd-IPA.TEST
2017-12-12T07:09:03Z DEBUG Process finished, return code=0
....
2017-12-12T07:09:04Z ERROR {'desc': "Can't contact LDAP server", 'errno': 111, 'info': 'Connection refused'}

For me it looks normal that uninstall script can not contact LDAP after removing the instance.

@tbordaz thanks for taking a look. DS logs are from different reproducer as per comment:

Will try to reproduce one more time as /var/log/dirsrv was already cleared.

However it is pretty easily reproducible with current master and COPR repo.

Actually I believe that ldap failure is normal. Indeed before the attempt to connect, the instance is deleted so (unless it is recreated later) it is normal further ldap req are failing.

Is not it a pb in the uninstall script that remove DS too early or access DS after the removal ?

Metadata Update from @mreznik:
- Issue priority set to: critical
- Issue tagged with: testblocker
7 years ago

Seems as a bug in keystore.remove_server_keys() or somewhere else in custodiainstance which cannot handle fact that DS is inaccessible.

Metadata Update from @pvoborni:
- Issue tagged with: regression
7 years ago

Looked at it with Flo, probably caused by: #8700101d982bd3bbf08f32019567edd8f0952538
Edited 7 years ago by pvoborni

Metadata Update from @pvoborni:
- Issue set to the milestone: FreeIPA 4.6.3
7 years ago

Metadata Update from @cheimes:
- Issue assigned to cheimes
7 years ago

I'll take care of the issue first thing on Monday.

Two questions

  • Why was the issue not discovered by our automated tests? Do we need an extra test case to check uninstallation?
  • Is there any way to dump a full traceback on error on stdout, e.g. verbose mode?

@cheimes As stated above: majority of our tests are affected as they keep installing and uninstalling ipa server. E.g. test_caless so the issue was clearly discovered by the automated tests.

@mreznik None of the pull request tests were affected. All tests on Travis and PR-CI have not failed since my bad code landed in master. Otherwise I would have found the bug in my code before the change has been committed.

Can we get a test case with ipa-server-install -- uninstall on PR-CI, please? I'd like to avoid this kind of issue in the future.

master:

  • bfd4e3e Custodia uninstall: Don't fail when LDAP is down

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
7 years ago

ipa-4-6:

  • 7a20cd9 Custodia uninstall: Don't fail when LDAP is down

Log in to comment on this ticket.

Metadata

testblocker regression

None
None
critical
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
Attachments 3
ipaserver-uninstall.log
Attached 7 years ago View Comment
dirsrv_access
Attached 7 years ago View Comment
dirsrv_errors
Attached 7 years ago View Comment
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.