Multiple test suites are failing when attempting to create ldap connection from controller machine to ipa server. This happens when in test configuration we have different values for hostname and external_hostname attributes of host objects. Such configuration with different internal and external hostnames is quite common and described in https://github.com/encukou/pytest-multihost.
Affected testuites are inipatests/test_integration:
Stacktrace for one of the failed tests:
___________________________________________________________________________________________________ TestClientInstall.test_client_install ___________________________________________________________________________________________________ self = <ipapython.ipaldap.LDAPClient object at 0x7fe38d072908>, arg_desc = None @contextlib.contextmanager def error_handler(self, arg_desc=None): """Context manager that handles LDAPErrors """ try: try: > yield freeipa/ipapython/ipaldap.py:1076: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <ipapython.ipaldap.LDAPClient object at 0x7fe38d072908> def _connect(self): with self.error_handler(): conn = ldap_initialize(self.ldap_uri, cacertfile=self._cacert) # SASL_NOCANON is set to ON in Fedora's default ldap.conf and # in the ldap_initialize() function. if not self._sasl_nocanon: conn.set_option(ldap.OPT_X_SASL_NOCANON, ldap.OPT_OFF) if self._start_tls and self.protocol == 'ldap': # STARTTLS applies only to ldap:// connections > conn.start_tls_s() freeipa/ipapython/ipaldap.py:1204: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <ldap.ldapobject.SimpleLDAPObject object at 0x7fe38d072780> def start_tls_s(self): """ start_tls_s() -> None Negotiate TLS with server. The `version' attribute must have been set to VERSION3 before calling start_tls_s. If TLS could not be started an exception will be raised. """ > return self._ldap_call(self._l.start_tls_s) /usr/lib64/python3.6/site-packages/ldap/ldapobject.py:864: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <ldap.ldapobject.SimpleLDAPObject object at 0x7fe38d072780>, func = <built-in method start_tls_s of LDAP object at 0x7fe38d0a92b0>, args = (), kwargs = {}, diagnostic_message_success = None, exc_type = None, exc_value = None exc_traceback = None def _ldap_call(self,func,*args,**kwargs): """ Wrapper method mainly for serializing calls into OpenLDAP libs and trace logs """ self._ldap_object_lock.acquire() if __debug__: if self._trace_level>=1: self._trace_file.write('*** %s %s - %s\n%s\n' % ( repr(self), self._uri, '.'.join((self.__class__.__name__,func.__name__)), pprint.pformat((args,kwargs)) )) if self._trace_level>=9: traceback.print_stack(limit=self._trace_stack_limit,file=self._trace_file) diagnostic_message_success = None try: try: result = func(*args,**kwargs) if __debug__ and self._trace_level>=2: if func.__name__!="unbind_ext": diagnostic_message_success = self._l.get_option(ldap.OPT_DIAGNOSTIC_MESSAGE) finally: self._ldap_object_lock.release() except LDAPError as e: exc_type,exc_value,exc_traceback = sys.exc_info() try: if 'info' not in e.args[0] and 'errno' in e.args[0]: e.args[0]['info'] = strerror(e.args[0]['errno']) except IndexError: pass if __debug__ and self._trace_level>=2: self._trace_file.write('=> LDAPError - %s: %s\n' % (e.__class__.__name__,str(e))) try: > reraise(exc_type, exc_value, exc_traceback) /usr/lib64/python3.6/site-packages/ldap/ldapobject.py:329: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ exc_type = <class 'ldap.CONNECT_ERROR'>, exc_value = CONNECT_ERROR({'desc': 'Connect error', 'info': 'TLS: hostname does not match CN in peer certificate'},), exc_traceback = <traceback object at 0x7fe38d077848> def reraise(exc_type, exc_value, exc_traceback): """Re-raise an exception given information from sys.exc_info() Note that unlike six.reraise, this does not support replacing the traceback. All arguments must come from a single sys.exc_info() call. """ # In Python 3, all exception info is contained in one object. > raise exc_value /usr/lib64/python3.6/site-packages/ldap/compat.py:44: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <ldap.ldapobject.SimpleLDAPObject object at 0x7fe38d072780>, func = <built-in method start_tls_s of LDAP object at 0x7fe38d0a92b0>, args = (), kwargs = {}, diagnostic_message_success = None, exc_type = None, exc_value = None exc_traceback = None def _ldap_call(self,func,*args,**kwargs): """ Wrapper method mainly for serializing calls into OpenLDAP libs and trace logs """ self._ldap_object_lock.acquire() if __debug__: if self._trace_level>=1: self._trace_file.write('*** %s %s - %s\n%s\n' % ( repr(self), self._uri, '.'.join((self.__class__.__name__,func.__name__)), pprint.pformat((args,kwargs)) )) if self._trace_level>=9: traceback.print_stack(limit=self._trace_stack_limit,file=self._trace_file) diagnostic_message_success = None try: try: > result = func(*args,**kwargs) E ldap.CONNECT_ERROR: {'desc': 'Connect error', 'info': 'TLS: hostname does not match CN in peer certificate'} /usr/lib64/python3.6/site-packages/ldap/ldapobject.py:313: CONNECT_ERROR During handling of the above exception, another exception occurred: self = <ipatests.test_integration.test_caless.TestClientInstall object at 0x7fe38f5cd3c8> def test_client_install(self): "IPA client install" self.create_pkcs12('ca1/server') self.prepare_cacert('ca1') result = self.install_server() assert result.returncode == 0 self.clients[0].run_command(['ipa-client-install', '--domain', self.master.domain.name, '--server', self.master.hostname, '-p', self.master.config.admin_name, '-w', self.master.config.admin_password, '-U']) > self.verify_installation() freeipa/ipatests/test_integration/test_caless.py:1159: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ freeipa/ipatests/test_integration/test_caless.py:385: in verify_installation ldap = host.ldap_connect() freeipa/ipatests/pytest_ipa/integration/host.py:58: in ldap_connect cacert=f.name freeipa/ipapython/ipaldap.py:824: in from_hostname_secure return cls(uri, start_tls=start_tls, cacert=cacert, **kwargs) freeipa/ipapython/ipaldap.py:794: in __init__ self._conn = self._connect() freeipa/ipapython/ipaldap.py:1204: in _connect conn.start_tls_s() /usr/lib64/python3.6/contextlib.py:99: in __exit__ self.gen.throw(type, value, traceback) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <ipapython.ipaldap.LDAPClient object at 0x7fe38d072908>, arg_desc = None @contextlib.contextmanager def error_handler(self, arg_desc=None): """Context manager that handles LDAPErrors """ try: try: yield except ldap.TIMEOUT: raise errors.DatabaseTimeout() except ldap.LDAPError as e: desc = e.args[0]['desc'].strip() info = e.args[0].get('info', '').strip() if arg_desc is not None: info = "%s arguments: %s" % (info, arg_desc) raise except ldap.NO_SUCH_OBJECT: raise errors.NotFound(reason=arg_desc or 'no such entry') except ldap.ALREADY_EXISTS: # entry already exists raise errors.DuplicateEntry() except ldap.TYPE_OR_VALUE_EXISTS: # attribute type or attribute value already exists, usually only # occurs, when two machines try to write at the same time. raise errors.DuplicateEntry(message=desc) except ldap.CONSTRAINT_VIOLATION: # This error gets thrown by the uniqueness plugin _msg = 'Another entry with the same attribute value already exists' if info.startswith(_msg): raise errors.DuplicateEntry() else: raise errors.DatabaseError(desc=desc, info=info) except ldap.INSUFFICIENT_ACCESS: raise errors.ACIError(info=info) except ldap.INVALID_CREDENTIALS: raise errors.ACIError(info="%s %s" % (info, desc)) except ldap.INAPPROPRIATE_AUTH: raise errors.ACIError(info="%s: %s" % (desc, info)) except ldap.NO_SUCH_ATTRIBUTE: # this is raised when a 'delete' attribute isn't found. # it indicates the previous attribute was removed by another # update, making the oldentry stale. raise errors.MidairCollision() except ldap.INVALID_SYNTAX: raise errors.InvalidSyntax(attr=info) except ldap.OBJECT_CLASS_VIOLATION: raise errors.ObjectclassViolation(info=info) except ldap.ADMINLIMIT_EXCEEDED: raise errors.AdminLimitExceeded() except ldap.SIZELIMIT_EXCEEDED: raise errors.SizeLimitExceeded() except ldap.TIMELIMIT_EXCEEDED: raise errors.TimeLimitExceeded() except ldap.NOT_ALLOWED_ON_RDN: raise errors.NotAllowedOnRDN(attr=info) except ldap.FILTER_ERROR: raise errors.BadSearchFilter(info=info) except ldap.NOT_ALLOWED_ON_NONLEAF: raise errors.NotAllowedOnNonLeaf() except ldap.SERVER_DOWN: raise errors.NetworkError(uri=self.ldap_uri, error=info) except ldap.LOCAL_ERROR: raise errors.ACIError(info=info) except ldap.SUCCESS: pass except ldap.CONNECT_ERROR: > raise errors.DatabaseError(desc=desc, info=info) E ipalib.errors.DatabaseError: Connect error: TLS: hostname does not match CN in peer certificate freeipa/ipapython/ipaldap.py:1136: DatabaseError
The problem appeared after commit https://pagure.io/freeipa/c/1a2ceb155759e6640ed8e07fdaa0da02ac1b60e9 This commit changed Host.ldap_connect method to use secure connection to ldap server. Connection fails because controller uses external_hostname to connect to ldap server and validate its certificate, but the certificate was created for internal hostname.
We can not simply use plain connection because after https://pagure.io/freeipa/c/5be9341fbabaf7bcb396a2ce40f17e1ccfa54b77 the LDAPClient.simple_bind method forbids to use password authentication over non-secure connection.
@cheimes, do you have ideas how could we solve the issue?
One possibility is to execute the ldapsearch on the remote host (master, client or replica). Most straightforward would probably be the host with the LDAP server. But in this case, it might be a bit harder to process the result.
@pvoborni In this case we will need to rewrite all the tests that use ldap connection. And there are not only searches, but also modifications. I still hope we can rollback to insecure ldap connections between tests controller and ipa server.
I don't agree with the conclusion. The error is: TLS: hostname does not match CN in peer certificate. It means that the hostname does not match the name in the certificate. I suggest to fix the hostname / DNS issue rather than to hack around the error message.
TLS: hostname does not match CN in peer certificate
@cheimes I do not understnad what you mean by "hack around the error message". I want to say that by enforcing secure connection between controller and ipa server we broke support for specific test environments. As I tried to explain in the description, this is not an issue with hostnames or DNS -- it is normal that hostname and external_hostname are different.
I argued that that the difference between hostname and external hostname is a design bug in our testing framework What would it take to fix the actual issue and get rid external hostname?
As you correctly stated, password based bind over a non-secure connection is no longer supported. As soon as realmd is fixed, IPA is going to configure 389-DS for strong authentication again.
If you are looking for a quick and dirty hack, then you could do something like (untested):
if self.master.hostname != self.master.external_hostname: conn.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, 0) conn.set_option(ldap.OPT_X_TLS_NEWCTX, 0)
@cheimes I have not seen the requirement for test machines hostnames to be documented anywhere. The only mentioning is in pytest-multihost docs and it suggests that they can differ. So I think that is not a bug, but a feature. Consider this scenario: while debugging/developing tests I execute integration tests from my laptop. The virtual machines might be located a) in some lab and accessible using names like host-123-212.lab.example.com b) in vagrant and accessible only by ip address The domain name defined during ipa server installation is ipa.test with host names like master.ipa.test, replica1.ipa.test. To make hostname and external_hostname equal I would need to modify my local /etc/hosts each time I setup new ipa test topology. And working with several topolgies in parallel becomes even more complicated.
Will test your proposed fix
The proposed fix does not work because ldap connection is established in LDAPClient constructor and it does not have parameter to pass options to the _connect() method.
Proposed solution: https://github.com/freeipa/freeipa/pull/3217
master:
Metadata Update from @sorlov: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.