#7985 test failure in test_dnssec.py::TestInstallDNSSECLast::()::test_disable_reenable_signing_replica::teardown
Closed: fixed 6 days ago by cheimes. Opened 6 months ago by frenaud.

Issue

Nightly tests recently fail in test_dnssec.py::TestInstallDNSSECLast::()::test_disable_reenable_signing_replica::teardown

For instance see the nightly run on fedora 28: testing_f28 or the nightly run on master: testing_master

transport.py 513 DEBUG RUN ['ipa-server-install', '--uninstall', '-U', '--ignore-topology-disconnect', '--ignore-last-of-role']
transport.py 558 DEBUG invalid 'DNSSec key master': all masters must have DNS server role enabled
transport.py 558 DEBUG The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
transport.py 217 DEBUG Exit code: 1
host.py 117 ERROR stderr: invalid 'DNSSec key master': all masters must have DNS server role enabled

Need to investigate if it's a regression or a test issue. The test fails when uninstalling the replica (master was DNSSEC master and was uninstalled just before).


This could be reproduced in the following way:
1) install ipa master
2) setup dnssec on master
3) install replica
4) force uninstall ipa master
5) on replica ipa -d dnsconfig-show spits out:

ipa: DEBUG: trying https://replica1.ipa.test/ipa/session/json                   
ipa: DEBUG: Created connection context.rpcclient_140151273434704                
ipa: DEBUG: raw: dnsconfig_show(version='2.232')                                
ipa: DEBUG: dnsconfig_show(version='2.232')                                     
ipa: DEBUG: [try 1]: Forwarding 'dnsconfig_show/1' to json server 'https://replica1.ipa.test/ipa/session/json'
ipa: DEBUG: New HTTP connection (replica1.ipa.test)                             
ipa: DEBUG: Destroyed connection context.rpcclient_140151273434704              
ipa: ERROR: invalid 'DNSSec key master': all masters must have DNS server role enabled

Some debugging info:

(Pdb) w                                                                         
  /usr/share/ipa/wsgi.py(59)application()                                       
-> return api.Backend.wsgi_dispatch(environ, start_response)                    
  /usr/lib64/python3/site-packages/ipaserver/rpcserver.py(268)__call__()        
-> return self.route(environ, start_response)                                   
  /usr/lib64/python3/site-packages/ipaserver/rpcserver.py(280)route()           
-> return app(environ, start_response)                                          
  /usr/lib64/python3/site-packages/ipaserver/rpcserver.py(865)__call__()        
-> response = super(jsonserver_session, self).__call__(environ, start_response) 
  /usr/lib64/python3/site-packages/ipaserver/rpcserver.py(474)__call__()        
-> response = super(jsonserver, self).__call__(environ, start_response)         
  /usr/lib64/python3/site-packages/ipaserver/rpcserver.py(430)__call__()        
-> response = self.wsgi_execute(environ)                                        
  /usr/lib64/python3/site-packages/ipaserver/rpcserver.py(370)wsgi_execute()    
-> result = command(*args, **options)                                           
  /usr/lib64/python3/site-packages/ipalib/frontend.py(450)__call__()            
-> return self.__do_call(*args, **options)                                      
  /usr/lib64/python3/site-packages/ipalib/frontend.py(478)__do_call()           
-> ret = self.run(*args, **options)                                             
  /usr/lib64/python3/site-packages/ipalib/frontend.py(800)run()                 
-> return self.execute(*args, **options)                                        
  /usr/lib64/python3/site-packages/ipaserver/plugins/dns.py(4224)execute()      
-> result = super(dnsconfig_show, self).execute(*keys, **options)               
  /usr/lib64/python3/site-packages/ipaserver/plugins/baseldap.py(1340)execute() 
-> self, ldap, entry_attrs.dn, entry_attrs, *keys, **options)                   
  /usr/lib64/python3/site-packages/ipaserver/plugins/dns.py(4230)post_callback()
-> entry_attrs, "DNS server", **options)                                        
  /usr/lib64/python3/site-packages/ipaserver/plugins/config.py(364)show_servroles_attributes()
-> self.update_entry_with_role_config(role, entry_attrs)                        
  /usr/lib64/python3/site-packages/ipaserver/plugins/config.py(344)update_entry_with_role_config()
-> role_config = backend.config_retrieve(role_name)                             
  /usr/lib64/python3/site-packages/ipaserver/plugins/serverroles.py(159)config_retrieve()
-> attr_value = attr.get(self.api)                                              
  /usr/lib64/python3/site-packages/ipaserver/servroles.py(449)get()             
-> masters = super(SingleValuedServerAttribute, self).get(api_instance)         
> /usr/lib64/python3/site-packages/ipaserver/servroles.py(284)get()             
-> try:                                                                         
(Pdb) search_base                                                               
ipapython.dn.DN('cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test')                      
(Pdb) search_filter                                                             
'(&(cn=DNSSEC)(ipaConfigString=dnssecKeyMaster))'                               

> /usr/lib64/python3/site-packages/ipaserver/servroles.py(289)get()             
-> master_cns = {e.dn[1]['cn'] for e in entries}                                
(Pdb) entries                                                                   
[LDAPEntry(ipapython.dn.DN('cn=DNSSEC,cn=master1.ipa.test,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test'), {'ipaConfigString': [b'startOrder 100', b'dnssecKeyMaster', b'enabledService'], 'cn': [b'DNSSEC'], 'objectClass': [b'nsContainer', b'ipaConfigObject', b'top']})]

(Pdb) master_cns                                                                
{'master1.ipa.test'}

As you can see the DNSSEC master is still the just deleted master1.ipa.test.
If bind-dyndb-ldap's design doc [0] is a trusted source of the current implementation then
such a scenario is not supported.

I think this test should reimplement uninstall class method to
remove all the dns-enabled replicas before the uninstallation of their master.

[0] https://docs.pagure.org/bind-dyndb-ldap/BIND9/Design/DNSSEC/Keys/Shortterm.html#replica-uninstallation-replica2

Actually, this happens often for me on Azure.

Instead, in 'PASS' clause:

ipa dnsconfig-show                                           
  Allow PTR sync: TRUE                                                          
  IPA DNS servers: master1.ipa.test, replica1.ipa.test                          
  IPA DNSSec key master: master1.ipa.test                                       

-> try:                                                                         
(Pdb) search_base                                                               
ipapython.dn.DN('cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test')                      
(Pdb) search_filter                                                             
'(&(cn=DNSSEC)(ipaConfigString=dnssecKeyMaster))'                               
(Pdb) n                                                                         
> /usr/lib64/python3/site-packages/ipaserver/servroles.py(284)get()             
-> entries = ldap2.get_entries(search_base, filter=search_filter)               
(Pdb) n                                                                         
> /usr/lib64/python3/site-packages/ipaserver/servroles.py(288)get()             
-> master_cns = {e.dn[1]['cn'] for e in entries}                                
(Pdb) entries                                                                   
[LDAPEntry(ipapython.dn.DN('cn=DNSSEC,cn=master1.ipa.test,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=test'), {'objectClass': [b'nsContainer', b'ipaConfigObject', b'top'], 'cn': [b'DNSSEC'], 'ipaConfigString': [b'startOrder 100', b'dnssecKeyMaster', b'enabledService']})]

(Pdb) associated_role_providers                                                 
{'replica1.ipa.test', 'master1.ipa.test'}                                       

(Pdb) self.associated_role.status(api_instance)                                 
[{'role_servrole': 'DNS server', 'server_server': 'master1.ipa.test', 'status': 'enabled'}, {'role_servrole': 'DNS server', 'server_server': 'replica1.ipa.test', 'status': 'enabled'}]

So, the difference here is:

self.associated_role.status(api_instance)
FAIL: 'master1.ipa.test', 'status': 'absent'
PASS: 'master1.ipa.test', 'status': 'enabled'

Metadata Update from @frenaud:
- Issue assigned to frenaud

10 days ago

Metadata Update from @frenaud:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/3943

10 days ago

master:

  • 7dfc6e0 ipatests: generic uninstall should call ipa server-del
  • 8cf4271 ipatests: fix teardown
  • b3d6503 ipatests: fix test_crlgen_manage

ipa-4-8:

  • c77b062 ipatests: generic uninstall should call ipa server-del
  • ab67e0e ipatests: fix teardown
  • dd06dfc ipatests: fix test_crlgen_manage

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

6 days ago

master:

  • c1272e4 ipatests: fix TestMigrateDNSSECMaster teardown

ipa-4-8:

  • bb8da7d ipatests: fix TestMigrateDNSSECMaster teardown

Login to comment on this ticket.

Metadata