Sometimes vault gets stuck in a try/repeat loop.
Version: freeipa-server-4.5.90.dev201703211543+git3de0970-0.fc25.x86_64 on Fedora 25 with @freeipa master copr.
freeipa-server-4.5.90.dev201703211543+git3de0970-0.fc25.x86_64
ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ipa: INFO: trying https://HOST/ipa/session/json ipa: INFO: Forwarding 'vault_add_internal/1' to json server 'https://HOST/ipa/session/json' ... ^CProcess PoolWorker-1: Traceback (most recent call last): File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python2.7/multiprocessing/pool.py", line 113, in worker result = (True, func(*args, **kwds)) File "./vault_perftest.py", line 27, in vault_populate api.Command.vault_add(KEY, ipavaulttype=u"standard") File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 447, in __call__ return self.__do_call(*args, **options) File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 475, in __do_call ret = self.run(*args, **options) File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 1199, in run return self.forward(*args, **options) File "/usr/lib/python2.7/site-packages/ipaclient/plugins/vault.py", line 350, in forward response = self.api.Command.vault_add_internal(*args, **options) File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 447, in __call__ return self.__do_call(*args, **options) File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 475, in __do_call ret = self.run(*args, **options) File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 798, in run return self.forward(*args, **options) File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 821, in forward *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) ...text File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1102, in forward return self.forward(name, *args, **kw) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1100, in forward serverproxy = self.create_connection(os.environ.get('KRB5CCNAME'), self.env.verbose, self.env.fallback, self.env.delegate) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 973, in create_connection urls = self.get_url_list(rpc_uri) File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 829, in get_url_list answers = resolver.query(name, rdatatype.SRV) File "/usr/lib/python2.7/site-packages/dns/resolver.py", line 1132, in query raise_on_no_answer, source_port) File "/usr/lib/python2.7/site-packages/dns/resolver.py", line 1044, in query time.sleep(sleep_time) KeyboardInterrupt
2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:52:48 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:52:49 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:52:49 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:52:58 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:07 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:07 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:07 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:17 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:26 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:36 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:59 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:59 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:59 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:59 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:59 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:53:59 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469 2620:52:0:224e:b201:663a:4b86:3d9a - - [21/Mar/2017:17:54:09 +0100] "POST /ipa/session/json HTTP/1.1" 401 1469
@cheimes do you have machine with reproducer at hand or better steps. Seems to me duplicate of #6775 but @tkrizek failed to reproduce it.
No, I don't have a reproducer any more. It all points to a problem that is caused by a bad Kerberos ccache or session. Neither ipactl restart nor cleanup of ~/.ipa or ~/.cache/ipa fixed the problem. The problem went away after I acquired a new TGT with kinit.
ipactl restart
~/.ipa
~/.cache/ipa
kinit
Duplicate of #6775
Metadata Update from @cheimes: - Issue close_status updated to: duplicate - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.