To reproduce, just do this from a Python shell on openqa-lab01:
import wikitcms.wiki wk = wikitcms.wiki.Wiki("stg.fedoraproject.org") wk.login()
It hits an exception in openidc_client:
openidc_client
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.10/site-packages/wikitcms/wiki.py", line 338, in login self.site_init() File "/usr/lib/python3.10/site-packages/mwclient/client.py", line 142, in site_init info = self.get('query', meta='userinfo', uiprop='groups|rights') File "/usr/lib/python3.10/site-packages/mwclient/client.py", line 234, in get return self.api(action, 'GET', *args, **kwargs) File "/usr/lib/python3.10/site-packages/mwclient/client.py", line 285, in api info = self.raw_api(action, http_method, **kwargs) File "/usr/lib/python3.10/site-packages/mwclient/client.py", line 437, in raw_api res = self.raw_call('api', data, retry_on_error=retry_on_error, File "/usr/lib/python3.10/site-packages/mwclient/client.py", line 399, in raw_call stream = self.connection.request(http_method, url, **args) File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 515, in request prep = self.prepare_request(req) File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 443, in prepare_request p.prepare( File "/usr/lib/python3.10/site-packages/requests/models.py", line 322, in prepare self.prepare_auth(auth, url) File "/usr/lib/python3.10/site-packages/requests/models.py", line 558, in prepare_auth r = auth(self) File "/usr/lib/python3.10/site-packages/openidc_client/requestsauth.py", line 55, in __call__ token = self.client.get_token(self.scopes, File "/usr/lib/python3.10/site-packages/openidc_client/__init__.py", line 156, in get_token uuid = self._get_new_token(scopes) File "/usr/lib/python3.10/site-packages/openidc_client/__init__.py", line 480, in _get_new_token raise Exception('We were unable to instantiate a webserver') Exception: We were unable to instantiate a webserver
not sure why. The wikitcms auth code is here, the openidc bits were written by Patrick - it basically creates an OpenIDCClient instance, wraps it in an OpenIDCClientAuther instance, and sets that as self.connection.auth, then calls self.site_init().
OpenIDCClient
OpenIDCClientAuther
self.connection.auth
self.site_init()
Prod seems fine, this is only happening on the staging wiki.
I think thats it trying to renew the token?
Not sure tho.
CC: @abompard
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation
I tried to investigate a little and looking at the openidc_client it seems that it is trying to create the server on 0.0.0.0:12345 or 0.0.0.0:23456 when trying to get new token. It seems there is something already listening on both ports:
0.0.0.0:12345
0.0.0.0:23456
# lsof -i -P -n | grep 12345 fedora-me 682111 root 12u IPv4 803275969 0t0 TCP *:12345 (LISTEN) # lsof -i -P -n | grep 23456 fedora-me 682110 root 13u IPv4 803293814 0t0 TCP *:23456 (LISTEN)
And according to the PID it's fedora messaging consumers:
root 682110 0.0 0.0 138272 49596 ? Ssl 04:03 0:01 /usr/bin/python3 /usr/bin/fedora-messaging --conf /etc/fedora-messaging/fedora_openqa_wiki_reporter.toml consume root 682111 0.0 0.0 165248 57880 ? Ssl 04:03 0:01 /usr/bin/python3 /usr/bin/fedora-messaging --conf /etc/fedora-messaging/relvalamiconsumer.toml consume
So, it's possible just restarting that stuff will get it working?
Shall we try that?
So, I did restart things and it does appear working now.
The reproducer in the first step works, and I see it posting results to stg wiki.
So, either it was broken by ports overlapping what it wanted to use, or it was broken by something else and the playbook run fixed it...
Please re-open if you still see this happening or if there's anything further for us to do.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
I also run into this problem on f37 system,the reproducer failed. I checked in f35 and f36 VMs created on the f37 host, the reproducer passed.
This happened again, and rebooting the system cleared it. I do run several different message consumers on that system that all hit the wiki. It seems like they can somehow get 'stuck' blocking the ports used by openidc_client sometimes...
Login to comment on this ticket.