The two openQA boxes - openqa01.qa and openqa-stg01.qa - both have a key that should allow them to authenticate to the appropriate wiki (prod for openqa01, stg or openqa-stg01). It is installed as /root/.openidc/oidc_wikitcms.json and is stored in the ansible secrets - the staging key is retrieved as {{ private }}/files/openidc/staging/wikitcms.json and the prod key as {{ private }}/files/openidc/production/wikitcms.json.
/root/.openidc/oidc_wikitcms.json
{{ private }}/files/openidc/staging/wikitcms.json
{{ private }}/files/openidc/production/wikitcms.json
On prod everything is fine - openqa01.qa can happily auth to the wiki and do stuff on it. But this isn't working right on staging. When staging tries to login to the wiki with the token file in place, this happens:
>>> import wikitcms.wiki >>> site = wikitcms.wiki.Wiki('stg.fedoraproject.org') >>> site.login() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.7/site-packages/wikitcms/wiki.py", line 214, in login self.site_init() File "/usr/lib/python3.7/site-packages/mwclient/client.py", line 143, in site_init info = self.get('query', meta='userinfo', uiprop='groups|rights') File "/usr/lib/python3.7/site-packages/mwclient/client.py", line 235, in get return self.api(action, 'GET', *args, **kwargs) File "/usr/lib/python3.7/site-packages/mwclient/client.py", line 286, in api info = self.raw_api(action, http_method, **kwargs) File "/usr/lib/python3.7/site-packages/mwclient/client.py", line 434, in raw_api http_method=http_method) File "/usr/lib/python3.7/site-packages/mwclient/client.py", line 405, in raw_call stream.raise_for_status() File "/usr/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://stg.fedoraproject.org/w/api.php?meta=userinfo%7Cuserinfo&uiprop=groups%7Crights%7Cblockinfo%7Chasmsg&continue=&action=query&format=json
after this, the token file is edited to contain only the string {}, and subsequent login attempts give this:
{}
>>> site.login() Please visit https://id.stg.fedoraproject.org/openidc/Authorization?scope=openid+https%3A%2F%2Ffedoraproject.org%2Fwiki%2Fapi&response_type=code&client_id=wikitcms&redirect_uri=http%3A%2F%2Flocalhost%3A23456%2F&response_mode=query to grant authorization
I don't know what's wrong, but this isn't how it's meant to be :)
@kevin @puiterwijk
@codeblock might have some idea (he was looking at issues from our ipsilon upgrade)
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: authentication
Metadata Update from @mizdebsk: - Issue tagged with: staging
Is this still failing ? or was it fixed ?
I disabled the services for now. I can test it again for you.
Same error.
Still happening.
Can you check this again now?
Or do we want to actually have openqa-labs report to the prod wiki in just a different place?
will do. no, I do actually want it to report to the staging wiki, treating staging wiki as a test space. It would be a bit difficult to make it use the real wiki because all the page names are basically hardcoded into python-wikitcms, I'd have to add some awful hack to stick a prefix on them all or something and the way the code is written that'd be a real mess.
oh, it'd probably also mess up some of the wikicode magic if we tried to that, as it has all kinds of transclusions and so on that wouldn't work if the page names were different.
@adamwill could we schedule some time for the two of us to have a look at this ? In case I'm able to help :)
Sorry, didn't get a chance to check it again in new infra with the F33 release. I'll try and check it again this week and we can go from there.
Seems to be behaving exactly the same as described above, still, just tested.
So, I am not sure how this is working at all. ;(
Both the stg and prod tokens show expired. I cannot find the prod one in the db. :(
I think I will need to try and get some @puiterwijk cycles to have him look and tell me what I am missing here.
I suppose it might be interesting to try the prod token in stg (since I copied the prod db to staging it should in theory have the prod token in it)
ok. I have reissued the stg token. Can you run the playbook to pick it up and then test again and see if you can write to the wiki now?
Metadata Update from @kevin: - Issue assigned to kevin
Nope. Still the same. Same errors, token file gets rewritten to {}, all the same.
Prod can log into prod wiki fine. it can't log into stg wiki, it gets given an auth URL to visit.
Metadata Update from @smooge: - Issue tagged with: high-trouble, medium-gain, ops
Is there a easy reproducter I could call to show this?
It's right there in the issue description. As root on openqa-lab01.iad2 , run those three calls in a Python shell. If things are working, site.login() should just complete successfully. You can try the same three calls (only change stg.fedoraproject.org to fedoraproject.org) on openqa01.iad2 to compare (it works fine there).
site.login()
stg.fedoraproject.org
fedoraproject.org
Still broken, still behaves as described initially.
I'm going to look into this again (and others welcome to follow along) next week... say monday the 11th at 17:30UTC. Find me in #fedora-admin looking at it if you want to join.
ok. I think this should be fixed now.
The problem was the .json had:
{"scopes": ["openid", "https://fedoraproject.org/wiki/api"]...
but this was staging! so, changed it to 'https://stg.fedoraproject.org/wiki/api" and it's working now.
:)
So, the auth was working all the time, but it was trying to login to the prod scope. ;)
Metadata Update from @kevin: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Well, it's still not working right. ;(
It now asks you to auth in a browser instead of erroring.
Metadata Update from @kevin: - Issue status updated to: Open (was: Closed)
Perhaps @abompard would take a crack at making this work?
OK I finally found the reason.
Contrary to your latest change @kevin, the scope URL should not be changed: it is an identifer that just happens to look like a URL but is basically a string that should match between what the client requests and what the server has stored. Scopes are like a set of permissions, and the end user may allow a third party app to only do A and B, but not C. (A, B and C being scopes). If the third party app asks for C, it will be denied.
And that's what was happening here actually: the token that was used by wikitcms had been wrongly created with the allowed scope https://stg.fedoraproject.org/wiki/api, including the .stg, in the Ipsilon database. As a result what the client requested and what the server had did not match, and the login was rejected.
https://stg.fedoraproject.org/wiki/api
.stg
Your change in the oidc_wikicms.json didn't actually change anything because I don't think the scope value in this file is actually used in the authentication process, it's more a reminder of what scope the token is allowed to access by the server. The scope that is actually asked by the client is hardcoded in the wikitcms/wiki.py file (which is fine, it is a constant after all).
oidc_wikicms.json
wikitcms/wiki.py
Well, that was hairy, but I learned quite a few things in the process :-)
Adam, please confirm that is works now.
Yup, it looks like it does work now! Thanks a lot. I'll turn the services that use the wiki back on in staging.
Metadata Update from @adamwill: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.