#9927 bodhi client fails to authenticate user on staging instance
Closed: Fixed 2 years ago by abompard. Opened 2 years ago by ryanlerch.

On the staging instance of bodh (new version 5.7.0)i, there is a reported issue that the bodhi client does not authenticate:

See: https://github.com/fedora-infra/bodhi/issues/4212


on the surface, this appears to be possibly the same issue encountered here:

https://pagure.io/fedora-infrastructure/issue/9773

on the surface, this appears to be possibly the same issue encountered here:

https://pagure.io/fedora-infrastructure/issue/9773

+1

@abompard did you fix both stg and prod? (I assume though, just double checking :))

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, ops

2 years ago

I did update staging, so it's probably a different issue. I'm not currently able to reproduce it, and it's too old to find in ipsilon's logs. Could someone reproduce it so I can check what's wrong with ipsilon?

@abompard just reproduced:

$ kdestroy -a

$ bodhi updates edit FEDORA-2021-ec209e9464 --type bugfix --staging --debug

Warning: openid_api and staging flags are both set. openid_api will be ignored.


Warning: url and staging flags are both set. url will be ignored.

Username: pingou
Password: 
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 260, in send_request
    output = func(method, **kwargs)
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 88, in _decorator
    raise LoginRequiredError(
fedora.client.LoginRequiredError: https://bodhi.stg.fedoraproject.org/updates/ requires a logged in user

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 117, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 280, in save
    return self.send_request('updates/', verb='POST', auth=True,
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 262, in send_request
    raise AuthError()
fedora.client.AuthError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/bodhi", line 33, in <module>
    sys.exit(load_entry_point('bodhi-client==5.6.1', 'console_scripts', 'bodhi')())
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/__init__.py", line 263, in wrapper
    method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/__init__.py", line 567, in edit
    resp = client.save(**kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 142, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 276, in save
    kwargs['csrf_token'] = self.csrf()
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 117, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 616, in csrf
    self.login(self.username, self.password)
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 303, in login
    response = openid_login(
  File "/usr/lib/python3.9/site-packages/fedora/client/openidproxyclient.py", line 132, in openid_login
    raise ServerError(FEDORA_OPENID_API, response.status_code,
fedora.client.ServerError: ServerError(https://id.stg.fedoraproject.org/api/v1/, 500, Error returned from our POST to ipsilon.)

Do those seeing this have a otp?

try entering your password + otp

Good catch kevin, I couldn't reproduce it until I tried my password without my OTP token. Then I get the same traceback.
We used to not require it, but now that it's all integrated the Bodhi client needs it too.
The error message could be better, though...

Hopefully this commit will make things clearer.

Should we move the "Port Bodhi to OpenID Connect" task up our priority list? ;-)

I'm not sure it's better with OIDC. ;( But I guess ipsilon can't really do much here other than saying 'password wrong' rather than traceback.

I originally reported the bug upstream, but I didn't have time to follow up this thread.

I've just read the latest comments, I do not have any OTP set neither on my prod or staging account, but I could successfully edit an update in prod, while in staging I get the error reply.

Bodhi staging is on 5.7.0, while prod is 5.6.1, but there wasn't any change in the authentication or client code.

I can try again when the above commit is pushed to staging ipsilon, just let me know.

@mattia can you login to https://accounts.stg.fedoraproject.org ? There was a time we redid everything in stg, so you may need to reset your password again?

@kevin I can login both in https://accounts.stg.fedoraproject.org and https://bodhi.stg.fedoraproject.org, so I suppose the password I'm using is fine. But bodhi client in staging still fails to authenticate me.

Ah, so it's the cli only thats the problem? I'm getting a different error now... which i think is related to 9977?

@mattia the authentication issues should be resolved now -- are you able to check again?

@ryanlerch I'm still having the same issue:

$ bodhi updates edit FEDORA-2021-f98216fed6 --type bugfix --staging --debug

Warning: openid_api and staging flags are both set. openid_api will be ignored.


Warning: url and staging flags are both set. url will be ignored.

Username: mattia
Password: 
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 260, in send_request
    output = func(method, **kwargs)
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 88, in _decorator
    raise LoginRequiredError(
fedora.client.LoginRequiredError: https://bodhi.stg.fedoraproject.org/updates/ requires a logged in user

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 117, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 280, in save
    return self.send_request('updates/', verb='POST', auth=True,
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 262, in send_request
    raise AuthError()
fedora.client.AuthError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/bodhi", line 33, in <module>
    sys.exit(load_entry_point('bodhi-client==5.7.0', 'console_scripts', 'bodhi')())
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/__init__.py", line 263, in wrapper
    method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/__init__.py", line 567, in edit
    resp = client.save(**kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 142, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 276, in save
    kwargs['csrf_token'] = self.csrf()
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 117, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 616, in csrf
    self.login(self.username, self.password)
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 303, in login
    response = openid_login(
  File "/usr/lib/python3.9/site-packages/fedora/client/openidproxyclient.py", line 132, in openid_login
    raise ServerError(FEDORA_OPENID_API, response.status_code,
fedora.client.ServerError: ServerError(https://id.stg.fedoraproject.org/api/v1/, 500, Error returned from our POST to ipsilon.)

So, this now gives me:

Invalid tag: entr-4.9-1.fc33 not tagged with any of the following tags ['f32-container-updates-candidate', 'f32-modular-updates-candidate', 'f32-flatpak-updates-candidate', 'epel7-testing-candidate', 'dist-5E-epel-testing-candidate', 'f27-modular-updates-candidate', 'f34-container-updates-candidate', 'eln-updates-candidate', 'f30-modular-updates-candidate', 'f28-modular-updates-candidate', 'f28-container-updates-candidate', 'f30-container-updates-candidate', 'epel8-testing-candidate', 'f30-flatpak-updates-candidate', 'f35-container-updates-candidate', 'f29-modular-updates-candidate', 'f29-container-updates-candidate', 'f29-flatpak-updates-candidate', 'f22-updates-candidate', 'f21-updates-candidate', 'f25-updates-candidate', 'f24-updates-candidate', 'f23-updates-candidate', 'f26-updates-candidate', 'f31-modular-updates-candidate', 'dist-6E-epel-testing-candidate', 'f27-updates-candidate', 'f28-updates-candidate', 'f32-updates-candidate', 'f30-updates-candidate', 'f29-updates-candidate', 'el8-modular-updates-candidate', 'f33-updates-candidate', 'f33-modular-updates-candidate', 'f33-container-updates-candidate', 'f33-flatpak-updates-candidate', 'f31-updates-candidate', 'f31-container-updates-candidate', 'f31-flatpak-updates-candidate', 'f34-updates-candidate', 'f34-modular-updates-candidate', 'f34-flatpak-updates-candidate', 'f35-updates-candidate', 'f32-container-updates-testing', 'f32-modular-updates-testing', 'f32-flatpak-updates-testing', 'epel7-testing', 'dist-5E-epel-testing', 'f27-modular-updates-testing', 'f34-container-updates-testing', 'eln-updates-testing', 'f30-modular-updates-testing', 'f28-modular-updates-testing', 'f28-container-updates-testing', 'f30-container-updates-testing', 'epel8-testing', 'f30-flatpak-updates-testing', 'f35-container-updates-testing', 'f29-modular-updates-testing', 'f29-container-updates-testing', 'f29-flatpak-updates-testing', 'f22-updates-testing', 'f21-updates-testing', 'f25-updates-testing', 'f24-updates-testing', 'f23-updates-testing', 'f26-updates-testing', 'f31-modular-updates-testing', 'dist-6E-epel-testing', 'f27-updates-testing', 'f28-updates-testing', 'f32-updates-testing', 'f30-updates-testing', 'f29-updates-testing', 'epel8-modular-updates-testing', 'f33-updates-testing', 'f33-modular-updates-testing', 'f33-container-updates-testing', 'f33-flatpak-updates-testing', 'f31-updates-testing', 'f31-container-updates-testing', 'f31-flatpak-updates-testing', 'f34-updates-testing', 'f34-modular-updates-testing', 'f34-flatpak-updates-testing', 'f35-updates-testing']

which looks like it authed ok? can you all retry now?

Nope, I still have the same error in return.
Logging into accounts.stg.fedoraproject.org or into Bodhi UI works with same username and password.

$ rpm -q bodhi-client
bodhi-client-5.7.0-1.fc34.noarch

Very weird.

In bodhi logs I see:

2021-07-27 18:37:12,389 INFO  [pyramid_fas_openid.view][MainThread] OpenID Info Status: success                        2021-07-27 18:37:12,389 INFO  [pyramid_fas_openid.view][MainThread] OpenID login successful.                       
2021-07-27 18:37:12,389 INFO  [pyramid_fas_openid.view][MainThread] Callback for storing result: bodhi.server.security:remember_me
...
2021-07-27 18:37:12,390 DEBUG [bodhi.server][MainThread] remember_me: groups = ['provenpackager', 'packager', 'astro-si
g']                                                                                                                    2021-07-27 18:37:12,390 INFO  [bodhi.server][MainThread] mattia successfully logged in
2021-07-27 18:37:12,581 DEBUG [bodhi.server][MainThread] mattia is in provenpackager admin group

Like it logged you in ok...

I think that was related to my login into web UI.

When I try from CLI the logs shows:

2021-07-30 15:39:19,912 INFO  [pyramid_fas_openid.view][MainThread] OpenID Field to search for: openid
2021-07-30 15:39:19,912 INFO  [pyramid_fas_openid.view][MainThread] OpenID URL supplied by user: https://id.stg.fedoraproject.org/ 
2021-07-30 15:39:20,029 INFO  [pyramid_fas_openid.view][MainThread] ax_required: {}
2021-07-30 15:39:20,029 INFO  [pyramid_fas_openid.view][MainThread] ax_optional: {}
2021-07-30 15:39:20,029 INFO  [pyramid_fas_openid.view][MainThread] sreg_required: ['email', 'nickname']
2021-07-30 15:39:20,029 INFO  [pyramid_fas_openid.view][MainThread] sreg_optional: []
2021-07-30 15:39:20,029 INFO  [root][MainThread] Generated checkid_setup request to https://id.stg.fedoraproject.org/openid/  using stateless mode.
2021-07-30 15:39:20,030 INFO  [pyramid_fas_openid.view][MainThread] Realm Name: https://bodhi.stg.fedoraproject.org 
2021-07-30 15:39:20,030 INFO  [pyramid_fas_openid.view][MainThread] Return URL from provider will be: https://bodhi.stg.fedoraproject.org/dologin.html 
2021-07-30 15:39:20,030 INFO  [pyramid_fas_openid.view][MainThread] Redirecting to: https://id.stg.fedoraproject.org/openid/?openid.cla.query_cla=http%3A%2F%2Fadmin.fedoraproject.org%2Faccounts%2Fcla%2Fdone&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.lp.query_membership=proventesters%2Cprovenpackager%2Creleng-team%2Csecurity_respons%2Cpackager%2Cbodhiadmin%2Cvirtmaint-sig%2Ckde-sig%2Ceclipse-sig%2Cinfra-sig%2Cgnome-sig%2Cpython-sig%2Crobotics-sig%2Cqa-tools-sig%2Cnodejs-sig%2Clxqt-sig%2Castro-sig%2Csysadmin-main&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.ns.cla=http%3A%2F%2Ffedoraproject.org%2Fspecs%2Fopen_id%2Fcla&openid.ns.lp=http%3A%2F%2Fns.launchpad.net%2F2007%2Fopenid-teams&openid.ns.sreg=http%3A%2F%2Fopenid.net%2Fextensions%2Fsreg%2F1.1&openid.realm=https%3A%2F%2Fbodhi.stg.fedoraproject.org&openid.return_to=https%3A%2F%2Fbodhi.stg.fedoraproject.org%2Fdologin.html%3Fjanrain_nonce%3D2021-07-30T15%253A39%253A20ZUw1ymL&openid.sreg.required=email%2Cnickname 

If it can be useful to track down the problem, I tried these combinations:

from fedora.client import OpenIdBaseClient
import fedora.client.openidproxyclient

fedora.client.openidproxyclient.FEDORA_OPENID_API = 'https://id.stg.fedoraproject.org/api/v1/'
base_url = 'https://bodhi.stg.fedoraproject.org/'
inst = OpenIdBaseClient(base_url, login_url=base_url+'login')
inst.login('mattia', 'PASSWORD')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 303, in login
    response = openid_login(
  File "/usr/lib/python3.9/site-packages/fedora/client/openidproxyclient.py", line 132, in openid_login
    raise ServerError(FEDORA_OPENID_API, response.status_code,
fedora.client.ServerError: ServerError(https://id.stg.fedoraproject.org/api/v1/, 500, Error returned from our POST to ipsilon.)

fedora.client.openidproxyclient.FEDORA_OPENID_API = 'https://id.fedoraproject.org/api/v1/'
base_url = 'https://bodhi.fedoraproject.org/'
inst = OpenIdBaseClient(base_url, login_url=base_url+'login')
inst.login('mattia','PASSWORD')
<Response [200]>

fedora.client.openidproxyclient.FEDORA_OPENID_API = 'https://id.fedoraproject.org/api/v1/'
base_url = 'https://bodhi.stg.fedoraproject.org/'
inst = OpenIdBaseClient(base_url, login_url=base_url+'login')
inst.login('mattia','PASSWORD')
<Response [200]>

So the problem appears to happen when using
fedora.client.openidproxyclient.FEDORA_OPENID_API = 'https://id.stg.fedoraproject.org/api/v1/'

I've just tried to enable two factor authentication on staging and now I can't login into any staging website (tried with bodhi.stg, src.stg.fp.org and a vagrant development machine of Bodhi).
Authentication in accounts.stg works...

Note that if you have a otp, you MUST append it to your password in the password field.

If you do that does it work?

Note that if you have a otp, you MUST append it to your password in the password field.

If you do that does it work?

No.
I tried to enable OTP and I'm still unable to authenticate through Bodhi CLI (stg) or with the Python example I reported in a previous comment.

I'm able to login in Bodhi web UI (stg) and accounts.stg.fedoraproject.org.

So, summarizing:
- login into accounts.stg.fedoraproject.org works (filling TOTP in the One-time password field)
- login into bodhi.stg.fedoraproject.org works (using password+TOTP in the password field)
- authenticating in Bodhi staging client doesn't work (appening TOTP to the password)
- authenticating with a sample openID client fail (appening TOTP to the password)

Either I'm doing something wrong, or there's something wrong with my account.
Does the following code works for anyone? Or is the code itself wrong?

from fedora.client import OpenIdBaseClient
import fedora.client.openidproxyclient

fedora.client.openidproxyclient.FEDORA_OPENID_API = 'https://id.stg.fedoraproject.org/api/v1/'
base_url = 'https://bodhi.stg.fedoraproject.org/'
inst = OpenIdBaseClient(base_url, login_url=base_url+'login')
inst.login('<USERNAME>', '<PASSWORD+TOTP>')

@abompard do you have any further ideas here?

Do you have anything in ~/.fedora/openidbaseclient-sessions.cache ? does moving that aside change anything?

Yes, that code works fine for me here. ;(

Do you have anything in ~/.fedora/openidbaseclient-sessions.cache ? does moving that aside change anything?

nope :-(

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 303, in login
    response = openid_login(
  File "/usr/lib/python3.9/site-packages/fedora/client/openidproxyclient.py", line 132, in openid_login
    raise ServerError(FEDORA_OPENID_API, response.status_code,
fedora.client.ServerError: ServerError(https://id.stg.fedoraproject.org/api/v1/, 500, Error returned from our POST to ipsilon.)

The code snippet works fine here too. Could you retry please? What username are you using, is it mattia?
Could you ping me on IRC/Matrix so we can check it out together, please? I'm @abompard there too, you'll find me in #fedora-apps or #fedora-infra or #fedora-aaa.
Thanks!

The code snippet works fine here too. Could you retry please? What username are you using, is it mattia?

Yes.

Could you ping me on IRC/Matrix so we can check it out together, please? I'm @abompard there too, you'll find me in #fedora-apps or #fedora-infra or #fedora-aaa.

Let me know when you're likely to be online. I can usually connect only between 5 and 7 p.m. UTC during business days.

Hey. Did you all get together and get anywhere with this?

OK, we worked on it with mattia today and found the error. The fix is this simple commit, it would crash on users who did not have a timezone set. I deployed it to staging and prod.

Metadata Update from @abompard:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog