#9320 Fedora 33 update cannot be pushed to stable, fails with some kind of server timeout
Closed: Fixed 2 days ago by rjones. Opened 3 days ago by rjones.

Describe what you would like us to do:


This update cannot be pushed to stable:

https://bodhi.fedoraproject.org/updates/FEDORA-2020-3f6760cc65

The web UI does nothing, and the command line fails with a lengthy Python error indicating some sort of time-out:

$ bodhi updates request FEDORA-2020-3f6760cc65 stable
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 249, in send_request
    data = output.json()
  File "/usr/lib/python3.9/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib64/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/bodhi", line 11, in <module>
    load_entry_point('bodhi-client==5.2.2', 'console_scripts', 'bodhi')()
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/__init__.py", line 263, in wrapper
    method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/__init__.py", line 690, in request
    resp = client.request(update, state)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 117, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/bodhi/client/bindings.py", line 297, in request
    return self.send_request(f'updates/{update}/request',
  File "/usr/lib/python3.9/site-packages/fedora/client/openidbaseclient.py", line 252, in send_request
    raise ServerError(
fedora.client.ServerError: ServerError(https://bodhi.fedoraproject.org/updates/FEDORA-2020-3f6760cc65/request, 504, Error returned from json module while processing b'https://bodhi.fedoraproject.org/updates/FEDORA-2020-3f6760cc65/request': b'Expecting value: line 1 column 1 (char 0)'
b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n")

When do you need this to be done by? (YYYY/MM/DD)


Ideally before F33 comes out.


The bodhi client seems to fail to authenticate against FAS (which I find rather slow on my side as well)

Sorry for the delay. It is giving the same error as above.

Is this using krb for auth? Anyway I have renewed my krb ticket, and the bodhi client should also have access to the Fedora ssh key, and neither seems to help ...

@rjones Can you try it with the bodhi cli?

Metadata Update from @mohanboddu:
- Issue tagged with: low-gain, low-trouble, ops

3 days ago

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Reporter (was: Needs Review)

3 days ago

bodhi cli means bodhi-client-5.2.2-2.fc33.noarch? That's what I'm
testing above. I tested it again a second ago, but same error.

Sorry for the delay. It is giving the same error as above.

Is this using krb for auth? Anyway I have renewed my krb ticket, and the bodhi client should also have access to the Fedora ssh key, and neither seems to help ...

Bodhi client does not use krb nor ssh keys to authenticate to FAS, you need to provide your FAS password. Once you authenticate successfully once your openid session is cached under the ~/.fedora directory.

Could you try removing the files under ~/.fedora and try again ?

I removed fedora_session and openidbaseclient-sessions.cache from that directory. When rerunning the command I had to enter my username and password which I didn't before. However it still failed with the same error.

Why is the status "waiting on reporter"?

Metadata Update from @pingou:
- Issue priority set to: Waiting on Assignee (was: Waiting on Reporter)

2 days ago

Why is the status "waiting on reporter"?

Because it wasn't changed this morning after you answered

One idea on this one: nginx and openshift both send 504 errors if the server takes too long to answer.

This is a large update (166 builds). So if bodhi takes a little while (more than 60 seconds for nginx, I haven't found the value for openshift yet) to respond, either nginx or openshift will raise that 504 error.

Err, it's not nginx, it's gunicorn that we're using there...

So both gunicorn and openshift have been instructed to wait 180s before killing the request (cf: https://pagure.io/fedora-infra/ansible/c/116987403f154266f17c8eb604b8b3b603cf31cb?branch=master and https://pagure.io/fedora-infra/ansible/c/97db4e4c5f6b3f7276833493ce879ac868659509?branch=master )

Bodhi has been rebuilt in openshift.

@rjones could you give it a try now?

Metadata Update from @pingou:
- Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)

2 days ago
              bodhi - 2020-09-16 13:47:54 (karma 0)
              This update has been submitted for stable by rjones.

Looks good! I guess we can close this now thanks.

Metadata Update from @rjones:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 days ago

Arf, I forgot to ask if you could check how long it took the command to run. Any guess estimate?

Metadata Update from @pingou:
- Issue untagged with: low-gain, low-trouble
- Issue assigned to pingou
- Issue tagged with: medium-gain, medium-trouble

2 days ago

Not sure. It certainly wasn't instant, so maybe a few mins.

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Done