It looks like robosignatory is not processing messages. Rawhide updates are stuck https://bodhi.fedoraproject.org/updates/?search=&releases=F33 and the Fedora CoreOS "next" pipeline is stuck in the "signing OStree" step.
If someone with access could try to restart the consumer that would be great :-)
It may be related to https://github.com/fedora-infra/fedora-messaging/issues/221
Looking at the logs on log01 we can see the following error
Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: Unhandled error in Deferred: Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: Traceback (most recent call last): Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib/python3.6/site-packages/pika/callback.py", line 233, in process Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: callback(*args, **keywords) Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib/python3.6/site-packages/pika/adapters/twisted_connection.py", line 1240, in _on_connection_failed Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: d.errback(exc) Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib64/python3.6/site-packages/twisted/internet/defer.py", line 501, in errback Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: self._startRunCallbacks(fail) Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib64/python3.6/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: self._runCallbacks() Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: --- <exception caught here> --- Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib64/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: current.result = callback(current.result, *args, **kw) Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib/python3.6/site-packages/fedora_messaging/twisted/factory.py", line 337, in on_ready_errback Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: self._client_deferred.errback(wrapped_failure) Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib64/python3.6/site-packages/twisted/internet/defer.py", line 501, in errback Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: self._startRunCallbacks(fail) Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib64/python3.6/site-packages/twisted/internet/defer.py", line 561, in _startRunCallbacks Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: raise AlreadyCalledError Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: twisted.internet.defer.AlreadyCalledError: Jul 16 11:05:16 autosign01.iad2.fedoraproject.org keyctl[3353128]: [fedora_messaging.twisted.protocol INFO] Disconnect requested, but AMQP connection already gone Jul 16 11:05:34 autosign01.iad2.fedoraproject.org keyctl[3353128]: [fedora_messaging.twisted.factory ERROR] The connection failed with an unexpected exception; please report this bug. Jul 16 11:05:34 autosign01.iad2.fedoraproject.org keyctl[3353128]: Traceback (most recent call last): Jul 16 11:05:34 autosign01.iad2.fedoraproject.org keyctl[3353128]: File "/usr/lib64/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks Jul 16 11:05:34 autosign01.iad2.fedoraproject.org keyctl[3353128]: result = g.send(result) Jul 16 11:05:34 autosign01.iad2.fedoraproject.org keyctl[3353128]: StopIteration
any way to get robosignatory out of that state for now?
ssh is disabled on that box and only a few people have access to it (@kevin or @mohanboddu) so we need them to look at it :(
I restarted robosignatory and it seemed to fixed it. Please reopen if you find any issues again.
Thanks.
Metadata Update from @mohanboddu: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.