It happens again, builds are stuck in f32-updates-candidate for hours:
https://koji.fedoraproject.org/koji/builds?inherited=0&tagID=11517&order=-build_id&latest=1
Can this be related to the f31 bodhi activation point? Is there some another batch of extensive signing?
There are no updates in https://bodhi.fedoraproject.org/updates/?releases=F32&status=stable since UTC ~midnight.
@kevin could you check if robosignatory is still running ? or just busy ?
Yeah, we have some breakage:
Aug 27 14:47:59 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:47:59][fedmsg.consumers INFO] enabled by config - robosignatory.atomicconsumer:AtomicSignerConsumer Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][moksha.hub INFO ] Blocking mode false for <robosignatory.atomicconsumer.AtomicSignerConsumer object at 0x7fd7e4d25a50>. Mess ages to be queued and distributed to 1 threads. Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][fedmsg.consumers INFO] Backlog handling setup. status: '/var/run/fedmsg/status/fedmsg-hub/AtomicSignerConsumer', url: 'https://apps.fedoraproject.org/datagrepper/raw' Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][robosignatory.atom icconsumer INFO] AtomicSignerConsumer ready for service Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][fedmsg.consumers INFO] enabled by config - robosignatory.tagconsumer:TagSignerConsumer Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][moksha.hub INFO] Blocking mode false for <robosignatory.tagconsumer.TagSignerConsumer object at 0x7fd7dd430850>. Messages t o be queued and distributed to 1 threads. Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][fedmsg.consumers INFO] Backlog handling setup. status: '/var/run/fedmsg/status/fedmsg-hub/TagSignerConsumer', url: 'https:/ /apps.fedoraproject.org/datagrepper/raw' Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][moksha.hub ERROR] Failed to init <class 'robosignatory.tagconsumer.TagSignerConsumer'> consumer. Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: Traceback (most recent call last): Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: File "/usr/lib/python2.7/site-packages/moksha/hub/hub.py", line 410, in __init_consumers Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: c = c_class(self) Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: File "/usr/lib/python2.7/site-packages/robosignatory/tagconsumer.py", line 61, in __init__ Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: raise Exception('From detected twice: %s ' % tag['from']) Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: Exception: From detected twice: f31-sign ing-pending
I'm not sure what it means... ;(
Can I create buildroot overrides to unblock myself?
Ah, fixed it.
I'll retag things...
I've retagged everything I think. Please let me know if you see anything still stuck (although it will take it a bit to chew through all the backlog).
:ideograph_advantage:
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Thanks @kevin. Is there anything that should be done to prevent this in the future? What was the cause?
It was a syntax error in a change made last night for beta freeze.
I guess closer scrutiny of those commits would be good...
I was thinking about some kind of monitoring.
We could monitor singing messages, but the koji fedora-messaging plugin seems to have broken those and they aren't sent anymore. If we can fix that we could easily alert on say 'no signed builds in 6 hours' or something.
Should I open a followup issue?
Metadata Update from @churchyard: - Issue status updated to: Open (was: Closed)
(sorry reopened by accident)
Metadata Update from @churchyard: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.