#8138 [CI] Builds stuck in f32-updates-candidate, no bodhi updates created
Closed: Fixed 2 months ago by churchyard. Opened 2 months ago by churchyard.

It happens again, builds are stuck in f32-updates-candidate for hours:

https://koji.fedoraproject.org/koji/builds?inherited=0&tagID=11517&order=-build_id&latest=1

Can this be related to the f31 bodhi activation point? Is there some another batch of extensive signing?


There are no updates in https://bodhi.fedoraproject.org/updates/?releases=F32&status=stable since UTC ~midnight.

@kevin could you check if robosignatory is still running ? or just busy ?

Yeah, we have some breakage:

Aug 27 14:47:59 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:47:59][fedmsg.consumers    INFO]   enabled by config  - robosignatory.atomicconsumer:AtomicSignerConsumer
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][moksha.hub    INFO
] Blocking mode false for <robosignatory.atomicconsumer.AtomicSignerConsumer object at 0x7fd7e4d25a50>.  Mess
ages to be queued and distributed to 1 threads.
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][fedmsg.consumers  
  INFO] Backlog handling setup.  status: '/var/run/fedmsg/status/fedmsg-hub/AtomicSignerConsumer', url: 'https://apps.fedoraproject.org/datagrepper/raw'
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][robosignatory.atom
icconsumer    INFO] AtomicSignerConsumer ready for service
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][fedmsg.consumers    INFO]   enabled by config  - robosignatory.tagconsumer:TagSignerConsumer
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][moksha.hub    INFO] Blocking mode false for <robosignatory.tagconsumer.TagSignerConsumer object at 0x7fd7dd430850>.  Messages t
o be queued and distributed to 1 threads.
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][fedmsg.consumers    INFO] Backlog handling setup.  status: '/var/run/fedmsg/status/fedmsg-hub/TagSignerConsumer', url: 'https:/
/apps.fedoraproject.org/datagrepper/raw'
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: [2019-08-27 14:48:00][moksha.hub   ERROR] Failed to init <class 'robosignatory.tagconsumer.TagSignerConsumer'> consumer.
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: Traceback (most recent call last):
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: File "/usr/lib/python2.7/site-packages/moksha/hub/hub.py", line 410, in __init_consumers
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: c = c_class(self)
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: File "/usr/lib/python2.7/site-packages/robosignatory/tagconsumer.py", line 61, in __init__
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: raise Exception('From detected twice: %s
' % tag['from'])
Aug 27 14:48:00 autosign01.phx2.fedoraproject.org fedmsg-hub[16420]: Exception: From detected twice: f31-sign
ing-pending

I'm not sure what it means... ;(

Can I create buildroot overrides to unblock myself?

Ah, fixed it.

I'll retag things...

I've retagged everything I think. Please let me know if you see anything still stuck (although it will take it a bit to chew through all the backlog).

:ideograph_advantage:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 months ago

Thanks @kevin. Is there anything that should be done to prevent this in the future? What was the cause?

It was a syntax error in a change made last night for beta freeze.

I guess closer scrutiny of those commits would be good...

I was thinking about some kind of monitoring.

We could monitor singing messages, but the koji fedora-messaging plugin seems to have broken those and they aren't sent anymore. If we can fix that we could easily alert on say 'no signed builds in 6 hours' or something.

Should I open a followup issue?

Metadata Update from @churchyard:
- Issue status updated to: Open (was: Closed)

2 months ago

(sorry reopened by accident)

Metadata Update from @churchyard:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 months ago

Login to comment on this ticket.

Metadata