I was looking at the sync script @kevin used in issue you mentioned. At least I assume ansible/playbooks/manual/staging-sync/koji.yml is the correct one. It seems that it actually redeploys whole builder DB on staging. I will wait for @kevin first, maybe it's different issue.

So, the current issue seems to be that the s390x builder wasn't processing, so there's a long line of jobs waiting for that.

I rebooted it and it's processing again.

I then looked and don't see any osbuild jobs pending. Can you try some more and see if it's still not picking them up?

New task -> https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=120070365

ok. I think I figured it out and fixed it...

needed to create the db index that was dropped in the last koji update... we did in prod, but not staging.

I think its processing again normally now... please retest.

I triggered https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=120070769

Unfortunately, it seems that it still doesn't work :confused:

So, there's still something going on for sure. ;(

But to avoid this issue for osbuild until I sort it out... I have removed the buildvm-x86-01.stg builder from the 'default' channel. This means it won't do any of those other builds, and should process osbuild tasks pretty quickly.

Please re-open if you still see this.