koji seems to have trouble today (as all infra), and at least one build (mine) finished as "completed" but looking inside, the tagBuild task failed and the build is not tagged for f33:
tagBuild
https://koji.fedoraproject.org/koji/taskinfo?taskID=45068078
I tried to pokie koji and retry tagging the build with f33-updates-candidate manually so that bodhi can pick it up, but that task is stuck pending in koji for a while now, too.
f33-updates-candidate
This is the traceback that's shown by koji for the failed tagBuild task:
Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/koji/daemon.py", line 1294, in runTask response = (handler.run(),) File "/usr/lib/python3.7/site-packages/koji/tasks.py", line 311, in run self.createWorkdir() File "/usr/lib/python3.7/site-packages/koji/tasks.py", line 337, in createWorkdir os.makedirs(self.workdir) File "/usr/lib64/python3.7/os.py", line 211, in makedirs makedirs(head, exist_ok=exist_ok) File "/usr/lib64/python3.7/os.py", line 221, in makedirs mkdir(name, mode) OSError: [Errno 30] Read-only file system: '/var/tmp/koji/tasks/8543'
It's a bit strange that it complains about a read-only FS here :interrobang:
It would probably good to check all koji builds from today (or however long the rabbitmq cluster is acting up) whether they were processed correctly or whether they got stuck in untagged koji limbo, waiting to be garbage collected :cry: instead of picked up by bodhi and submitted to stable.
Whenever all other :fire: has been dealt with :)
Metadata Update from @mohanboddu: - Issue assigned to kevin - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: groomed, high-gain, high-trouble
Update: My manually re-trying to tag my borked build succeeded.
However, I suspect that other builds might be affected by similar issues ... (the python 3.9 mass rebuild is happening in f33-python right now, so @churchyard might see some missing builds there as well?)
I've seen a couple of Read-only file system errors in f33-python. But they've always nuked the build.
Sigh. firefox just ate my long detailed comment here. :( Oh well.
THere were some builds that succeded but the tag failed at the end. I resubmitted those tag tasks and those builds should be ok now.
There were some builds that failed, but the top level task didn't fail. I canceled those and resubmitted them.
The read-only tasks were (as far as I can tell) because buildhw-aarch64-07.arm.fedoraproject.org had ext4 errors and a r/o filesystem. I have taken it out of koji now.
Is there any further items you all see we need to fix? Or should we close this now?
From my point of view, things are looking good again.
Thanks for looking into this and resubmitting the broken builds!
ok. Let me know if you see any further issues.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.