#10833 armv7hl koji builders randomly get stuck with dnf waiting for I/O
Closed: Fixed with Explanation 2 years ago by kevin. Opened 2 years ago by decathorpe.

Describe what you would like us to do:

Every once in a while, koji builders on armv7hl get stuck, either when dnf is initializing the bootstrap chroot, or when dnf is starting a transaction to install packages, apparently, waiting for I/O (according to @kevin ).

This has now happened ~a dozen times to me, out of a few dozen package builds. The koji tasks on armv7hl appear to be hanging indefinitely (I've waited on some of them for hours), and the only way to make things "go" again is to cancel the task, resubmit it, and hope that it won't get stuck on armv7hl again.

If it helps, here's a list of recent affected tasks:

I don't know if this is "fixable", and with armv7hl support being on its way out, it's probably also low priority. But if this will just "keep happening" on f35 and f36 until they are EOL, it might be a good idea to send a heads-up email to the devel list, with a warning like "if you're thinking, man, that armv7hl build is always slow, but it's reeeeeally slow today, then you might be experiencing X bug, please cancel and resubmit the task".

When do you need this to be done by? (YYYY/MM/DD)


This is not urgent, but it's very annoying, and wastes builder resources on non-armv7hl, when I need to resubmit a task on all architectures when only armv7hl needs to be "unstuck".


They are in fact locking up... :(

I'm trying to track down why this is happening...

Metadata Update from @zlopez:
- Issue assigned to kevin
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation

2 years ago

Koschei builds are starting to be affected too (if the buildSRPMfromSCM task runs on an armv7hl builder, the builds itself don't run there):

https://koji.fedoraproject.org/koji/taskinfo?taskID=90305613
https://koji.fedoraproject.org/koji/taskinfo?taskID=90302329

I can't cancel those because they're started by koschei, please stop them.

Metadata Update from @mizdebsk:
- Issue tagged with: koji

2 years ago

So, good news. Looks like the 5.19.x kernel is much happier on these. I updated them last week and no hangups since then.

So, I am going to consider this closed. Please do reopen or file a new issue if anyone sees this again.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata