#11422 URGENT: buildvm-ppc64le-06.iad2.fedoraproject.org - `OSError(28, 'No space left on device')`- impacting mass rebuild
Closed: Fixed with Explanation 9 months ago by kevin. Opened 9 months ago by decathorpe.

See $SUBJECT.

Almost one out of three builds is currently failing. Please disable this builder ASAP, otherwise the results of the mass rebuild will be almost unusable (starting around packages "r*").


Sorry was away from my laptop.

Disabled now, will look more closely when I can.

Thank you! Looks like this mostly impacted Rust packages - I resubmitted them and out of the initial ~200 failures ~180 builds passed on the second pass. I haven't checked how bad the impact was on other packages that are alphabetically after "rust-", but looks like you turned off the bored builder fast enough after all :)

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, medium-gain, ops

9 months ago

So, yeah, this is that same weird thing where df says the host has a bunch of free space, but journald and kojid and syslog say it's out of space.

I haven't been able to isolate whats causing it yet. ;(

In the mean time, with no intervention, that host fixed itself and seems fine now, so I can't even use it to debug this. ;(

Will keep an eye out for it again (and please do file a ticket if you see it again) and try and figure it out.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

9 months ago

Metadata Update from @kevin:
- Issue assigned to kevin

9 months ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog