I'm seeing more builds failing recently due to running out of disk space. An example:
https://koji.fedoraproject.org/koji/taskinfo?taskID=38424882
Also had one on another arch but don't have that build.
That is weird.. these boxes have 125 GB of disk space with 75+ GB free
/dev/vda1 125G 43G 77G 36% /
Add in various compilers etc in a mock.. and I would think you would have 70GB to work with. On the other hand that build log is 408 MB in size so that must be a large package. How much does it need?
Is there any sense of how much space this package does need to build? Is it only paraview?
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review)
I found the other build - https://koji.fedoraproject.org/koji/taskinfo?taskID=38442558 this is cross-binutils on s390x. I'm trying to get stats on the paraview build - but I suspect it's less than 70GB.
So it's big, but should be less than 70GB. I've only seen it the twice so far, but since I hadn't seen it before I figured I'd point it out early.
I suspect that mock chroots may not be cleaned up properly on time. All staging Koji builders were out of disk space a few days ago and I had to manually stop kojid and clean old chroots in /var/lib/mock. Now I checked them again and most of them are almost full again.
It might be due to us increasing the failed_buildroot_lifetime to 24 hours? ie, you get N really big builds in 24 hours and things run out of space?
Perhaps we should lower that back down?
Metadata Update from @mizdebsk: - Issue tagged with: koji
I lowered the failed_buildroot_lifetime to 8 hours.
Can you please let us know if you see this again?
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.