#8309 Builds failing due to running out of disk space
Opened a month ago by orion. Modified 23 days ago

I'm seeing more builds failing recently due to running out of disk space. An example:

https://koji.fedoraproject.org/koji/taskinfo?taskID=38424882

Also had one on another arch but don't have that build.


That is weird.. these boxes have 125 GB of disk space with 75+ GB free

/dev/vda1                                                        125G   43G   77G  36% /

Add in various compilers etc in a mock.. and I would think you would have 70GB to work with. On the other hand that build log is 408 MB in size so that must be a large package. How much does it need?

Is there any sense of how much space this package does need to build? Is it only paraview?

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

a month ago

I found the other build - https://koji.fedoraproject.org/koji/taskinfo?taskID=38442558 this is cross-binutils on s390x. I'm trying to get stats on the paraview build - but I suspect it's less than 70GB.

I found the other build - https://koji.fedoraproject.org/koji/taskinfo?taskID=38442558 this is cross-binutils on s390x. I'm trying to get stats on the paraview build - but I suspect it's less than 70GB.

  • du -sh . /builddir/build/BUILDROOT/paraview-5.6.0-11.fc32.x86_64
    33G .
    11G /builddir/build/BUILDROOT/paraview-5.6.0-11.fc32.x86_64

So it's big, but should be less than 70GB. I've only seen it the twice so far, but since I hadn't seen it before I figured I'd point it out early.

I suspect that mock chroots may not be cleaned up properly on time. All staging Koji builders were out of disk space a few days ago and I had to manually stop kojid and clean old chroots in /var/lib/mock. Now I checked them again and most of them are almost full again.

It might be due to us increasing the failed_buildroot_lifetime to 24 hours? ie, you get N really big builds in 24 hours and things run out of space?

Perhaps we should lower that back down?

Login to comment on this ticket.

Metadata