It's the weaker ZVM builders (builders 1-14) that don't have enough disk space to build webkitgtk. Right now it's about 50% of chance of hitting the right s390x builder that can actually complete the builds (builders 15 and onwards build webkitgtk just fine).

They both had around 100 GB of free space and that wasn't enough. Is there any chance that it can be increased? 50 GB extra should work for now, I think?

Storage (chroot, cache_topdir):
Filesystem      Size  Used Avail Use% Mounted on
/dev/md127      111G   12G   99G  11% /
/dev/md127      111G   12G   99G  11% /

Storage (chroot, cache_topdir):
Filesystem      Size  Used Avail Use% Mounted on
/dev/md127      111G   14G   98G  13% /
/dev/md127      111G   14G   98G  13% /

It's the weaker ZVM builders (builders 1-14) that don't have enough disk space to build webkitgtk. Right now it's about 50% of chance of hitting the right s390x builder that can actually complete the builds (builders 15 and onwards build webkitgtk just fine).

They both had around 100 GB of free space and that wasn't enough. Is there any chance that it can be increased? 50 GB extra should work for now, I think?

I doubt it, but I can ask...

Turn off some of the ZVM builders? Seems clear that avoiding the heavy builder channel requires beefier standard builders. We're not going to be able to reduce disk space requirements like we were with RAM.

Turning a few off would just mean more load on the larger ones... if we turned them all off, it would mean all those builds that could finish on them go to the larger ones and you would likely have to wait for one... ;(

I've asked how possible adding more disk space is. Failing that I could lower the capacity I suppose... if it was enough lower it should prevent webkitgtk builds landing on them...

Turning a few off would just mean more load on the larger ones... if we turned them all off, it would mean all those builds that could finish on them go to the larger ones and you would likely have to wait for one... ;(

I've asked how possible adding more disk space is. Failing that I could lower the capacity I suppose... if it was enough lower it should prevent webkitgtk builds landing on them...

If I understood a previous message from Kevin on this properly, turning off zVM systems will not allow for more disk space to be available for the others. Each one is a different system and allocations are done in an older system.

Failing that I could lower the capacity I suppose... if it was enough lower it should prevent webkitgtk builds landing on them...

Time to try this?

I see Kalev managed to complete the builds I had started by using his superpowers, but that's of course not a sustainable solution.

@kevin can you try lowering the capacity please?

Alternatively, just turn off all these lower-capacity builders.

So, looking more closely at this.

The z/vm and kvm builders all have only 100GB disk.

Disk space in koji is complex, because builders store failed builds, bootstrap chroots and buildroots. If a large disk space using build failed on the builder you get before you, it will have a lot less space, because it's keeping that failed build around. buildroots also take up space and we can;t expire them too quickly or builds that are waiting fail when they start by not having the buildroot around anymore. :(

So, I am now puzzled why you are seeing out of disk space only on the z/vm builders, since they have the same space. ;(

Turning off z/vm builders would just result in you having to wait longer because all those other jobs that would have run on them would instead run on the kvm builders you want to use.
Smooge is correct in that the z/vm builders are their own small instances. We cannot reclaim any resources by turning them off. They are smaller than the kvm ones, but they are useful for smaller packages.

I lowered the weight on the z/vm instances to 1.5 perhaps that will help.

The z/vm and kvm builders all have only 100GB disk.

Interesting. I've seen it run out of space a number of times and on all occasions it's been on z/vm builders. It's puzzles me why though because like you say, they all seem to have roughly the same amount of disk space.

So I think we got this solved by adjusting the debuginfo?

Let us know if there's anything further to do...