The disk on s390x builders (KVM, 18-23) is too small for building cross-gcc (and possibly for some other packages too). I got a successful build on a zVM builder at one time, but 3 or 4 builds failed on low disk on the KVM ones. The disk sizes differ by ~7GB (KVM = ~96GB, zVM = ~103GB). Another issue might be that the KVM builders have 30 - 60GB disk space already used when the big build starts, while I see only 12 - 17GB used on zVM ones. I wonder if the koji builder daemon garbage collects the failed builds in time or there are some stale data somewhere ...
I am looking to see what is getting held onto this box. my understanding is that is as much disk space we can give per builder on the kvm. We would have to drop a builder and combine two to make a big builder.
20, 21, 22, 23 all are the same in stage. I don't see anything stuck around for a long time.
We went over this in the Fedora review meeting and there isn't much we can do. The solutions are currently either with koji not picking random builders better or needing hardware we don't have. We are going to have to close CANT FIX
Metadata Update from @smooge: - Issue close_status updated to: Will Not/Can Not fix - Issue status updated to: Closed (was: Open) - Issue tagged with: hardware
I see part of the problem now, we keep the failed buildroots for 24h instead of the default of 4h, thus the higher disk usage in /var/lib/mock. Still might be useful to check if it's /var/lib/mock taking all the disk space. I guess the builder itself is under 5GB. I could check it myself, but I lack the permissions due the non-working sudo.
/var/lib/mock
sudo should be working after smooge's freeze break goes out... but yeah, when we had the default maintainers complained that they didnt have enough times to look at logs. ;(
@smooge mentioned 2factor should work today (thanks a lot), so I'll take a look on the builders myself. And yeah, I know about the reasons for longer delay.
an update to document the current state of the things - z/VM builders with 103GB disk are big enough to accommodate the cross-gcc build, they use xfs without reserved space for root, with koji assign-task we can redirect the builds there - KVM builders with 96 GB disk are too small, but they use ext4 with ~5GB (5%) reserved for root, the builds might fit in without the reserved space, but z/VM is a safer option - the cross-gcc buildroot/chroot size is ~85-90 GB
koji assign-task
CC @pbrobinson
Instead of koji assign-task, we could make a zvm channel and hub policy to send all cross-gcc builds to that channel?
Is there anything in the cross-gcc spec that could be tweaked to use less space?
yes, that's an option too
I don't think there is anything except building fewer cross-compilers. Or reducing the debuginfo level.
Login to comment on this ticket.