fedora-infrastructure

Clone
Source Code
GIT

#8563 x86_64 builder running out of space
Closed: Fixed 4 years ago by kevin. Opened 4 years ago by ahughes.

Closed: Fixed
Reopen Issue

I'm seeing sporadic cases of our builds of java-11-openjdk failing on x86_64 because of lack of space:

https://koji.fedoraproject.org/koji/getfile?taskID=40390363&volume=DEFAULT&name=build.log&offset=-4000
https://koji.fedoraproject.org/koji/getfile?taskID=40773028&volume=DEFAULT&name=build.log&offset=-4000

The above seem to be from different hosts (buildvm-11.phx2.fedoraproject.org & buildvm-15.phx2.fedoraproject.org) so not sure what the issue is.

Happens in non-scratch builds too:

https://koji.fedoraproject.org/koji/taskinfo?taskID=40608507

So, this could be any one of a number of possible issues. :(

  • Could be when kojid restarts it forgets about the old buildroots it needs to cleanup.
  • koschi might be submitting too many failures and those buildroots could be filling things up.
  • It could be just back luck (a builder has a failure with gcc10 or libreoffice or some other large package and you get that builder before it cleans up that failed buildroot)
  • We may need to adjust the 'keep failed buildroots' setting down some more (we upped it a bunch a while back).
  • something else...

However, I don't think I am going to have time to debug it right now and I don't want builds failing because of it. (I'm traveling then out of office then traveling back)

So, I stopped kojid, cleaned out all old buildroots on all builders, and then restarted kojid. We should likely be ok for quite a while now.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
4 years ago

Login to comment on this ticket.

Metadata
None
None
None
None
Needs Review
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.