#388 Buildservice hosed
Closed: Fixed None Opened 11 years ago by ixs.

Buildsystem is hosed right now:

418533 build (dist-f9, devel:mboxgrep-0_7_9-5_fc9): open (xenbuilder4.fedora.phx.redhat.com) -> FAILED: Fault: <fault 1:="" "exceptions.oserror:="" [errno="" 31]="" too="" many="" links:="" '="" mnt="" koji="" work="" tasks="" 418535'"="">

Furthermore please clean up http://koji.fedoraproject.org/koji/buildinfo?buildID=35211 http://koji.fedoraproject.org/koji/buildinfo?buildID=35547 and http://koji.fedoraproject.org/koji/buildinfo?buildID=35554. They've been running for more than a day.

Okay, I've done something which should provide a short-term band-aid to get builds working again.

Basically what's happening is that we're overflowing the maximum files (32k) in a directory for /mnt/koji/work/tasks. This directory was in the past being gc'd I think, but not sure where from. For right now I created ../tasks.old and moved ~1200 of the oldest directories there to get us some breathing room. But we should figure out why the garbage collection isn't happening since the move to the new storage

the old storage was on a netapp. new storage is a ext3 filesystem nfs exported. We need to be and are now more agressive in cleaning up.

Login to comment on this ticket.