Currently, tasks do not delete their workdirs after execution. While this isn't a huge deal for tasks like rpmlint, depcheck can download a lot of stuff which fills up the limited disk space on clients rather quickly.
Clean up the workdir after task execution using one of the following methods: add a cleanup step to the libtaskotron runner add a cleanup step to buildbot that removes everything from /var/tmp/taskotron/*
/var/tmp/taskotron/*
All of our stg clients were out of disk when I looked at them this morning and it looks like they started running out of disk several hours ago.
I could reconfigure the clients to be fewer in number with larger disks but if they're running out of disk space overnight, I'm not sure doubling the disk size would help much.
I've updated the buildmaster config to remove /var/tmp/taskotron before every task, this isn't an ideal solution in my mind since it has the potential to stomp on data needed for triage but it should work for now.
/var/tmp/taskotron
Keeping this open for a bit to make sure that the change actually does what we want it to do
This doesn't quite work because depcheck uses its own tmpdir which is outside /var/tmp/taskotron. Looking into options
I know you know, Tim, but just to make things clear for the ticket's purposes: Since D164, depcheck uses the provided ${workdir}.
${workdir}
In general, I think we should recommend check authors to use ${workdir} and store all files in there, especially if they're large.
Also, our pre-run/post-run step could remove everything older than a day from /var/tmp. That could help a lot even with "not well behaving" checks.
/var/tmp
! In #253#8, @jskladan wrote: I know you know, Tim, but just to make things clear for the ticket's purposes: Since D164, depcheck uses the provided ${workdir}.
Thanks for linking that here, I forgot to do that after filing the ticket yesterday.
! In #253#9, @kparal wrote: In general, I think we should recommend check authors to use ${workdir} and store all files in there, especially if they're large.
That makes sense to me.
I'd hesitate a bit on this until we're sure that nothing else is using /var/tmp. At the moment, our perf monitoring solution has stuff in /var/tmp and from time to time, other things will show up there, as well.
Disk space is going to be less of an issue as we get our new hardware. The current machines have almost no disk but the new machines we're getting have something like 10x higher disk/cpu ratio so we won't be limited to 20G per client machine.
I think that the current set of solutions is working well enough for now, closing
Metadata Update from @tflink: - Issue marked as depending on: #173
Login to comment on this ticket.