Now that taskotron-dev has been migrated to have multiple buildslaves per client VM, buildbot is supposed to be cleaning out /var/tmp/taskotron/<slavename> but we've had two client VMs run out of disk space today due to a bunch of not-removed tmp dirs.
/var/tmp/taskotron/<slavename>
Figure out why the changed "remove tmp" step is not working and fix it so that we're not running out of disk space on the clients so often
This started causing failed jobs again so I stopped ignoring it :)
Turns out the issue was a subtle difference between the ways that you can pass commands to buildbot's [[http://docs.buildbot.net/current/manual/cfg-buildsteps.html#shellcommand|ShellCommand buildstep]]. When the components of the command are passed in as a list of strings, it appears to look for a file which has the name * instead of treating * like a wildcard.
*
I changed the command to be a single interpolated string (which is passed into bash before execution) and the wildcard seems to be interpreted properly - disk space on the clients is not continually filling up.
Ansible Change (run against dev): https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=389305cd9764e719f8a93a5e935f6f54d4b6413b
Closing as this appears to be fixed now, please reopen if the issue comes up again
headdesk So that's why the command was written that way before. :( Should have figured that it was for a reason before rewritting it. Thanks for the fix.
I actually suspect it might have been luck the first time around :). The buildbot docs suggest writing the commands as a list of strings so it made sense to change while refactoring
Metadata Update from @tflink: - Issue tagged with: infrastructure
Login to comment on this ticket.