After F30 mass rebuild koji-prunesigs cron job was/is running for more than a day. This causes several instances of the same job running at the same time. This in turns causes OOM on koji02.
I think it would be good to use lock-wrapper to wrap the cron job to make sure at most one instance is running at the same time.
I was going to put this in place a long time ago, but I was told it was fine for multiple jobs to be running at the same time.
I suspect the reason we are hitting OOM now is that mass rebuild just happpened so we signed everything and rebuilt everything so all the old stuff is getting gc'ed...
Anyhow, I think it would still be a good idea. Perhaps we could ask koji folks their thoughts?
@mikem should we put a lock wrapper around the koji-gc jobs? Or no?
Note that this is not about koji-gc jobs, but koji-prunesigs job, that runs koji --noauth prune-signed-copies --verbose
koji --noauth prune-signed-copies --verbose
I have put lock wrappers on each one. We had one for every day since beginning of February running with various ones oom'd out in the past. I think the pile of ones needing to be pruned is really long and needs some time to process.
Metadata Update from @smooge: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.