Lately all the ppc64le builders in koji are OOM-ing randomly. This is a ticket to track the issue and look into it.
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: groomed, medium-gain, medium-trouble
I guess this started after the builders were upgraded to F-32, right?
No, it was happening in phx2 also.
I think we need to rebalance things and now we should be able to. In phx2 we had 2 power9 boxes... in iad2 now we have 4.
Right now buildvm-ppc64le's are 20 per on 2 of the power9 boxes. They are 12288 mem and 5 cpus.
We are seeing some OOMs on the host (killing a guest) and some on guests (killing kojid).
I suspect if we spread them out over the 4 power9 boxes we should have the host side well solved... as for the guest side, perhaps they need more memory for the number of cpus they have?
If we move them to 10 per power9 we should be able to double their memory and cpus... but should we not use all the cpus?
Is there any rule of thumb that might help us here?
In any case we still need to get the other power9's installed first.
ok. I redid ppc64le builders. Now we have 10 on each power9 box.
They have 8 cpus and 20GB mem... that should give more room on the boxes.
Fingers crossed that this is fixed, if not we can re-open and revisit.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.