We have has some lockup issues on our power9 machines for a while, and all the power cycling seems to have messed up some of the ppc64le builders. They have missing files from packages they have installed and are not operating ideally.
So, I am going to reinstall them all.
While I am at it, we had 10builders per bvmhost, but the load was causing issues, so we stopped 1 vm on each (9 per vmhost) and that has made it better. I am thinking now that perhaps 8 per bvmhost would be best. It will result in fewer builders, but they should be faster, so I think it will be a gain.
One of the issues I found when looking at the power systems in the past was that they had 'bad' IO due to spinning SATA drives. A couple of builds on different virt guests would cause the system IO to 'max' out but nothing seemed to be written. [CPU and memory contention did not seem to come up but disk IO seemed to cliff dive.]
I don't know if that is still happening but if so cutting down the builders sounds like a great idea until some other fix is possible. (+1 if it is needed)
I got this mostly all done, but I think I still need to adjust them all for nested virt.
So, images might have problems, I will try and look after dinner or tomorrow morning.
ok. I fixed them all. We should be all set.
Metadata Update from @kevin: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.