Snippet from IRC conversation on #fedora-releng

<mboddu> ksinny: Nothing has changed on the builders side
<nirik> yeah, nothing I know of off hand.
<ksinny> weird :/
<ksinny> nirik: imagebuilders areon Power9 now, right?
<nirik> there is a fair number of stuck vm's... I can clean all those up.
<nirik> yes
<ksinny> ok
<ksinny> let's see if stuck vms clean-up helps with next compose
<ksinny> nirik: imagebuild of F29 AH cloud Image went fine for me on Power 9 vm locally(on F30), looks like builder specific issue. Did you get a chance to clean-up stuck builds?
<nirik> yes, I did that right after I mentioned it yesterday.
<nirik> could be a kernel issue? or host...
<ksinny> nirik: P9 vm I am using has kernel-5.0.7-300.fc30.ppc64le, what kernel version our koji imagebuilders are on?
<mboddu> ksinny: 5.0.6-200.fc29.ppc64le is the kernel version on the builders

Good to see that ppc64le AH cloudImage built fine for F29 updates and updates-testing compose from 20190429.0 . Not sure what fixed it as I don't see any package update in ostree since last compose from 20190427.0 . Any possible changes has been made on or after 20190427 which could possibly have fixed it?

I spent several hours yesterday looking into this and finally tracked it down to firewall rules. The iptables firewall had been restarted, but libvirtd wasn't so it didn't insert it's nat rules in. This meant that guest vm's couldn't get out to the net for the initramfs/kernel to boot.

So, this is hopefully all fixed now.

Wow, that was really a tricky fix. Thanks a lot @kevin for fixing it!

@kevin Thats some fix. Thanks for the help.

Thanks @kevin, I've spend some time with it last week and couldn't reproduce the problem.

Thanks @kevin - We appreciate you looking into this!