Note @kevin looked earlier in the day today and did see this in the logs:

13:41:43         +nirik | [Sat Feb 17 15:19:25 2018] Couldn't emulate instruction 0x7c00491c (op 31 xop 142)
13:41:43         +nirik | [Sat Feb 17 15:19:25 2018] kvmppc_exit_pr_progint: emulation at c00000000004e0b0 failed (7c00491c)
13:41:43         +nirik | [Sat Feb 17 15:19:25 2018] Couldn't emulate instruction 0x7c00491c (op 31 xop 142)
13:41:43         +nirik | [Sat Feb 17 15:19:25 2018] kvmppc_exit_pr_progint: emulation at c00000000004e0b0 failed (7c00491c)

hey @sinnykumari - can you look at this bug on monday and see if there is an issue with running an f28 ppc KVM guest on top of an F27 ppc hypervisor?

Try to do an anaconda install for f28 on top of f27 hypervisor and if that doesn't work then try to do an anaconda install for f28 on top of f28 hypervisor.

Anaconda text based install of F28 server dvd from Rawhide-20180218 nightly on F27 hypervisor happened successfully for me. Also, it boots fine.
qemu version - qemu-2.10.1-2.fc27.ppc64le
kernel - 4.15.3-300.fc27.ppc64le

Will try to run imagefactory locally and see if I can figure out something.

Will try to run imagefactory locally and see if I can figure out something.

thanks!

Filed a bug against qemu for this issue - https://bugzilla.redhat.com/show_bug.cgi?id=1547467
Let's see what is their thought on this.

There was at least a successful Atomic cloudImage build in rawhide from Fedora-Atomic-Rawhide-20180221.n.2 compose https://koji.fedoraproject.org/koji/taskinfo?taskID=25224905 .
qemu still crashes in my local system, so not sure what caused this successful imagefactory run in koji builder.
I will wait for today's nightly compose to finish in rawhide and f28 and will observe the result.

still failing. from last night's run:

• https://koji.fedoraproject.org/koji/taskinfo?taskID=25278703
• https://koji.fedoraproject.org/koji/taskinfo?taskID=25278700
• https://koji.fedoraproject.org/koji/taskinfo?taskID=25278689
• https://koji.fedoraproject.org/koji/taskinfo?taskID=25278680

Now, I am getting successful imagefactory run for F29 Atomic cloud and container image by using required artifacts from 20180224.n.0 rawhide compose . Host machine I am using is latest Fedora 27, ppc64le. I still see problem if host machine used is rawhide.

May I know which Fedora version we are running on ppc64le koji builders where these failed composes are running?

May I know which Fedora version we are running on ppc64le koji builders where these failed composes are running?

Fedora 27 with a 4.15.3-300.fc27.ppc64le kernel

May I know which Fedora version we are running on ppc64le koji builders where these failed composes are running?

Fedora 27 with a 4.15.3-300.fc27.ppc64le kernel

Hmm, even I have same kernel version.

So, it looks like concurrency issue when some jobs are running in parallel on Power box.
For imageafctory run on our builders, we define cpus=2 in oz config defined in ansible script. If we just use single cpu core on power box, imagefactory runs fine. Ran locally imagefactory for cloud(f28), container(f28) and atomic (f29) builds and they all succeeded.

So, until we know what has caused this problem on power for F28 and onward image builds, maybe we can update oz.cfg with cpus=1 for power builders used for building images?

oz defaults to 1 cpu, we had to add more due to a bug in anaconda years ago. we can look at changing it back to 1, do we know why 2 cores is failing and 1 is not? I ask because I expect that users will typically have more than one core, are we just delaying the issues to them?

oz defaults to 1 cpu, we had to add more due to a bug in anaconda years ago. we can look at changing it back to 1, do we know why 2 cores is failing and 1 is not? I ask because I expect that users will typically have more than one core, are we just delaying the issues to them?

No, I don't know why this is failing when multiple cores used. I filed a bug against qemu but we don't have much information yet that why install vm crashes in between the image build process.

as my investigation show the issue is a "nested virt with a multi-CPU guest" problem, so having a workaround by changing the oz.cfg to cpus=1 in the infra would be useful

I've pushed a change for now that will set it to 1 on ppc64{le} builders.

And it helped, there are ppc64le images today. Thanks.

I'm glad we got this worked around for now. @sharkcz are we still working to find the root cause and get that fixed (via the BZ) ?

@dustymabe , yes, that's the plan, AFAICT the root cause is known, now working on getting it fixed.

@sinnykumari , is this still an issue?

This issue should be fixed long time back. We have various successful imagebuild run including cloud and container image in F29 release and various F29 TwoWeek releases

This issue should be fixed long time back. We have various successful imagebuild run including cloud and container image in F29 release and various F29 TwoWeek releases

ppc64 (not le) cloud images have been failing, but we removed them for building.

Just FYI.

Closing this ticket as its not needed any more.

@mohanboddu can we just remove all f28 building that is happening now? the only builds that may still be wanted are container builds maybe. We definitely want to drop cloud builds.