@kevin points out that the storage backing the ppc64le builders isn't great (7200rpm spinning rust), but that doesn't seem like it can the whole story, as Workstation live image build takes about 40 minutes on both ppc64le and x86_64. We don't know if this is because lmc somehow disables sync and oz doesn't, or what.
@ngompa is going to try and do an equivalent Kiwi image description so we can try building it with Kiwi and see if that's faster.
When do you need this? (YYYY/MM/DD) No specific date
When is this no longer needed or useful? (YYYY/MM/DD) No specific date
If we cannot complete your request, what is the impact? Composes will take about 90 minutes longer than necessary
ngompa's PR for kiwi: https://pagure.io/fedora-kiwi-descriptions/pull-request/97
I came up with a tentative suggestion that we could add cache="unsafe" to the options for the disk in the VM oz spins up to run its install to, I can try and write a patch for that. No idea how much (if any) it would help. edit: https://koji.fedoraproject.org/koji/taskinfo?taskID=124902943 is an f40 oz build with that patch applied.
cache="unsafe"
The kiwi build took about ~15m
so, we need to talk to server WG about making this official for rawhide. then we can maybe get compose time down to two hours! @pboy
I've filed https://pagure.io/fedora-server/issue/146 to track this on Server WG side.
Metadata Update from @james: - Issue tagged with: medium-gain, medium-trouble
This should be fixed now that we switched to kiwi, no?
well not entirely, I think - it seems like it's partly also some kind of hardware slowness on ppc64le. Switching to kiwi was good, but I'm not sure it entirely solved the problem, there seems to be some inherent variability (based on which build host the task hits and/or how busy it is with other stuff at the time) that distorted the results a bit (we were comparing happened-to-be-slow imgfac runs against happened-to-be-fast kiwi runs).
@kevin 's recent change to have some of the ppc64 hosts use iSCSI was meant to help with this, but we're not sure it did, yet.
I'll have to look at timings from some recent composes to see where we are with this now.
All of them are now on iscsi... which should hae made them much faster, but doesn't seem to have. ;(
oops. I missed a nested virt setting on some. will fix those.
Well, I just checked the last week of Fedora-Server-Guest-Generic kiwi image create tasks for ppc64le, and they hit a variety of hosts and all took under 10 minutes:
So...maybe this really is fixed?
Looks fixed to me!
OK. Let's say the combo of kiwi and ppc64le iSCSI change has fixed this.
Metadata Update from @adamwill: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.