We do have resource issues for aarch64 testing. We currently have only one aarch64 worker host each for lab and prod. Each of them is set to run 15 workers (and that's kinda pushing the limits of what they can manage, I think). Adding this might possibly overload them, at least at busy times...

We now have a lot more aarch64 resources in rdu3 (yaaay), so I am planning to do this. I already enabled workstation update tests on aarch64. I can do kde compose and update tests next.

Woohoo!

I've now landed the initial set of compose tests on the KDE disk image, in line with the tests we currently run of the Workstation disk image. Some tests are left out on Workstation disk for no particularly obvious reason, and we don't do "workstation-upgrade" tests on aarch64 like we do on x86_64 (that's where we run an upgrade from the previous release, then run all the desktop tests on the upgraded system). I'll look later at enhancing coverage in those areas, but since it's not KDE specific I'll do it on different tickets.

Next up for this ticket is KDE update tests.

https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/429 enables the regular KDE update tests on aarch64.

So what's missing now is something like the KDE live ISO test, but logically we'd probably want it to be a 'build a KDE disk image and boot it' test. I started working on that before my PTO, on the updates-kde-disk-aarch64 branch. I don't think it entirely worked yet, but I don't remember exactly where I was up to. Will pick it up now. Updating the story points as that's more work.

I'd actually like both if we can have it. While we're not yet at "KDE live ISO is release-blocking" yet, I don't think we're far away from it either.

So, hmm. I got quite far with the 'build a disk image' test, but I realized I hadn't really reckoned with an important factor: file sizes. An uncompressed KDE disk image is 15G.

The way openQA works with compressed disk images for compose tests is that it downloads the compressed disk image then uncompresses it and treats the uncompressed file as its 'asset'. openQA's just not built to store a compressed disk image as an 'asset' and uncompress it on the fly for testing. For update tests this means we'd have to have the build test upload an uncompressed image, and the install-and-boot test use it.

The biggest images we currently generate and store in updates tests are live ISOs, which are like 2-2.5G in size. Suddenly adding some 15G images is gonna throw off all my calculations about required asset storage space. Also, on a practical level, the upload actually times out, I need to send a patch upstream to allow extending the timeout on the upload command, at a minimum.

I'm gonna stop here for today and cogitate a bit about how to move forward. Another ticket that becomes kinda related at this point is https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/447 , where I proposed dropping the separate job groups for aarch64 jobs so they'd be in the same group as the x86_64 jobs. Asset size limits are configured and applied at the job group level, so doing that would mean less complication in trying to figure out how to allocate asset space (we'd basically just have two 'buckets' at that point, the bucket for compose tests and the bucket for update tests, instead of four).

I might even conclude it'd make sense to try and write an upstream PR to support storing assets as compressed images and decompressing them at run time - on the worker host rather than the server - but that would likely be a substantial chunk of work...

So I've now enabled KDE (and Workstation, for that matter) live ISO tests on aarch64 for updates.

I think the last thing I'm going to do for this ticket is enable a few compose tests which are missing compared to x86_64 - just various miscellaneous ones. Then I'll close this and open a new ticket for the 'build/test a disk image' task as it's kinda a separate thing with its own problems.

Thank you so much! :wine_glass:

https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/459 adds most of the remainder. The fingerprint login test fails on aarch64 and I'm not sure why, I've asked hadess. Aside from that I think the last thing I'll look at is install_usb, then I'll close this.

OK, I added some silverblue tests and another install_usb test on aarch64 and I think that's good for now. Gonna go ahead and close this out.