#217 Silverblue/OStree testing & gating in openQA
Closed: Fixed a year ago by adamwill. Opened 3 years ago by frantisekz.

We should start doing basic sanity checks and gating of Silverblue/OStree updates in openQA to prevent things like https://pagure.io/fedora-infrastructure/issue/9634 from happenning.

Testcases from https://pagure.io/fedora-qa/issue/659 can be part of the gating matrix, apart from other basic testcases we have in OpenQA.

Relevant discussion: https://github.com/coreos/fedora-coreos-tracker/issues/17

cc @tpopela


So basically what we'd need to do this is a test that builds an ostree with the update used in both the build process and the generated ostree, just as we do with building an installer image (in _installer_build.pm) and a live image (in _live_build.pm). We would also need a way to boot a system with that ostree and run the tests on it, whether that means building an entire ostree installer image as well, or using a base disk image built - somehow? - and updating the ostree deployment on it before running the tests...

We have all that infrastructure already in coreos-assembler and it's executed by the FCOS pipeline.

Long term I'd like to see Silverblue derive from FCOS, which would mean it automatically inherits all of that testing. There's a lot of detail to how that works though.

We would also need a way to boot a system with that ostree and run the tests on it, whether that means building an entire ostree installer

Building a new installer image isn't necessary, any more than it's necessary to build a new installer image to test yum update.

or using a base disk image built - somehow?

For the kola upgrade test that's using the latest FCOS stable image. The current Silverblue equivalent is installing from the "gold" ISO. But that said, if Silverblue was to work like FCOS there'd also be "last stable ISO" refreshed every release too that's also covered by our CI.

There are lots of things that would be nice if they happened, but right now we have to work with the world as it is...

Building a new installer image is not necessary but it is one way to implement this, and it might also be useful to detect if there are any problems with the process of actually building or running an installer containing the ostree. This is why we have a test for building a live image and a test for building a netinst image - to see if the update causes any problems for those things.

The other options are to use a base disk image somehow (the ones we use for non-ostree tests are built by virt-install, but I don't know if it can build ostree-based images) or redeploy from the gold installer on every test, yes.

Metadata Update from @adamwill:
- Issue tagged with: newtest

2 years ago

Note, I did actually start working on this a few months ago, got interrupted by the F37 release process. I'm picking it up again now. I think I got as far as building an ostree successfully, but didn't manage to build the installer image with it embedded yet.

So, I've now merged this.

Those tests do the following:

  • Build a Silverblue ostree with the contents of current stable plus whatever is in the update
  • Build a Silverblue ostree installer image with that ostree embedded
  • Run an install from that image, check the installed system boots to gnome-initial-setup, run through gnome-initial-setup, check we successfully reach a desktop

That covers a decent amount of ground, but I think does not cover the ground from the bug that was the catalyst for this request. So, I'll look at extending the test set. We can have the 'install and boot' test upload its disk image, then run subsequent tests that boot from that disk image and run tests of rebasing and overlaying. We have already written rebasing and overlaying tests, but they are currently only run on IoT images, I'll look at extending this.

Please nobody look too hard at ostree-parse-pungi.py...:D

I've now merged the ostree-generic branch to main and deployed it to production; so we now test Silverblue ostree creation, installer build, install, g-i-s and login to desktop, and ostree overlay and rebase, for all critpath updates.

We could possibly add a few more of the base tests to the test set, but I think that should be enough to close the ticket at least.

Metadata Update from @adamwill:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a year ago

Login to comment on this ticket.

Metadata