#45 Test graphical upgrade of Workstation
Closed: Fixed 16 days ago by adamwill. Opened 7 years ago by adamwill.

I've had this on my personal task list for a while now, but it turns out to be rather subtler than I thought, at least for Rawhide.

We can do it quite straightforwardly for Branched, but we can't really do it yet (except blind) as we don't have a Branched right now.

The problem for Rawhide is that Rawhide doesn't really work like other releases. The fact that our console upgrade test works for Rawhide is actually kind of odd, and relies on a messy hack someone put in place at some point.

So here's the skinny. There are various Fedora repo definitions. The 'stock' ones are in fedora-repos. There are special Rawhide repo definitions in the package fedora-repos-rawhide.

According to Dennis Gilmore, the 'official' way to use Rawhide is to install fedora-repos-rawhide and then enable the repository rawhide and disable the repos fedora, updates and updates-testing. To upgrade to Rawhide, you're supposed to do that, and then run dnf system-upgrade download --releasever=26 --nogpgcheck (at present - you have to know the 'right' release number for Rawhide and use it).

We don't do anything like that. Instead we just leave the repo definitions as they are (we don't install fedora-repos-rawhide at all) and do dnf system-upgrade download --releasever=rawhide --nogpgcheck.

...and this works. Which Dennis was quite surprised about, so I went and looked into why. Here's why!

When you pass --releasever to dnf you're basically telling it 'just go ahead and use whatever repos are configured (or otherwise specified on the command line), but when you see $releasever, don't replace it with whatever you detected the current installed system's release version is. Replace it with this value instead'. So when we install Fedora - say Fedora 25 - and use the default repos from fedora-repos and pass --releasever=rawhide, we're telling dnf to use the repos fedora and updates - since those are the ones that are enabled by default - but to replace $releasever in their values with rawhide, not 25.

OK, so what do the repos actually say? Well, the interesting bits of fedora look like this:

[fedora]
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

the interesting bits of updates look like this:

[updates]
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/$releasever/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

so basically we're going to wind up hitting these URLs, on x86_64:

the first of those is fine, it's what the rawhide repo definition from fedora-repos-rawhide says anyway, if you use that repo. The second one, however, is kinda crazy, and it's what Dennis was expecting to trip us up, because Rawhide has no updates repository. However, at some point, someone has put some kinda hack in place: if you hit that URL, you'll see that it works, and it's actually set up to work exactly like the fedora-rawhide target. This is also the case for the updates-testing-frawhide target. Basically, someone at some point told MirrorManager to treat updates-testing-frawhide and updates-released-frawhide just the same as fedora-rawhide, which makes using dnf --releasever=rawhide with the default repos 'work' (though it's a bit of a hack as you wind up using two or three identical copies of the same repo, depending on whether you have u-t enabled).

Why does all this matter? Well, because now we more or less know how dnf behaves if you just tell it "I want to upgrade to 'rawhide'", but I don't know what GNOME Software will do. I've poked into a bit and so far made it as far as pk_task_upgrade_system_async in PackageKit. Right now GNOME Software won't even offer an upgrade to Rawhide, ever; that's fairly trivial to fix, and I might patch it tomorrow and play around just to see what happens next. But I wanted to jot all this down somewhere so we have the notes available while we look at what G-S / PK actually do if you say 'upgrade me to Rawhide, please'. We should probably test both what happens if you leave the 'default' Fedora repos in place, and what happens if you disable fedora and updates and enable rawhide.

Right now, Dennis still maintains that just using the default repos and dnf --releasever=rawhide is wrong and we shouldn't do that, we should switch the repo definitions before running a Rawhide upgrade. I'm kinda of the opinion that it'd be nice to just bless the --releasever=rawhide approach, coming up with a slightly more elegant hack for what MirrorManager does for the updates and updates-testing repos in that case, because it'd prevent us having to have a variant repo definition for rawhide and tell people to manually change the repo configuration when they want to use Rawhide (and having the fedora and updates repos disabled by default in the Rawhide fedora-repos package, and remembering to keep flipping that back to enabled when we branch). But I could be convinced otherwise, I want to kick it around with Dennis. If we do decide the separate rawhide repo config is the one and only true way to do it, we should adjust the console tests to do the repository reconfiguration when upgrading to Rawhide.


The dupe status is intentional; this task requires changes to both projects.

One rather obvious point I didn't note here before is: we can potentially suppress the test for Rawhide, but this depends a bit on openQA. In the templates you can specify which distro versions each test should run for. However, I'm not sure if it supports any nice ways to say "anything but Rawhide". If it does, this should be pretty easy. If not, maybe we should patch it.

Metadata Update from @adamwill:
- Issue tagged with: newtest

6 years ago

I have submitted this issue as a project for Red Hat Open Source contest. Let us see, what it brings to us.

Yep. In our meeting I forgot I'd already written up the issues in this ticket :)

The write-up above is still pretty much correct and explains the 'unique' problems we have here. The ideas I've had for dealing with this so far basically boil down to:

  1. Actually manage to convince GNOME Software to successfully handle an upgrade to Rawhide (I still haven't fully investigated how difficult this is).
  2. Figure out a way to make openQA skip scheduling the test only for version 'rawhide'. This would require patching the capability into openQA somehow, and I would strongly prefer if we could convince upstream of its desirability. It'd likely need to be an extension to the handling of the version key in the product dict in job templates; currently you can only give either * to mean "schedule on all versions" or a specific version string to mean "schedule on this version".
  3. Do some kind of hack so that the test gets scheduled on Rawhide but is just treated as a no-op, so it either always passes or always fails. This is the easiest but nastiest choice and I really don't want to do it if at all possible.

https://gitlab.gnome.org/GNOME/gnome-software/issues/124 is a ticket where I requested GNOME Software upstream make upgrading to Rawhide possible, though it seems that so far the answer is 'nope'.

So, would there be an option to create the images for two previous versions and have them upgraded to the current version using the gsettings set org.gnome.software show-upgrade-prerelease true? when we cannot make Gnome to upgrade to Rawhide.

Last time when we discussed this, we had space issues, but maybe we would have some space available nowadays? @adamwill

We still have "the Rawhide problem" here, AFAIK. Nothing has been done to resolve that.

OK, let's resolve it today! 🙂

We can easily convince GNOME Software to upgrade to Rawhide by editing ~/.cache/gnome-software/fedora-pkgdb-collections/fedora.json (this file is created once gnome-software process starts, and then re-downloaded when it refreshes the repos periodically or manually), finding this section:

    {
      "allow_retire": true,
      "branchname": "master",
      "date_created": "2014-05-14 12:36:15",
      "date_updated": "2023-08-08 17:07:23",
      "dist_tag": ".fc40",
      "koji_name": "rawhide",
      "name": "Fedora Linux",
      "status": "Under Development",
      "version": "devel"
    },

and then changing "version": "devel" to "version": "40" (if needed, that number can be inferred from "dist_tag": ".fc40"). You don't even need to restart gnome-software process (but it can't hurt), and F40 is now visible in the Updates tab, yay!

I just tested F39->F40 upgrade using GNOME Software in a VM, worked without an issue. 🎉

I can help you to write a jq or python command to adjust the json, if you want.

Now, can we please finally get the GNOME Software test cases automated? It's a lot of manual work each release. Thanks! 😎 🍹 🥥 🌴

thanks for that, yes, that should be possible I think. I can try and get to it at some point, or lruzicka can...or you can take a shot...:D

note this probably relies on mirrormanager behaviour which you're technically not supposed to rely on, but....mehhh. it's worked for years, and I think we possibly rely on it in other places anyway.

I will take it, @adamwill, and start working on it.

Metadata Update from @lruzicka:
- Issue assigned to lruzicka

5 months ago

Fix available as #312.

I have uploaded the changes to staging to see if it works without issues.

In Staging, I was able to run three of the four tests:

The last one ended up somewhere else, I guess I will need to revisit some needle. Please, do not wipe any stored needles beginning in upgrade_ as I did not have time to copy them over.

Metadata Update from @adamwill:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

16 days ago

🎆 🎉 🥳
Thanks, Lukas!

Login to comment on this ticket.

Metadata