PR#177: Add QA:Testcase_RpmOstree_Rebase for iot testing (#167) - fedora-qa/os-autoinst-distri-fedora

fedora-qa / os-autoinst-distri-fedora

#177 Add QA:Testcase_RpmOstree_Rebase for iot testing (#167)

Merged 3 years ago by adamwill. Opened 3 years ago by pwhalen.

Unknown source iot-rpmostree-rebase into master

Add QA:Testcase_RpmOstree_Rebase for iot testing (#167)

Paul Whalen • 3 years ago

fd440bf

templates.fif.json

file modified

+14

		`@@ -1518,6 +1518,20 @@`
		`"START_AFTER_TEST": "install_default_upload"`
		`}`
		`},`
		`+ "iot_rpmostree_rebase": {`
		`+ "profiles": {`
		`+ "fedora-IoT-dvd_ostree-iso-aarch64-*-aarch64": 20,`
		`+ "fedora-IoT-dvd_ostree-iso-x86_64-*-64bit": 40`
		`+ },`
		`+ "settings": {`
		`+ "BOOTFROM": "c",`
		`+ "HDD_1": "disk_%FLAVOR%_%MACHINE%.qcow2",`
		`+ "POSTINSTALL": "iot_rpmostree_rebase",`
		`+ "ROOT_PASSWORD": "weakpassword",`
		`+ "START_AFTER_TEST": "install_default_upload",`
		`+ "USER_LOGIN": "false"`
		`+ }`
		`+ },`
		`"mediakit_fileconflicts": {`
		`"profiles": {`
		`"fedora-Server-dvd-iso-aarch64-*-aarch64": 10,`

tests/iot_rpmostree_rebase.pm

file added

+61

		`@@ -0,0 +1,61 @@`
		`+ use base "installedtest";`
		`+ use strict;`
		`+ use testapi;`
		`+ use utils;`
		`+`
		`+ sub run {`
		`+`
		`+ my $self = shift;`
		`+ $self->root_console(tty=>3);`
		`+`
		`+ # list available branches`
		`+ assert_script_run "ostree remote refs fedora-iot";`
		`+`
		`+ # check arch`
		`+ my $arch = lc(get_var("ARCH"));`
		`+`
		`+ # set default for rawhide or devel`
		`+ my $rebase = "stable";`
		`+`
		`+ # if testing the current release, rebase to devel`
		`+ unless (script_run "rpm-ostree status -b \| grep stable") {`
		`+ $rebase = "devel"`
		`+ }`
		`+ # rebase to the appropriate release, arch`
		`+ validate_script_output "rpm-ostree rebase fedora/${rebase}/${arch}/iot", sub { m/systemctl reboot/ }, 300;`
		`+ script_run "systemctl reboot", 0;`
		`+`
		`+ boot_to_login_screen;`
		`+ $self->root_console(tty=>3);`
		`+`
		`+ # check booted branch to make sure successful rebase`
		`+ if ($rebase eq "devel") {`
		`+ validate_script_output "rpm-ostree status -b", sub { m/devel/ }, 300;`
		`+ }`
		`+ if ($rebase eq "stable") {`
		`+ validate_script_output "rpm-ostree status -b", sub { m/stable/ }, 300;`
		`+ }`
		`+`
		`+ # rollback and reboot`
		`+ validate_script_output "rpm-ostree rollback", sub { m/systemctl reboot/ }, 300;`
		`+ script_run "systemctl reboot", 0;`
		`+`
		`+ boot_to_login_screen;`
		`+ $self->root_console(tty=>3);`
		`+`
		`+ # check to make sure rollback successful, also account for branched (devel)`
		`+ if ($rebase eq "devel") {`
		`+ validate_script_output "rpm-ostree status -b", sub { m/stable/ }, 300;`
		`+ }`
		`+ if ($rebase eq "stable") {`
		`+ validate_script_output "rpm-ostree status -b", sub { m/rawhide\|devel/ }, 300;`
		`+ }`
		`+ }`
		`+`
		`+ sub test_flags {`
		`+ return { fatal => 1 };`
		`+ }`
		`+`
		`+ 1;`
		`+`
		`+ # vim: set sw=4 et:`

pwhalen commented 3 years ago

This adds QA:Testcase_RpmOstree_Rebase for iot testing (#167). Tested with
Fedora32 and Rawhide.

Signed-off-by: Paul Whalen pwhalen@fedoraproject.org

zuul commented 3 years ago

Build succeeded.

os-autoinst-tox : SUCCESS in 2m 18s
os-autoinst-perl : SUCCESS in 4m 31s

lruzicka commented 3 years ago

Looks great. I wanted to see it on Staging but I am not sure how to get it there from your fork without merging it first. @adamwill will probably know.

adamwill commented 3 years ago

Mostly looks good, but I'm a bit concerned the logic based on using CURRREL to decide whether to rebase to 'stable' or 'rawhide' is going to turn out to be not quite right. Can we not simply use the output of rpm-ostree status -b to find out what branch we're starting out on and thus which one we should rebase to? That feels like it would be more reliable.

I'd also note this test will fail if the thing we rebase to isn't working, even if the rebase mechanism itself is working fine. It would be good if we can make the rebase targets things that are likely to be stable, but I don't know the process well enough to know what choices we have here. If we can avoid having to rebase to Rawhide, that might be useful...

pwhalen commented 3 years ago

Mostly looks good, but I'm a bit concerned the logic based on using CURRREL to decide whether to rebase to 'stable' or 'rawhide' is going to turn out to be not quite right.

Do you have an example of when that might happen?

Can we not simply use the output of rpm-ostree status -b to find out what branch we're starting out on and thus which one we should rebase to? That feels like it would be more reliable.

That was my first thought as well, but I wasnt sure how- parsing the output of 'rpm-ostree -b' would be a little messier (imo), but then also saving that variable to use when verifying that the rebase was successful. Is there an example in another test I can look at?

I'd also note this test will fail if the thing we rebase to isn't working, even if the rebase mechanism itself is working fine. It would be good if we can make the rebase targets things that are likely to be stable, but I don't know the process well enough to know what choices we have here. If we can avoid having to rebase to Rawhide, that might be useful...

I can change it to rebase to 'devel' when testing the current release, rather than rawhide. That should be more stable as the last 'devel' would be close to GA of the current release.

1 new commit added

Change rebase to devel when testing current release.

3 years ago

zuul commented 3 years ago

Build succeeded.

os-autoinst-tox : SUCCESS in 2m 24s
os-autoinst-perl : SUCCESS in 5m 38s

adamwill commented 3 years ago

Do you have an example of when that might happen?

Well, CURRREL is ultimately set by fedfind's get_current_release(), which I have lots of fun implementing. Until recently it was based on the collections JSON thingy, which we've had problems with not getting updated when it should be. Now it asks Bodhi, but again with the last release, we had an issue - it actually got updated in Bodhi before the release happened, so fedfind was reporting the current release as 32 several days before 32 was officially released.

What if the IoT cutover for 'stable' happens sometime after the Fedora stable release? Say 33 comes out, but we still treat IoT 32 as 'stable' for a week. If we test an IoT 32 compose in that week, we'll see that 32 is not the 'current release' so we'll try and rebase to stable, which is the same thing we're testing, right?

That was my first thought as well, but I wasnt sure how- parsing the output of 'rpm-ostree -b' would be a little messier (imo), but then also saving that variable to use when verifying that the rebase was successful. Is there an example in another test I can look at?

How about this?

# set default rebase target for rawhide or devel
my $rebase = "stable";

# if we already are "stable", rebase to devel. note script_run returns
# the exit code: we get 0 if 'stable' is in the output, hence `unless`
unless (script_run "rpm-ostree status -b | grep stable") {
    $rebase = "devel";
}

1 new commit added

Update logic for determining rebase target.

3 years ago

pwhalen commented 3 years ago

Oh, that's much better. Thanks, updated.

zuul commented 3 years ago

Build succeeded.

os-autoinst-tox : SUCCESS in 2m 19s
os-autoinst-perl : SUCCESS in 5m 52s

adamwill commented 3 years ago

Thanks! Can you just drop the setting of $currrel (it is not used any more) and squash this down to one commit? Then I'll merge it.

Edited 3 years ago by adamwill

rebased onto fd440bf

3 years ago

pwhalen commented 3 years ago

Updated and squashed.

zuul commented 3 years ago

Build succeeded.

os-autoinst-tox : SUCCESS in 2m 19s
os-autoinst-perl : SUCCESS in 4m 54s

adamwill commented 3 years ago

Thanks!

adamwill commented 3 years ago

So we also need to update fedora_openqa to do wiki reporting for this test. It's easy for me to do it, but would you be interested in doing it as a learning exercise? You can pattern it on the Greenboot changes in this commit.