PR#109: Create updates tests for Cockpit. - fedora-qa/os-autoinst-distri-fedora

		`@@ -0,0 +1,46 @@`
		`+ package utils;`
		`+`
		`+ use strict;`
		`+`
		`+ use base 'Exporter';`
		`+ use Exporter;`
		`+ use lockapi;`
		`+ use testapi;`
		`+`
		`+ our @EXPORT = qw/start_cockpit select_cockpit_update/;`
		`+`
		`+`
		`+ sub start_cockpit {`
		`+ # Starting from a console, get to a browser with Cockpit (running`
		`+ # on localhost) shown. If $login is truth-y, also log in. Assumes`
		`+ # X and Firefox are installed.`
		`+ my $login = shift \|\| 0;`
		`+ # https://bugzilla.redhat.com/show_bug.cgi?id=1439429`
		`+ assert_script_run "sed -i -e 's,enable_xauth=1,enable_xauth=0,g' /usr/bin/startx";`
		`+ disable_firefox_studies;`
		`+ # run firefox directly in X as root. never do this, kids!`
		`+ type_string "startx /usr/bin/firefox -width 1024 -height 768 http://localhost:9090\n";`
		`+ assert_screen "cockpit_login", 30;`
		`+ # this happened on early Modular Server composes...`
		`+ record_soft_failure "Unbranded Cockpit" if (match_has_tag "cockpit_login_unbranded");`
		`+ wait_still_screen 5;`
		`+ if ($login) {`
		`+ type_safely "root";`
		`+ wait_screen_change { send_key "tab"; };`
		`+ type_safely get_var("ROOT_PASSWORD", "weakpassword");`
		`+ send_key "ret";`
		`+ assert_screen "cockpit_main";`
		`+ # wait for any animation or other weirdness`
		`+ # can't use wait_still_screen because of that damn graph`
		`+ sleep 3;`
		`+ }`
		`+ }`
		`+`
		`+ sub select_cockpit_update {`
		`+ # This method navigates to to the updates screen`
		`+ assert_and_click "cockpit_software_updates", '', 120;`
		`+ # wait for the updates to download`
		`+ while (!check_screen('cockpit_updates_check',1)) {`
		`+ sleep 5;`
		`+ }`
		`+ }`

needles/cockpit/cockpit_software_updates.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_software_updates"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 68,`
		`+ "ypos": 538,`
		`+ "width": 51,`
		`+ "height": 18,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_software_updates.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_all_install.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_all_install"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 778,`
		`+ "ypos": 266,`
		`+ "width": 56,`
		`+ "height": 17,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_all_install.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_auto-20190529.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "height": 34,`
		`+ "type": "match",`
		`+ "ypos": 190,`
		`+ "width": 48,`
		`+ "xpos": 468`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_auto"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_auto-20190529.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_auto.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_auto"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 443,`
		`+ "ypos": 204,`
		`+ "width": 29,`
		`+ "height": 17,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_auto.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_auto_on.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_auto_on"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 368,`
		`+ "ypos": 199,`
		`+ "width": 66,`
		`+ "height": 27,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_auto_on.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_available.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_available"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 264,`
		`+ "ypos": 259,`
		`+ "width": 185,`
		`+ "height": 28,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_available.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_check.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_check"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 787,`
		`+ "ypos": 142,`
		`+ "width": 110,`
		`+ "height": 18,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_check.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_console_update.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 304,`
		`+ "ypos": 309,`
		`+ "width": 143,`
		`+ "height": 17,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_console_update"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_console_update.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_dnf_install.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_dnf_install"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 846,`
		`+ "ypos": 274,`
		`+ "width": 43,`
		`+ "height": 17,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_dnf_install.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_reconnect.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "area": [`
		`+ {`
		`+ "xpos": 565,`
		`+ "ypos": 381,`
		`+ "width": 75,`
		`+ "height": 19,`
		`+ "type": "match"`
		`+ }`
		`+ ],`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_reconnect"`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_reconnect.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_restart_confirm.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_restart_confirm"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 576,`
		`+ "ypos": 260,`
		`+ "width": 46,`
		`+ "height": 15,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_restart_confirm.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_restart_ignore.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_restart_ignore"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 519,`
		`+ "ypos": 260,`
		`+ "width": 43,`
		`+ "height": 16,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_restart_ignore.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_security_install.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_security_install"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 619,`
		`+ "ypos": 267,`
		`+ "width": 93,`
		`+ "height": 16,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_security_install.png

file added

empty or binary file added

needles/cockpit/cockpit_updates_updated.json

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ {`
		`+ "properties": [],`
		`+ "tags": [`
		`+ "cockpit_updates_updated"`
		`+ ],`
		`+ "area": [`
		`+ {`
		`+ "xpos": 524,`
		`+ "ypos": 276,`
		`+ "width": 56,`
		`+ "height": 43,`
		`+ "type": "match"`
		`+ }`
		`+ ]`
		`+ }`
		`\ No newline at end of file`

needles/cockpit/cockpit_updates_updated.png

file added

empty or binary file added

templates

file modified

+58

		`@@ -785,6 +785,17 @@`
		`flavor => "Server-dvd-iso",`
		`version => "*",`
		`},`
		`+ test_suite => { name => "server_cockpit_updates" },`
		`+ },`
		`+ {`
		`+ machine => { name => "64bit" },`
		`+ prio => 30,`
		`+ product => {`
		`+ arch => "x86_64",`
		`+ distri => "fedora",`
		`+ flavor => "Server-dvd-iso",`
		`+ version => "*",`
		`+ },`
		`test_suite => { name => "realmd_join_cockpit" },`
		`},`
		`{`
		`@@ -2216,6 +2227,18 @@`
		`flavor => "Server-dvd-iso",`
		`version => "*",`
		`},`
		`+ test_suite => { name => "server_cockpit_updates" },`
		`+ },`
		`+ {`
		`+ group_name => "Fedora PowerPC",`
		`+ machine => { name => "ppc64le" },`
		`+ prio => 30,`
		`+ product => {`
		`+ arch => "ppc64le",`
		`+ distri => "fedora",`
		`+ flavor => "Server-dvd-iso",`
		`+ version => "*",`
		`+ },`
		`test_suite => { name => "server_cockpit_basic" },`
		`},`
		`{`
		`@@ -3032,6 +3055,18 @@`
		`flavor => "Server-dvd-iso",`
		`version => "*",`
		`},`
		`+ test_suite => { name => "server_cockpit_updates" },`
		`+ },`
		`+ {`
		`+ group_name => "Fedora PowerPC",`
		`+ machine => { name => "ppc64" },`
		`+ prio => 30,`
		`+ product => {`
		`+ arch => "ppc64",`
		`+ distri => "fedora",`
		`+ flavor => "Server-dvd-iso",`
		`+ version => "*",`
		`+ },`
		`test_suite => { name => "realmd_join_cockpit" },`
		`},`
		`{`
		`@@ -3872,6 +3907,18 @@`
		`flavor => "Server-dvd-iso",`
		`version => "*",`
		`},`
		`+ test_suite => { name => "server_cockpit_updates },`
		`+ },`
		`+ {`
		`+ group_name => "Fedora AArch64",`
		`+ machine => { name => "aarch64" },`
		`+ prio => 30,`
		`+ product => {`
		`+ arch => "aarch64",`
		`+ distri => "fedora",`
		`+ flavor => "Server-dvd-iso",`
		`+ version => "*",`
		`+ },`
		`test_suite => { name => "realmd_join_cockpit" },`
		`},`
		`{`
		`@@ -5641,6 +5688,17 @@`
		`],`
		`},`
		`{`
		`+ name => "server_cockpit_updates",`
		`+ settings => [`
		`+ { key => "POSTINSTALL", value => "server_cockpit_updates server_cockpit_autoupdate" },`
		`+ { key => "USER_LOGIN", value => "false" },`
		`+ { key => "ROOT_PASSWORD", value => "weakpassword" },`
		`+ { key => "+START_AFTER_TEST", value => "server_cockpit_default" },`
		`+ { key => "BOOTFROM", value => "c" },`
		`+ { key => "+HDD_1", value => "disk_%MACHINE%_cockpit.qcow2" },`
		`+ ],`
		`+ },`
		`+ {`
		`name => "realmd_join_cockpit",`
		`settings => [`
		`{ key => "POSTINSTALL", value => "realmd_join_cockpit freeipa_webui freeipa_password_change freeipa_client" },`

tests/server_cockpit_autoupdate.pm

file added

+31

		`@@ -0,0 +1,31 @@`
		`+ use base "installedtest";`
		`+ use strict;`
		`+ use testapi;`
		`+ use utils;`
		`+ use packagetest;`
		`+ use cockpit;`
		`+`
		`+ sub run {`
		`+ my $self=shift;`
		`+ bypass_1691487;`
		`+`
		`+ # Start Cockpit`
		`+ start_cockpit(1);`
		`+`
		`+ # Navigate to the Update screen`
		`+ select_cockpit_update;`
		`+`
		`+ # Switch on automatic updates`
		`+ assert_and_click 'cockpit_updates_auto', '', 120;`
		`+ assert_and_click 'cockpit_updates_dnf_install', '', 120;`
		`+ assert_screen 'cockpit_updates_auto_on';`
		`+`
		`+ }`
		`+`
		`+ sub test_flags {`
		`+ return { always_rolllback => 1 };`
		`+ }`
		`+`
		`+ 1;`
		`+`
		`+ # vim: set sw=4 et:`

tests/server_cockpit_basic.pm

file modified

		`@@ -2,6 +2,7 @@`
		`use strict;`
		`use testapi;`
		`use utils;`
		`+ use cockpit;`

		`sub run {`
		`my $self=shift;`

tests/server_cockpit_default.pm

file modified

		`@@ -2,6 +2,7 @@`
		`use strict;`
		`use testapi;`
		`use utils;`
		`+ use cockpit;`

		`sub run {`
		`my $self = shift;`

tests/server_cockpit_updates.pm

file added

+97

		`@@ -0,0 +1,97 @@`
		`+ use base "installedtest";`
		`+ use strict;`
		`+ use testapi;`
		`+ use utils;`
		`+ use packagetest;`
		`+ use cockpit;`
		`+`
		`+ sub run {`
		`+ my $self=shift;`
		`+ bypass_1691487;`
		`+ # Test if updates are available for the system -> if they are not`
		`+ # prepare test packages that provide a possibilty to update`
		`+ # something (See lib/packagetest.pm).`
		`+`
		`+ script_run "dnf check-update ; echo \$? > status";`
		`+ my $result = script_output "cat status";`
		`+ my $cockdate = "0";`
		`+ if ($result eq "0") {`
		`+ print "INFO: DNF did not find any packages to update. Will continue using modified packages.";`
		`+ prepare_test_packages;`
		`+ verify_installed_packages;`
		`+ }`
		`+ elsif ($result eq "100") {`
		`+ print "INFO: DNF found packages to update. Will continue with real packages.";`
		`+ # When Cockpit is updated through cockpit, the interface gets disconnected`
		`+ # and the user logged out. Let us check if that is the situation.`
		`+ }`
		`+ else {`
		`+ record_soft_failure "DNF returned an error exit code. I will try to run the test anyway.";`
		`+ }`
		`+`
		`+ # Start Cockpit`
		`+ start_cockpit(1);`
		`+ # Navigate to update screen`
		`+ select_cockpit_update;`
		`+`
		`+`
		`+ # If security updates are available, install them first,`
		`+ # so that we test the most buttons in one go.`
		`+ # After updates, ignore restart because we do not need`
		`+ # restarting the machine to continue. Also, we would have`
		`+ # to deal with much more complicated workflow.`
		`+ #`
		`+ if (check_screen('cockpit_updates_security_install',1)) {`
		`+ assert_and_click 'cockpit_updates_security_install';`
		`+ sleep 60; # wait until things are installed`
		`+ assert_and_click 'cockpit_updates_restart_ignore';`
		`+ wait_still_screen 2;`
		`+ }`
		`+`
		`+`
		`+ # Install the rest of the updates, or any updates`
		`+ # that have not been previously installed.`
		`+ assert_and_click 'cockpit_updates_all_install';`
		`+ my $run = 0;`
		`+ while ($run < 40) {`
		`+ # When Cockpit packages are also included in the updates`
		`+ # the user is forced to reconnect, i.e. to restart the Web Application`
		`+ # and relog for further interaction. We will check if reconnection is`
		`+ # needed and if so, we will restart Firefox and login again.`
		`+ #`
		`+ if (check_screen("cockpit_updates_reconnect", 1)) {`
		`+ send_key "ctrl-q";`
		`+ sleep 5;`
		`+ cockpit_update;`
		`+ $run = 50;`
		`+`
		`+ }`
		`+ elsif (check_screen('cockpit_updates_restart_ignore', 1)) {`
		`+ # ignore restart, because we have not finished yet`
		`+ assert_and_click 'cockpit_updates_restart_ignore';`
		`+ $run = 50;`
		`+ }`
		`+ else {`
		`+ sleep 10;`
		`+ $run = $run + 1;`
		`+ }`
		`+ }`
		`+ # Check that the system is updated`
		`+ assert_screen 'cockpit_updates_updated';`
		`+`
		`+ # Switch off Cockpit`
		`+ send_key "ctrl-q";`
		`+`
		`+ if ($result eq "0") {`
		`+ verify_updated_packages;`
		`+ }`
		`+`
		`+ }`
		`+`
		`+ sub test_flags {`
		`+ return { always_rollback => 1 };`
		`+ }`
		`+`
		`+ 1;`
		`+`
		`+ # vim: set sw=4 et:`

lruzicka commented 4 years ago

Create the test case.
Fix bugs and finish.

1 new commit added

Add to templates.

4 years ago

rebased onto c91f8ef96ae7ca7fbef952888aebf5eed023542e

4 years ago

adamwill commented 4 years ago

We have a whole thing for ensuring a specific update is available, and then checking after the update process that that update was really installed - see the other update tests. Can you please revise this to use that, instead of just soft failing if no updates happen to be available? Thanks.

1 new commit added

Add methods to provide packages to update.

4 years ago

1 new commit added

Experiment with the logic.

4 years ago

1 new commit added

Experiment with the logic.

4 years ago

1 new commit added

Reprogram the update test.

4 years ago

Metadata Update from @lruzicka:
- Request assigned

4 years ago

1 new commit added

Save progress.

4 years ago

1 new commit added

Save progress.

4 years ago

rebased onto 902136f6a4ba6a4ef1cfb98945a5a94f24abca46

4 years ago

rebased onto c79292b

4 years ago

adamwill commented on line 87 of templates 4 years ago

is the context of this comment correct? I think you copied and pasted it from elsewhere.

adamwill commented 4 years ago

The entire "tests/server_cockpit_autoupdate.pm" file appears to be unnecessary. Nothing seems to use it.

lruzicka commented 4 years ago

is the context of this comment correct? I think you copied and pasted it from elsewhere.

You mean this?

# to understand the '+' values in this test and the next,
# see the `jobs_from_update` docstring in fedora_openqa
# schedule.py

I do not know where that has come from. I did not write that comment. I will check again.

3 new commits added

Fix templates to enable autoupdate test and delete comment.
Merge branch 'cockpit_dnf' of ssh://pagure.io/fedora-qa/os-autoinst-distri-fedora into cockpit_dnf
Create updates tests for Cockpit.

4 years ago

lruzicka commented 4 years ago

The entire "tests/server_cockpit_autoupdate.pm" file appears to be unnecessary. Nothing seems to use it.

No, it is not unnecessary. I think that doing updates via cockpit and switching autoupdates should be split between to tests. I have fixed the templates to incorporate that test and I also deleted the useless comment from above. Indeed, it was probably copied from the server_cockpit_default test.

adamwill commented 4 years ago

OK...so in that case, please put the cockpit_update sub into lib/utils.pm and share it between the two tests rather than having it duplicated.

Also...what does the autoupdate test actually test exactly? It seems like it only tests that we can flip some switches on, it doesn't test that they actually do what they're meant to do?

lruzicka commented 4 years ago

It seems like that. But I suppose that if Cockpit reports that autoupdate was switched on, it knows that the service was installed and started, right? But it should not be a problem to check explicitly that it was. I can do it.

The problem is that the issue said: "Test the DNF module in Cockpit", but there is not such a module, since Cockpit uses PackageKit. Moreover, it only uses it for updates or installation of Cockpit extensions. You cannot install normal packages, as it seems. The autoupdate test is the only thing where something called dnf is mentioned - which is the installation of dnf-automatic when switching the autoupdates on.

I have sent emails to both reporters to enquire more, but I haven't heard back from them yet, so I wanted to have at least something.

lruzicka commented 4 years ago

OK...so in that case, please put the cockpit_update sub into lib/utils.pm and share it between the two tests rather than having it duplicated.

I think that the cockpit_update method should not go to lib/utils.pm because it basically does nothing except that it starts cockpit (there is a method for that already) and navigates into the Updates tab, so it will never be used anywhere else (just in these two tests).

I only created it because if Cockpit updates Cockpit, it needs a reconnection, and I did not want to repeat the steps to start cockpit and navigate to the correct tab in that one particular test (it originally was one test). When I split them, I only copied that over.

If you terribly dislike the few duplicated lines, maybe I could pull the method from the server_cockpit_update directly - it should be possible - rather than putting a sort-of stupid and useless method into a library.

What do you think?

adamwill commented 4 years ago

Um, I'm not sure if that'd work, because I'm not sure the tests directories are on the library include path for os-autoinst. The lib directory is.

Another option is to split cockpit-related functions out of lib/utils.pm and into a lib/cockpit.pm, like we have a lib/freeipa.pm, and put this in it too.

adamwill commented 4 years ago

"It seems like that. But I suppose that if Cockpit reports that autoupdate was switched on, it knows that the service was installed and started, right? But it should not be a problem to check explicitly that it was. I can do it."

Mmm...on the whole, yeah, I think it'd be nice to check the outcome of bonking the button is as expected, yes. It's probably not something that'd fail a lot, but let's check it.

lruzicka commented 4 years ago

Time to move this a little bit :)

adamwill commented 4 years ago

Well, my last comments asked you to do some stuff, so...I've been waiting for stuff to be done :)

lruzicka commented 4 years ago

This was not a bump for you, this was an information that I know about that
and I will work on it.

Dne p=C3=A1 30. srp 2019 22:35 u=C5=BEivatel Adam Williamson pagure@pagure= .io
napsal:

adamwill commented on the pull-request: Create updates tests for Cockpit. that you are following:
Well, my last comments asked you to do some stuff, so...I've been waiting for stuff to be done :)

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/109

1 new commit added

Split cockpit functions into separate library.

4 years ago

lruzicka commented 4 years ago

Because this branch has outdated and diverged so much that it cannot be easily merged, I am closing this pull request and will open a new one, from a different branch, in which I will continue to work on Adam's suggestions.

Edited 4 years ago by lruzicka

Pull-Request has been closed by lruzicka

4 years ago