fedora-qa/os-autoinst-distri-fedora

Fedora 'distribution' for os-autoinst (the openQA test runner). Contains the openQA tests for Fedora itself.  |  https://openqa.fedoraproject.org

Commit 92d588f Add support for testing updates

18 files Authored and Committed by adamwill a year ago
Add support for testing updates

Summary:
This adds an entirely new workflow for testing distribution
updates. The `ADVISORY` variable is introduced: when set,
`main.pm` will load an early post-install test that sets up
a repository containing the packages from the specified update,
runs `dnf -y update`, and reboots. A new templates file is
added, `templates-updates`, which adds two new flavors called
`updates-server` and `updates-workstation`, each containing
job templates for appropriate post-install tests. Scheduler is
expected to post `ADVISORY=(update ID) HDD_1=(base image)
FLAVOR=updates-(server|workstation)`, where (base image) is one
of the stable release base disk images produced by `createhdds`
and usually used for upgrade testing. This will result in the
appropriate job templates being loaded.

We rejig postinstall test loading and static network config a
bit so that this works for both the 'compose' and 'updates' test
flows: we have to ensure we bring up networking for the tap
tests before we try and install the updates, but still allow
later adjustment of the configuration. We take advantage of the
openQA feature that was added a few months back to run the same
module multiple times, so the `_advisory_update` module can
reboot after installing the updates and the modules that take
care of bootloader, encryption and login get run again. This
looks slightly wacky in the web UI, though - it doesn't show the
later runs of each module.

We also use the recently added feature to specify `+HDD_1` in
the test suites which use a disk image uploaded by an earlier
post-install test, so the test suite value will take priority
over the value POSTed by the scheduler for those tests, and we
will use the uploaded disk image (and not the clean base image
POSTed by the scheduler) for those tests.

My intent here is to enhance the scheduler, adding a consumer
which listens out for critpath updates, and runs this test flow
for each one, then reports the results to ResultsDB where Bodhi
could query and display them. We could also add a list of other
packages to have one or both sets of update tests run on it, I
guess.

Test Plan:
Try a post something like:
HDD_1=disk_f25_server_3_x86_64.img DISTRI=fedora VERSION=25
FLAVOR=updates-server ARCH=x86_64 BUILD=FEDORA-2017-376ae2b92c
ADVISORY=FEDORA-2017-376ae2b92c CURRREL=25 PREVREL=24

Pick an appropriate `ADVISORY` (ideally, one containing some
packages which might actually be involved in the tests), and
matching `FLAVOR` and `HDD_1`. The appropriate tests should run,
a repo with the update packages should be created and enabled
(and dnf update run), and the tests should work properly. Also
test a regular compose run to make sure I didn't break anything.

Reviewers: jskladan, jsedlak

Reviewed By: jsedlak

Subscribers: tflink

Differential Revision: https://phab.qa.fedoraproject.org/D1143

    
1 @@ -76,7 +76,7 @@
2   4. Link your newly created Test suite to medium type in [WebUI -> Job groups](https://localhost:8080/admin/groups).
3   5. Run test (see [openqa_fedora_tools repository](https://bitbucket.org/rajcze/openqa_fedora_tools)).
4   6. Create needles (images) by using interactive mode and needles editor in WebUI.
5 - 7. Add new Job template and Test suite into `templates` file.
6 + 7. Add new Job template and Test suite into `templates` file (and `templates-updates`, if the test is applicable to the update testing workflow)
7   8. Add new Test suite and Test case into [`conf_test_suites.py`](https://pagure.io/fedora-qa/fedora_openqa/blob/master/f/fedora_openqa/conf_test_suites.py) file in fedora_openqa repository.
8   9. Open differential request via phabricator, set os-autoinst-disti-fedora as a project and repository.
9   10. Open differential request for fedora_openqa change if necessary.
 1 @@ -74,6 +74,8 @@
 2   | `POSTINSTALL` | string | not set | nothing | If set, `tests/(value)_postinstall.pm` will be loaded after install, boot, login, and other postinstall tests
 3   | `UEFI` | boolean | `false`/not set | nothing | whether to use UEFI, this variable isn't usually set in test suites but in machine definition |
 4   | `ANACONDA_TEXT` | boolean | `false`/not set | all | when specified, anaconda will run in text mode |
 5 + | `ANACONDA_STATIC` | string (IPv4 address) | not set | `ANACONDA_TEXT` | If set, will set up static networking using the chosen IP address during install |
 6 + | `POST_STATIC` | string (space-separated IPv4 address and hostname) | not set | nothing | If set, will set up static networking using the chosen IP address and hostname during early post-install |
 7   
 8   Run variables
 9   -------------
10 @@ -81,8 +83,9 @@
11   
12   | Variable | Explanation |
13   | -------- | ----------- |
14 - | `ISO` | contains filename of ISO that is used for booting. If ISOURL is not set, file must exist in /var/lib/openqa/share/factory/iso. If ISOURL is set, it will be downloaded to this name instead of its original name |
15 + | `ISO` | contains filename of ISO that is used for booting. If `ISO_URL` is not set, file must exist in /var/lib/openqa/share/factory/iso. If `ISO_URL` is set, it will be downloaded to this name instead of its original name |
16   | `ISO_URL` | contains URL for ISO to boot, openQA will download it |
17 + | `ADVISORY` | A Bodhi update ID. If set, the 'update testing' flow will be used: post-install tests will be run with the packages from the update, starting from the stable release base disk images |
18   | `DISTRI` | contains distribution name (should be same as in WebUI, probably `fedora`) |
19   | `VERSION` | contains version of distribution |
20   | `FLAVOR` | indicates what type of distribution is used. Three Pungi properties, joined with `-`: `variant`, `type`, and `format`. e.g.: `Server-dvd-iso`. Special value `universal` is used to schedule the group of tests that should be run once each per arch per compose, against the 'best' available ISO |
 1 @@ -40,9 +40,20 @@
 2           # add entry to /etc/hosts
 3           assert_script_run "echo '$ip $hostname $short' >> /etc/hosts";
 4       }
 5 +     # use host's name servers (this is usually going to be correct,
 6 +     # tests which don't want this can overwrite resolv.conf)
 7 +     my @dns = get_host_dns();
 8 +     my $dnstext = '';
 9 +     for (my $i=0; $i < @dns; $i++) {
10 +         my $num = $i + 1;
11 +         $dnstext .= "\nDNS" . ${num} . "=" . $dns[$i];
12 +     }
13       # bring up network. DEFROUTE is *vital* here
14 -     assert_script_run "printf 'DEVICE=eth0\nBOOTPROTO=none\nIPADDR=$ip\nGATEWAY=10.0.2.2\nPREFIX=24\nDEFROUTE=yes' > /etc/sysconfig/network-scripts/ifcfg-eth0";
15 -     script_run "systemctl restart NetworkManager.service";
16 +     my $conftext = "DEVICE=eth0\nBOOTPROTO=none\nIPADDR=$ip\nGATEWAY=10.0.2.2\nPREFIX=24\nDEFROUTE=yes\nONBOOT=yes" . $dnstext;
17 +     assert_script_run "printf '${conftext}\n' > /etc/sysconfig/network-scripts/ifcfg-eth0";
18 +     assert_script_run "systemctl restart NetworkManager.service";
19 +     # the above doesn't seem to reliably set up resolv.conf, so...
20 +     clone_host_file "/etc/resolv.conf";
21   }
22   
23   sub get_host_dns {
 1 @@ -284,11 +284,13 @@
 2       }
 3   }
 4   
 5 - sub repo_setup {
 6 + sub _repo_setup_compose {
 7 +     # Appropriate repo setup steps for testing a compose
 8       # disable updates-testing and updates and use the compose location
 9       # as the target for fedora and rawhide rather than mirrorlist, so
10       # tools see only packages from the compose under test
11       my $location = get_var("LOCATION");
12 +     return unless $location;
13       assert_script_run 'dnf config-manager --set-disabled updates-testing updates';
14       # we use script_run here as the rawhide repo file won't always exist
15       # and we don't want to bother testing or predicting its existence;
16 @@ -298,3 +300,45 @@
17       script_run "cat /etc/yum.repos.d/{fedora,fedora-rawhide}.repo", 0;
18   }
19   
20 + sub _repo_setup_updates {
21 +     # Appropriate repo setup steps for testing a Bodhi update
22 +     # Check if we already ran, bail if so
23 +     return unless script_run "test -f /etc/yum.repos.d/advisory.repo";
24 +     # Use baseurl not metalink so we don't wind up timing out getting
25 +     # metadata from a slow source...baseurl will always give us dl
26 +     # in infra
27 +     assert_script_run "sed -i -e 's,^metalink,#metalink,g' -e 's,^#baseurl,baseurl,g' /etc/yum.repos.d/fedora*.repo";
28 +     # Set up an additional repo containing the update packages. We do
29 +     # this rather than simply running a one-time update because it may
30 +     # be the case that a package from the update isn't installed *now*
31 +     # but will be installed by one of the tests; by setting up a repo
32 +     # containing the update and enabling it here, we ensure all later
33 +     # 'dnf install' calls will get the packages from the update.
34 +     assert_script_run "mkdir -p /opt/update_repo";
35 +     assert_script_run "cd /opt/update_repo";
36 +     assert_script_run "dnf -y install bodhi-client git createrepo", 300;
37 +     # download the packages
38 +     my $version = lc(get_var("VERSION"));
39 +     if ($version eq 'rawhide' || $version > 25) {
40 +         # bodhi client 2.x
41 +         assert_script_run "bodhi updates download --updateid " . get_var("ADVISORY"), 300;
42 +     }
43 +     else {
44 +         # bodhi client 0.9
45 +         # latest git python-fedora fixes bug which makes bodhi -D UPDATE_ID fail
46 +         assert_script_run "git clone https://github.com/fedora-infra/python-fedora.git";
47 +         assert_script_run "PYTHONPATH=python-fedora/ bodhi -D " . get_var("ADVISORY"), 300;
48 +     }
49 +     # create the repo metadata
50 +     assert_script_run "createrepo .";
51 +     # write a repo config file
52 +     assert_script_run 'printf "[advisory]\nname=Advisory repo\nbaseurl=file:///opt/update_repo\nenabled=1\nmetadata_expire=3600\ngpgcheck=0" > /etc/yum.repos.d/advisory.repo';
53 +     # run an update now
54 +     script_run "dnf -y update", 300;
55 + }
56 + 
57 + sub repo_setup {
58 +     # Run the appropriate sub-function for the job
59 +     get_var("ADVISORY") ? _repo_setup_updates : _repo_setup_compose;
60 + }
61 + 
main.pm +31 -8
 1 @@ -127,7 +127,7 @@
 2   
 3       ## Networking
 4       if (get_var('ANACONDA_STATIC')) {
 5 -         autotest::loadtest "tests/_network_static.pm";
 6 +         autotest::loadtest "tests/_anaconda_network_static.pm";
 7       }
 8   
 9       ## Installation source
10 @@ -172,13 +172,9 @@
11       autotest::loadtest "tests/_do_install_and_reboot.pm";
12   }
13   
14 - sub load_postinstall_tests() {
15 -     # special case for the memory check test, as it doesn't need to boot
16 -     # the installed system: just load its test and return
17 -     if (get_var("MEMCHECK")) {
18 -         autotest::loadtest "tests/_memcheck.pm";
19 -         return;
20 -     }
21 + sub _load_early_postinstall_tests() {
22 +     # Early post-install test loading. Split out as a separate sub
23 +     # because we do this all twice on update tests.
24   
25       # Unlock encrypted storage volumes, if necessary. The test name here
26       # follows the 'storage post-install' convention, but must be run earlier.
27 @@ -197,7 +193,34 @@
28       unless (get_var("DESKTOP")) {
29           autotest::loadtest "tests/_console_wait_login.pm";
30       }
31 + }
32 + 
33 + sub load_postinstall_tests() {
34 +     # special case for the memory check test, as it doesn't need to boot
35 +     # the installed system: just load its test and return
36 +     if (get_var("MEMCHECK")) {
37 +         autotest::loadtest "tests/_memcheck.pm";
38 +         return;
39 +     }
40 + 
41 +     # load the early tests
42 +     _load_early_postinstall_tests();
43   
44 +     # do standard post-install static network config if the var is set
45 +     # this is here not in early_postinstall_tests as there's no need
46 +     # to do it twice
47 +     if (get_var("POST_STATIC")) {
48 +         autotest::loadtest "tests/_post_network_static.pm";
49 +     }
50 + 
51 +     # if scheduler passed an advisory, update packages from that advisory
52 +     # (intended for the updates testing workflow, so we install the updates
53 +     # to be tested)
54 +     if (get_var("ADVISORY")) {
55 +         autotest::loadtest "tests/_advisory_update.pm";
56 +         # now load the early boot tests again, as _advisory_update reboots
57 +         _load_early_postinstall_tests();
58 +     }
59       # from now on, we have fully installed and booted system with root/specified user logged in
60   
61       # If there is a post-install test to verify storage configuration worked
 1 @@ -1639,6 +1639,7 @@
 2                           { key => "NICTYPE", value => "tap" },
 3                           { key => "WORKER_CLASS", value => "tap" },
 4                           { key => "GRUB_POSTINSTALL", value => "net.ifnames=0 biosdevname=0" },
 5 +                         { key => "POST_STATIC", value => "10.0.2.110 support.domain.local" },
 6                           { key => "TEST_TARGET", value => "NONE" },
 7                         ],
 8                       },
 9 @@ -2198,6 +2199,7 @@
10                           { key => "BOOTFROM", value => "c" },
11                           { key => "HDD_1", value => "disk_%FLAVOR%_%MACHINE%.qcow2" },
12                           { key => "GRUB", value => "net.ifnames=0 biosdevname=0" },
13 +                         { key => "POST_STATIC", value => "10.0.2.100 ipa001.domain.local" },
14                           { key => "NICTYPE", value => "tap" },
15                           { key => "WORKER_CLASS", value => "tap" },
16                         ],
17 @@ -2238,7 +2240,7 @@
18                           { key => "ROOT_PASSWORD", value => "weakpassword" },
19                           { key => "START_AFTER_TEST", value => "server_cockpit_default" },
20                           { key => "BOOTFROM", value => "c" },
21 -                         { key => "HDD_1", value => "disk_%MACHINE%_cockpit.qcow2" },
22 +                         { key => "+HDD_1", value => "disk_%MACHINE%_cockpit.qcow2" },
23                         ],
24                       },
25                       {
26 @@ -2250,8 +2252,9 @@
27                           { key => "START_AFTER_TEST", value => "server_cockpit_default" },
28                           { key => "PARALLEL_WITH", value => "server_role_deploy_domain_controller" },
29                           { key => "BOOTFROM", value => "c" },
30 -                         { key => "HDD_1", value => "disk_%MACHINE%_cockpit.qcow2" },
31 +                         { key => "+HDD_1", value => "disk_%MACHINE%_cockpit.qcow2" },
32                           { key => "GRUB_POSTINSTALL", value => "net.ifnames=0 biosdevname=0" },
33 +                         { key => "POST_STATIC", value => "10.0.2.102 client002.domain.local" },
34                           { key => "NICTYPE", value => "tap" },
35                           { key => "WORKER_CLASS", value => "tap" },
36                         ],
37 @@ -2267,6 +2270,7 @@
38                           { key => "BOOTFROM", value => "c" },
39                           { key => "HDD_1", value => "disk_%FLAVOR%_%MACHINE%.qcow2" },
40                           { key => "GRUB_POSTINSTALL", value => "net.ifnames=0 biosdevname=0" },
41 +                         { key => "POST_STATIC", value => "10.0.2.103 client003.domain.local" },
42                           { key => "NICTYPE", value => "tap" },
43                           { key => "WORKER_CLASS", value => "tap" },
44                         ],
45 @@ -2281,6 +2285,7 @@
46                           { key => "BOOTFROM", value => "c" },
47                           { key => "HDD_1", value => "disk_%FLAVOR%_%MACHINE%.qcow2" },
48                           { key => "GRUB_POSTINSTALL", value => "net.ifnames=0 biosdevname=0" },
49 +                         { key => "POST_STATIC", value => "10.0.2.104 db.domain.local" },
50                           { key => "NICTYPE", value => "tap" },
51                           { key => "WORKER_CLASS", value => "tap" },
52                         ],
53 @@ -2296,6 +2301,7 @@
54                           { key => "BOOTFROM", value => "c" },
55                           { key => "HDD_1", value => "disk_%FLAVOR%_%MACHINE%.qcow2" },
56                           { key => "GRUB_POSTINSTALL", value => "net.ifnames=0 biosdevname=0" },
57 +                         { key => "POST_STATIC", value => "10.0.2.105 dbclient.domain.local" },
58                           { key => "NICTYPE", value => "tap" },
59                           { key => "WORKER_CLASS", value => "tap" },
60                         ],
  1 @@ -0,0 +1,349 @@
  2 + #!/usr/share/openqa/script/load_templates
  3 + #
  4 + # Fedora Machines, Products, TestSuites and JobTemplates
  5 + #
  6 + # use load_templates to load the file into the database
  7 + #
  8 + {
  9 +   JobTemplates => [
 10 +                     {
 11 +                       machine    => { name => "64bit" },
 12 +                       prio       => 40,
 13 +                       product    => {
 14 +                                       arch    => "x86_64",
 15 +                                       distri  => "fedora",
 16 +                                       flavor  => "updates-workstation",
 17 +                                       version => "*",
 18 +                                     },
 19 +                       test_suite => { name => "base_selinux" },
 20 +                     },
 21 +                     {
 22 +                       machine    => { name => "64bit" },
 23 +                       prio       => 40,
 24 +                       product    => {
 25 +                                       arch    => "x86_64",
 26 +                                       distri  => "fedora",
 27 +                                       flavor  => "updates-server",
 28 +                                       version => "*",
 29 +                                     },
 30 +                       test_suite => { name => "base_selinux" },
 31 +                     },
 32 +                     {
 33 +                       machine    => { name => "64bit" },
 34 +                       prio       => 42,
 35 +                       product    => {
 36 +                                       arch    => "x86_64",
 37 +                                       distri  => "fedora",
 38 +                                       flavor  => "updates-kde",
 39 +                                       version => "*",
 40 +                                     },
 41 +                       test_suite => { name => "base_selinux" },
 42 +                     },
 43 +                     {
 44 +                       machine    => { name => "64bit" },
 45 +                       prio       => 40,
 46 +                       product    => {
 47 +                                       arch    => "x86_64",
 48 +                                       distri  => "fedora",
 49 +                                       flavor  => "updates-workstation",
 50 +                                       version => "*",
 51 +                                     },
 52 +                       test_suite => { name => "base_services_start" },
 53 +                     },
 54 +                     {
 55 +                       machine    => { name => "64bit" },
 56 +                       prio       => 40,
 57 +                       product    => {
 58 +                                       arch    => "x86_64",
 59 +                                       distri  => "fedora",
 60 +                                       flavor  => "updates-server",
 61 +                                       version => "*",
 62 +                                     },
 63 +                       test_suite => { name => "base_services_start" },
 64 +                     },
 65 +                     {
 66 +                       machine    => { name => "64bit" },
 67 +                       prio       => 42,
 68 +                       product    => {
 69 +                                       arch    => "x86_64",
 70 +                                       distri  => "fedora",
 71 +                                       flavor  => "updates-kde",
 72 +                                       version => "*",
 73 +                                     },
 74 +                       test_suite => { name => "base_services_start" },
 75 +                     },
 76 +                     {
 77 +                       machine    => { name => "64bit" },
 78 +                       prio       => 40,
 79 +                       product    => {
 80 +                                       arch    => "x86_64",
 81 +                                       distri  => "fedora",
 82 +                                       flavor  => "updates-workstation",
 83 +                                       version => "*",
 84 +                                     },
 85 +                       test_suite => { name => "base_service_manipulation" },
 86 +                     },
 87 +                     {
 88 +                       machine    => { name => "64bit" },
 89 +                       prio       => 40,
 90 +                       product    => {
 91 +                                       arch    => "x86_64",
 92 +                                       distri  => "fedora",
 93 +                                       flavor  => "updates-server",
 94 +                                       version => "*",
 95 +                                     },
 96 +                       test_suite => { name => "base_service_manipulation" },
 97 +                     },
 98 +                     {
 99 +                       machine    => { name => "64bit" },
100 +                       prio       => 42,
101 +                       product    => {
102 +                                       arch    => "x86_64",
103 +                                       distri  => "fedora",
104 +                                       flavor  => "updates-kde",
105 +                                       version => "*",
106 +                                     },
107 +                       test_suite => { name => "base_service_manipulation" },
108 +                     },
109 +                     {
110 +                       machine    => { name => "64bit" },
111 +                       prio       => 40,
112 +                       product    => {
113 +                                       arch    => "x86_64",
114 +                                       distri  => "fedora",
115 +                                       flavor  => "updates-workstation",
116 +                                       version => "*",
117 +                                     },
118 +                       test_suite => { name => "base_update_cli" },
119 +                     },
120 +                     {
121 +                       machine    => { name => "64bit" },
122 +                       prio       => 40,
123 +                       product    => {
124 +                                       arch    => "x86_64",
125 +                                       distri  => "fedora",
126 +                                       flavor  => "updates-server",
127 +                                       version => "*",
128 +                                     },
129 +                       test_suite => { name => "base_update_cli" },
130 +                     },
131 +                     {
132 +                       machine    => { name => "64bit" },
133 +                       prio       => 42,
134 +                       product    => {
135 +                                       arch    => "x86_64",
136 +                                       distri  => "fedora",
137 +                                       flavor  => "updates-kde",
138 +                                       version => "*",
139 +                                     },
140 +                       test_suite => { name => "base_update_cli" },
141 +                     },
142 +                     {
143 +                       machine    => { name => "64bit" },
144 +                       prio       => 30,
145 +                       product    => {
146 +                                       arch    => "x86_64",
147 +                                       distri  => "fedora",
148 +                                       flavor  => "updates-workstation",
149 +                                       version => "*",
150 +                                     },
151 +                       test_suite => { name => "desktop_update_graphical" },
152 +                     },
153 +                     {
154 +                       machine    => { name => "64bit" },
155 +                       prio       => 32,
156 +                       product    => {
157 +                                       arch    => "x86_64",
158 +                                       distri  => "fedora",
159 +                                       flavor  => "updates-kde",
160 +                                       version => "*",
161 +                                     },
162 +                       test_suite => { name => "desktop_update_graphical" },
163 +                     },
164 +                     {
165 +                       machine    => { name => "64bit" },
166 +                       prio       => 30,
167 +                       product    => {
168 +                                       arch    => "x86_64",
169 +                                       distri  => "fedora",
170 +                                       flavor  => "updates-workstation",
171 +                                       version => "*",
172 +                                     },
173 +                       test_suite => { name => "desktop_terminal" },
174 +                     },
175 +                     {
176 +                       machine    => { name => "64bit" },
177 +                       prio       => 32,
178 +                       product    => {
179 +                                       arch    => "x86_64",
180 +                                       distri  => "fedora",
181 +                                       flavor  => "updates-kde",
182 +                                       version => "*",
183 +                                     },
184 +                       test_suite => { name => "desktop_terminal" },
185 +                     },
186 +                     {
187 +                       machine    => { name => "64bit" },
188 +                       prio       => 30,
189 +                       product    => {
190 +                                       arch    => "x86_64",
191 +                                       distri  => "fedora",
192 +                                       flavor  => "updates-workstation",
193 +                                       version => "*",
194 +                                     },
195 +                       test_suite => { name => "desktop_browser" },
196 +                     },
197 +                     {
198 +                       machine    => { name => "64bit" },
199 +                       prio       => 32,
200 +                       product    => {
201 +                                       arch    => "x86_64",
202 +                                       distri  => "fedora",
203 +                                       flavor  => "updates-kde",
204 +                                       version => "*",
205 +                                     },
206 +                       test_suite => { name => "desktop_browser" },
207 +                     },
208 +                     {
209 +                       machine    => { name => "64bit" },
210 +                       prio       => 30,
211 +                       product    => {
212 +                                       arch    => "x86_64",
213 +                                       distri  => "fedora",
214 +                                       flavor  => "updates-workstation",
215 +                                       version => "*",
216 +                                     },
217 +                       test_suite => { name => "desktop_notifications_postinstall" },
218 +                     },
219 +                     {
220 +                       machine    => { name => "64bit" },
221 +                       prio       => 32,
222 +                       product    => {
223 +                                       arch    => "x86_64",
224 +                                       distri  => "fedora",
225 +                                       flavor  => "updates-kde",
226 +                                       version => "*",
227 +                                     },
228 +                       test_suite => { name => "desktop_notifications_postinstall" },
229 +                     },
230 +                     {
231 +                       machine    => { name => "64bit" },
232 +                       prio       => 40,
233 +                       product    => {
234 +                                       arch    => "x86_64",
235 +                                       distri  => "fedora",
236 +                                       flavor  => "updates-server",
237 +                                       version => "*",
238 +                                     },
239 +                       test_suite => { name => "server_role_deploy_domain_controller" },
240 +                     },
241 +                     {
242 +                       machine    => { name => "64bit" },
243 +                       prio       => 40,
244 +                       product    => {
245 +                                       arch    => "x86_64",
246 +                                       distri  => "fedora",
247 +                                       flavor  => "updates-server",
248 +                                       version => "*",
249 +                                     },
250 +                       test_suite => { name => "server_cockpit_default" },
251 +                     },
252 +                     {
253 +                       machine    => { name => "64bit" },
254 +                       prio       => 40,
255 +                       product    => {
256 +                                       arch    => "x86_64",
257 +                                       distri  => "fedora",
258 +                                       flavor  => "updates-server",
259 +                                       version => "*",
260 +                                     },
261 +                       test_suite => { name => "server_cockpit_basic" },
262 +                     },
263 +                     {
264 +                       machine    => { name => "64bit" },
265 +                       prio       => 40,
266 +                       product    => {
267 +                                       arch    => "x86_64",
268 +                                       distri  => "fedora",
269 +                                       flavor  => "updates-server",
270 +                                       version => "*",
271 +                                     },
272 +                       test_suite => { name => "realmd_join_cockpit" },
273 +                     },
274 +                     {
275 +                       machine    => { name => "64bit" },
276 +                       prio       => 30,
277 +                       product    => {
278 +                                       arch    => "x86_64",
279 +                                       distri  => "fedora",
280 +                                       flavor  => "updates-server",
281 +                                       version => "*",
282 +                                     },
283 +                       test_suite => { name => "realmd_join_sssd" },
284 +                     },
285 +                     {
286 +                       machine    => { name => "64bit" },
287 +                       prio       => 40,
288 +                       product    => {
289 +                                       arch    => "x86_64",
290 +                                       distri  => "fedora",
291 +                                       flavor  => "updates-server",
292 +                                       version => "*",
293 +                                     },
294 +                       test_suite => { name => "server_role_deploy_database_server" },
295 +                     },
296 +                     {
297 +                       machine    => { name => "64bit" },
298 +                       prio       => 40,
299 +                       product    => {
300 +                                       arch    => "x86_64",
301 +                                       distri  => "fedora",
302 +                                       flavor  => "updates-server",
303 +                                       version => "*",
304 +                                     },
305 +                       test_suite => { name => "server_database_client" },
306 +                     },
307 +                     {
308 +                       machine    => { name => "64bit" },
309 +                       prio       => 40,
310 +                       product    => {
311 +                                       arch    => "x86_64",
312 +                                       distri  => "fedora",
313 +                                       flavor  => "updates-server",
314 +                                       version => "*",
315 +                                     },
316 +                       test_suite => { name => "server_firewall_default" },
317 +                     },
318 +                   ],
319 +   Products     => [
320 +                     {
321 +                       arch      => "x86_64",
322 +                       distri    => "fedora",
323 +                       flavor    => "updates-workstation",
324 +                       name      => "",
325 +                       settings  => [
326 +                         { key => "DESKTOP", value => "gnome" },
327 +                                    ],
328 +                       version   => "*",
329 +                     },
330 +                     {
331 +                       arch      => "x86_64",
332 +                       distri    => "fedora",
333 +                       flavor    => "updates-kde",
334 +                       name      => "",
335 +                       settings  => [
336 +                         { key => "DESKTOP", value => "kde" }
337 +                                    ],
338 +                       version   => "*",
339 +                     },
340 +                     {
341 +                       arch      => "x86_64",
342 +                       distri    => "fedora",
343 +                       flavor    => "updates-server",
344 +                       name      => "",
345 +                       settings  => [
346 +                                    ],
347 +                       version   => "*",
348 +                     },
349 +                   ],
350 + }
 1 @@ -0,0 +1,26 @@
 2 + use base "installedtest";
 3 + use strict;
 4 + use testapi;
 5 + use utils;
 6 + 
 7 + sub run {
 8 +     my $self = shift;
 9 +     # do the 'repo setup' steps, which set up a repo containing the
10 +     # update packages and run 'dnf update'
11 +     $self->root_console(tty=>3);
12 +     repo_setup;
13 +     # reboot, in case any of the updates need a reboot to apply
14 +     script_run "reboot", 0;
15 + }
16 + 
17 + sub test_flags {
18 +     # without anything - rollback to 'lastgood' snapshot if failed
19 +     # 'fatal' - whole test suite is in danger if this fails
20 +     # 'milestone' - after this test succeeds, update 'lastgood'
21 +     # 'important' - if this fails, set the overall state to 'fail'
22 +     return { fatal => 1 };
23 + }
24 + 
25 + 1;
26 + 
27 + # vim: set sw=4 et:
 1 @@ -0,0 +1,41 @@
 2 + use base "anacondatest";
 3 + use strict;
 4 + use testapi;
 5 + use utils;
 6 + use tapnet;
 7 + 
 8 + sub run {
 9 +     my $self = shift;
10 +     assert_and_click "anaconda_main_hub_network_host_name";
11 +     assert_and_click "anaconda_network_configure";
12 +     assert_and_click "anaconda_network_ipv4";
13 +     assert_and_click "anaconda_network_method";
14 +     assert_and_click "anaconda_network_method_manual";
15 +     assert_and_click "anaconda_network_address_add";
16 +     type_safely get_var('ANACONDA_STATIC');
17 +     # netmask is automatically set
18 +     type_safely "\t\t";
19 +     # assume gateway
20 +     type_safely "10.0.2.2";
21 +     # move to DNS servers
22 +     type_safely "\n\t\t\t";
23 +     # set DNS from host
24 +     type_safely join(',', get_host_dns());
25 +     type_safely "\t\t\t\t\t\n";
26 +     # can take a bit of time as it seems to wait for all the pending
27 +     # DHCP requests to time out before applying the static config
28 +     assert_screen "anaconda_network_connected", 90;
29 +     assert_and_click "anaconda_spoke_done";
30 + }
31 + 
32 + sub test_flags {
33 +     # without anything - rollback to 'lastgood' snapshot if failed
34 +     # 'fatal' - whole test suite is in danger if this fails
35 +     # 'milestone' - after this test succeeds, update 'lastgood'
36 +     # 'important' - if this fails, set the overall state to 'fail'
37 +     return { fatal => 1 };
38 + }
39 + 
40 + 1;
41 + 
42 + # vim: set sw=4 et:
 1 @@ -38,34 +38,42 @@
 2           send_key "ret";
 3   
 4           # Handle initial-setup, for GNOME, unless START_AFTER_TEST
 5 -         # is set in which case it will have been done already
 6 -         if (get_var("DESKTOP") eq 'gnome' && !get_var("START_AFTER_TEST")) {
 7 -             assert_screen "next_button", 120;
 8 -             # wait a bit in case of animation
 9 -             wait_still_screen 3;
10 -             for my $n (1..3) {
11 -                 # click 'Next' three times, moving the mouse to avoid
12 -                 # highlight problems, sleeping to give it time to get
13 -                 # to the next screen between clicks
14 -                 mouse_set(100, 100);
15 -                 wait_screen_change { assert_and_click "next_button"; };
16 -                 # for Japanese, we need to workaround a bug on the keyboard
17 -                 # selection screen
18 -                 if ($n == 1 && get_var("LANGUAGE") eq 'japanese') {
19 -                     if (!check_screen 'initial_setup_kana_kanji_selected', 5) {
20 -                         record_soft_failure 'kana kanji not selected: bgo#776189';
21 -                         assert_and_click 'initial_setup_kana_kanji';
22 +         # is set in which case it will have been done already. Always
23 +         # do it if ADVISORY is set, as for the update testing flow,
24 +         # START_AFTER_TEST is set but a no-op and this hasn't happened
25 +         if (get_var("DESKTOP") eq 'gnome' && (get_var("ADVISORY") || !get_var("START_AFTER_TEST"))) {
26 +             # as this test gets loaded twice on the ADVISORY flow,
27 +             # keep track of whether this happened already
28 +             unless (get_var("_gis_done")) {
29 +                 assert_screen "next_button", 120;
30 +                 # wait a bit in case of animation
31 +                 wait_still_screen 3;
32 +                 for my $n (1..3) {
33 +                     # click 'Next' three times, moving the mouse to avoid
34 +                     # highlight problems, sleeping to give it time to get
35 +                     # to the next screen between clicks
36 +                     mouse_set(100, 100);
37 +                     wait_screen_change { assert_and_click "next_button"; };
38 +                     # for Japanese, we need to workaround a bug on the keyboard
39 +                     # selection screen
40 +                     if ($n == 1 && get_var("LANGUAGE") eq 'japanese') {
41 +                         if (!check_screen 'initial_setup_kana_kanji_selected', 5) {
42 +                             record_soft_failure 'kana kanji not selected: bgo#776189';
43 +                             assert_and_click 'initial_setup_kana_kanji';
44 +                         }
45                       }
46                   }
47 +                 # click 'Skip' one time
48 +                 mouse_set(100,100);
49 +                 wait_screen_change { assert_and_click "skip_button"; };
50 +                 send_key "ret";
51 +                 # wait for the stupid 'help' screen to show and kill it
52 +                 assert_screen "getting_started";
53 +                 send_key "alt-f4";
54 +                 wait_still_screen 5;
55 +                 # don't do it again on second load
56 +                 set_var("_gis_done", 1);
57               }
58 -             # click 'Skip' one time
59 -             mouse_set(100,100);
60 -             wait_screen_change { assert_and_click "skip_button"; };
61 -             send_key "ret";
62 -             # wait for the stupid 'help' screen to show and kill it
63 -             assert_screen "getting_started";
64 -             send_key "alt-f4";
65 -             wait_still_screen 5;
66           }
67   
68           # Move the mouse somewhere it won't highlight the match areas
 1 @@ -1,41 +0,0 @@
 2 - use base "anacondatest";
 3 - use strict;
 4 - use testapi;
 5 - use utils;
 6 - use tapnet;
 7 - 
 8 - sub run {
 9 -     my $self = shift;
10 -     assert_and_click "anaconda_main_hub_network_host_name";
11 -     assert_and_click "anaconda_network_configure";
12 -     assert_and_click "anaconda_network_ipv4";
13 -     assert_and_click "anaconda_network_method";
14 -     assert_and_click "anaconda_network_method_manual";
15 -     assert_and_click "anaconda_network_address_add";
16 -     type_safely get_var('ANACONDA_STATIC');
17 -     # netmask is automatically set
18 -     type_safely "\t\t";
19 -     # assume gateway
20 -     type_safely "10.0.2.2";
21 -     # move to DNS servers
22 -     type_safely "\n\t\t\t";
23 -     # set DNS from host
24 -     type_safely join(',', get_host_dns());
25 -     type_safely "\t\t\t\t\t\n";
26 -     # can take a bit of time as it seems to wait for all the pending
27 -     # DHCP requests to time out before applying the static config
28 -     assert_screen "anaconda_network_connected", 90;
29 -     assert_and_click "anaconda_spoke_done";
30 - }
31 - 
32 - sub test_flags {
33 -     # without anything - rollback to 'lastgood' snapshot if failed
34 -     # 'fatal' - whole test suite is in danger if this fails
35 -     # 'milestone' - after this test succeeds, update 'lastgood'
36 -     # 'important' - if this fails, set the overall state to 'fail'
37 -     return { fatal => 1 };
38 - }
39 - 
40 - 1;
41 - 
42 - # vim: set sw=4 et:
 1 @@ -0,0 +1,25 @@
 2 + use base "installedtest";
 3 + use strict;
 4 + use testapi;
 5 + use tapnet;
 6 + use utils;
 7 + 
 8 + sub run {
 9 +     my $self = shift;
10 +     my ($ip, $hostname) = split(/ /, get_var("POST_STATIC"));
11 +     $hostname //= 'localhost.localdomain';
12 +     # set up networking
13 +     setup_tap_static($ip, $hostname);
14 + }
15 + 
16 + sub test_flags {
17 +     # without anything - rollback to 'lastgood' snapshot if failed
18 +     # 'fatal' - whole test suite is in danger if this fails
19 +     # 'milestone' - after this test succeeds, update 'lastgood'
20 +     # 'important' - if this fails, set the overall state to 'fail'
21 +     return { fatal => 1 };
22 + }
23 + 
24 + 1;
25 + 
26 + # vim: set sw=4 et:
 1 @@ -7,14 +7,6 @@
 2   
 3   sub run {
 4       my $self=shift;
 5 -     # clone host's /etc/hosts (for phx2 internal routing to work)
 6 -     # must come *before* setup_tap_static or else it would overwrite
 7 -     # its changes
 8 -     clone_host_file("/etc/hosts");
 9 -     # set up networking
10 -     setup_tap_static("10.0.2.110", "support.domain.local");
11 -     clone_host_file("/etc/resolv.conf");
12 - 
13       ## DNS / DHCP (dnsmasq)
14       # create config
15       assert_script_run "printf 'domain=domain.local\ndhcp-range=10.0.2.112,10.0.2.199\ndhcp-option=option:router,10.0.2.2' > /etc/dnsmasq.conf";
 1 @@ -7,14 +7,6 @@
 2   
 3   sub run {
 4       my $self=shift;
 5 -     # clone host's /etc/hosts (for phx2 internal routing to work)
 6 -     # must come *before* setup_tap_static or else it would overwrite
 7 -     # its changes
 8 -     clone_host_file("/etc/hosts");
 9 -     # set up networking
10 -     setup_tap_static("10.0.2.105", "dbclient.domain.local");
11 -     # clone host's resolv.conf to get name resolution
12 -     clone_host_file("/etc/resolv.conf");
13       # use compose repo, disable u-t, etc.
14       repo_setup();
15       # install postgresql
 1 @@ -7,18 +7,14 @@
 2   
 3   sub run {
 4       my $self = shift;
 5 -     # clone host's /etc/hosts (for phx2 internal routing to work)
 6 -     # must come *before* setup_tap_static or else it would overwrite
 7 -     # its changes
 8 -     clone_host_file("/etc/hosts");
 9 -     # set up networking
10 -     setup_tap_static("10.0.2.102", "client002.domain.local");
11       # use FreeIPA server as DNS server
12       assert_script_run "printf 'search domain.local\nnameserver 10.0.2.100' > /etc/resolv.conf";
13       # wait for the server to be ready (do it now just to make sure name
14       # resolution is working before we proceed)
15       mutex_lock "freeipa_ready";
16       mutex_unlock "freeipa_ready";
17 +     # do repo setup
18 +     repo_setup();
19       # run firefox and login to cockpit
20       # note: we can't use wait_screen_change, wait_still_screen or
21       # check_type_string in cockpit because of that fucking constantly
 1 @@ -7,12 +7,6 @@
 2   
 3   sub run {
 4       my $self=shift;
 5 -     # clone host's /etc/hosts (for phx2 internal routing to work)
 6 -     # must come *before* setup_tap_static or else it would overwrite
 7 -     # its changes
 8 -     clone_host_file("/etc/hosts");
 9 -     # set up networking
10 -     setup_tap_static("10.0.2.103", "client003.domain.local");
11       # use FreeIPA server as DNS server
12       assert_script_run "printf 'search domain.local\nnameserver 10.0.2.100' > /etc/resolv.conf";
13       # wait for the server to be ready (do it now just to make sure name
 1 @@ -8,11 +8,6 @@
 2   
 3   sub run {
 4       my $self=shift;
 5 -     clone_host_file("/etc/hosts");
 6 -     # set up networking
 7 -     setup_tap_static("10.0.2.104", "db.domain.local");
 8 -     # clone host's resolv.conf to get name resolution
 9 -     clone_host_file("/etc/resolv.conf");
10       # use compose repo, disable u-t, etc.
11       repo_setup();
12       # deploy the database server role
 1 @@ -10,14 +10,6 @@
 2       my $self = shift;
 3       # login
 4       $self->root_console();
 5 -     # clone host's /etc/hosts (for phx2 internal routing to work)
 6 -     # must come *before* setup_tap_static or else it would overwrite
 7 -     # its changes
 8 -     clone_host_file("/etc/hosts");
 9 -     # set up networking
10 -     setup_tap_static("10.0.2.100", "ipa001.domain.local");
11 -     # clone host's resolv.conf to get name resolution
12 -     clone_host_file("/etc/resolv.conf");
13       # use compose repo, disable u-t, etc.
14       repo_setup();
15       # we need a lot of entropy for this, and we don't care how good