sumantrom / fedora-qa / relval

Forked from fedora-qa/relval 7 years ago
Clone
README.md

relval

relval is a CLI tool, using python-wikitcms, which aids in the creation of the wiki pages that are used to track the results of Fedora release validation events, in creating statistics like the heroes of Fedora testing and test coverage statistics, and also can report test results. If you're interested in relval, you may also be interested in testdays, which is to Test Day pages as relval is to release validation pages.

Put simply, you can run relval compose --release 21 --milestone Beta --compose TC1 --username (yourfasname) and all the wiki pages that are needed for the Fedora 21 Beta TC1 release validation test event will be created (if they weren't already). The nightly sub-command does the same for nightly compose testing events, the user-stats and testcase-stats sub-commands handle statistics generation, and report-results can report test results.

Installation and use

relval is packaged in the official Fedora and EPEL 7+ repositories: to install on Fedora run dnf install relval, on RHEL / CentOS with EPEL enabled, run yum install relval. You may need to enable the updates-testing repository to get the latest version. To install on other distributions, you can run python setup.py install.

You can browse relval source, and clone with git clone https://www.happyassassin.net/cgit/relval. Tarballs are also available.

You can use the relval CLI from the tarball without installing it, as ./relval.py from the root of the tarball. You will need all its dependencies, which are listed in setup.py.

Bugs, pull requests etc.

You can file bugs as 'tasks' against relval in the Fedora QA Phabricator. You can log in with a Fedora FAS account - enter your email address as fasname@fedoraproject.org and the FAS authentication system will be used. You can submit pull requests via the same Phabricator instance using the arc diff tool - see this page for how to install and use the arc tool. Just branch off master, make your changes, and submit the arc diff from your branch.

Usage

The validation event SOP provides the correct invocation of relval to use when you simply wish to create the pages for a new compose (the most common use case).

User authentication (for commands requiring login)

The following applies for all commands that require login - anything that writes to the wiki, currently compose, nightly, report-results, report-auto, and size-check.

You may specify the FAS / Wiki user name to be used with --username. You may also create a file ~/.fedora/credentials containing one username and password pair per line, separated by a space, e.g.

username1 password1
username2 password2

If you do create this file, you should ensure it cannot be read by any other user account (chmod 0600 ~/.fedora/credentials, for e.g.)

For python-wikitcms users, note that from 1.11 onwards, the login() method of Wiki instances will use the first username and password from the credentials file if it exists and credentials are not passed as arguments.

If you do not specify the option --username (or -u), relval will attempt to guess one, as follows:

  • If the credentials file exists, the first username listed in it will be used.
  • Otherwise, if ~/.fedora.cert exists and the fedora-cert package is installed, the username will be read from that file (it is the certificate file Fedora packagers are required to have in order to use the build system).
  • Otherwise, it will attempt to find out the current user's system login username, and use that.
  • If all else fails, it will ask you to enter the user name.

Once the user name has been determined according to the above rules, if the credentials file exists and contains an entry for that user, the password from it will be used. Otherwise, relval will prompt you to type in the password.

Storing passwords in files comes with various possible security issues, and so the credentials file mechanism is really primarily intended for unattended, scripted operation of relval, with a dedicated account that can easily be revoked and other security precautions in place. It is not recommended for everyday interactive use.

Common options

All sub-commands honor the option --test, to operate on the staging wiki instead of the production wiki, which can be useful for testing. Please use this option if you are experimenting with the result page creation or result reporting sub-commands, especially if you also pass --force.

All options mentioned here have short names (e.g. -r for --release, but the long names are given here for clarity. Usually the short name is the first letter of the long name. The help pages (relval <sub-command> -h) list all options with both their long and short names.

compose

For TC/RC compose event page creation, use the compose sub-command: relval compose. You must pass the parameters --milestone and --compose to identify the compose for which pages will be created.

You may pass --release to specify the release to operate on; otherwise, relval will attempt to discover the 'next' release, and use that. You may pass --testtype to specify a particular 'test type' (e.g. Base or Desktop); if you pass a test type, only the page for that type (and the summary page and category pages) will be written, while if you do not, the pages for all test types will be written. You may pass --no-current to specify that the Test_Results:Current redirect pages should not be updated to point to the newly-created pages (by default, they will). You may pass --force to force the creation of pages that already exist: this applies to the results pages category page contents, and summary page, but not to the Current redirects, which will always be written if page creation succeeds (unless --no-current is passed). You may pass --download-only to specify that only the Download template (which provides the table included in the instructions section of all the results pages) should be written; this is handy if you need to create or update the Download page for an existing event.

nightly

nightly is pretty much the same as compose, but for nightly composes. It has no required options.

You may pass --release to specify the release to operate on; otherwise, relval will attempt to discover the 'next' release, and use that. You may pass --compose to specify the compose date to create pages for; otherwise, relval will use today's date. You may pass --milestone to specify the tree from which to get the compose (Rawhide or Branched); otherwise, relval will try to guess the appropriate tree based on the current stage of the release cycle and the presence or absence of a compose in each tree, or fail if it cannot make a sufficiently good guess. You may pass --if-needed to run some checks and create the pages only if they pass. Currently, the checks will pass only if it has been at least three days since the last test event, and if it has been less than 14 days, the versions of some significant packages are compared between the two and at least one difference must be found. Even if --if-needed is not passed, nightly checks that the specified compose exists and has an x86_64 boot.iso file, and will not create the event if it does not. You may pass --email with an email address; if you do, an announcement mail for the compose will be generated and sent from the specified address to the test-announce mailing list (or back to the sender address, if --test is also passed, for testing purposes). You may pass --download-only to specify that only the Download template (which provides the table included in the instructions section of all the results pages) should be written; this is handy if you need to create or update the Download page for an existing event.

user-stats

For user statistics generation, use relval user-stats. It has no required options.

You may pass --release to specify the release to operate on; otherwise, relval will attempt to discover the 'next' release, and use that. You may optionally specify a milestone to operate against, with --milestone (Alpha|Beta|Final) (it does not accept Branched or Rawhide, but if you do not pass --milestone at all, Branched and Rawhide result pages will be included). You may also pass the --filter option as many times as you like. If passed, only pages whose name matches any of the --filter parameters will be included. For instance, relval user-stats --release 21 --milestone Beta --filter TC3 --filter Desktop will operate against all Fedora 21 Beta pages with "TC3" or "Desktop" in their names. You may pass --bot to include 'bot' results (those from automated test systems) in the statistics; by default they are excluded.

The result will be a simple HTML page source printed directly to the console which you can save or paste into for e.g. a blog post, containing statistics on the users who contributed results to the chosen set of pages.

testcase-stats

For test coverage statistics generation, use relval testcase-stats. The parameters are the same as those for user-stats. The output will be an entire directory of HTML pages in /tmp with a top-level index.html that links to summary pages for each "test type", and detailed pages for each "unique test" that are linked from the summary pages. You can also pass --out to specify an output directory, which will be deleted if it already exists. You can simply place the entire directory on your web server in a sensible location. Note that the top-level directory will have 0700 permissions by default and you may have to change this before the content will be visible on the server.

report-results

report-results lets you...report results. It edits the result pages in the wiki for you. Why yes, a hacky TUI that pretends mediawiki is a structured data store is a deeply ridiculous thing, thank you for asking.

You may pass --release, --milestone, --compose and --testtype if you like. If you don't fully specify a compose version, it will first attempt to detect the 'current' compose and offer to let you report results against that; if you want to report against a different compose, it will prompt you for the details.

Once you've chosen a compose to report against one way or another, it will then ask you which page section to report a result in, and then which test to report a result for, then what type of result to submit, then whether you want to specify associated bug IDs and/or a comment. And then it will submit the result. Once you're done, you can submit another result for the same section, page, or test type (avoiding the need to re-input those choices).

Please do keep an eye on the actual result wiki pages and make sure the tool edited them correctly.

report-auto

report-auto is also for reporting test results, but it is a non-interactive implementation. You must provide arguments sufficient to identify the compose to report the result against, and the result to report.

It is primarily intended for scripted use - the case where you have written a script to automate one of the test cases, and you want to have your script report the result whenever it's run against a new compose - but you can also use it for ad hoc manual result submission.

The username must be passed as --username or otherwise discoverable as described above. You must provide the account password using the credentials file. You may pass --release, --milestone and --compose to specify the compose: if --compose is a date in YYYYMMDD format, this signifies a nightly compose, and --milestone should be Rawhide or Branched; otherwise, --compose must be a TC/RC identifier (TC1, RC3 etc) and --milestone should be a milestone (Alpha, Beta, Final). If you do not pass any or all of the release identification arguments, relval will try to guess what event you meant, in different ways depending on the arguments you do pass; if you pass none of the arguments it will report results for the current event. For best results, pass all the arguments, or none to report against the current event.

You must pass --testtype to specify the test type (Installation, Base etc).

You must also provide sufficient arguments to identify a single test instance. You must pass at least --testcase. If the same test case occurs more than once in the page, you can pass --testname and/or --section to narrow down the instance you want.

The string you provide for each of these will be used as a match - that is, if you pass --testcase disk, and the page contains the test cases QA:Testcase_single_disk and QA:Testcase_multi_disk (for instance), both would be matched, and the submission would fail, as you must identify exactly one test instance. --testcase multi_disk would identify one test instance, and so be OK.

If the page contained QA:Testcase_single_disk|single disk ext4 and QA:Testcase_single_disk|single disk btrfs, you could pass --testcase single_disk --testname btrfs to identify the instance you wanted.

If the page contained QA:Testcase_single_disk twice, without a 'name', but in two different sections, you could pass --section to specify which one you wanted.

Note that for all three, spaces and underscores are treated as identical, so --testcase Testcase_foo will match if the page text actually has [[QA:Testcase foo]].

You must also pass pass, fail or warn as the final parameter, identifying the result status.

You may pass --bugs (a comma-separated list of bug IDs) and/or --comment (a free text comment, with a maximum length of 250 characters).

You may pass --no-bot to indicate that the result is not a 'bot' (automated test system) result (by default we assume it is an automated test result, as this sub-command is expected to be used mainly by such systems).

size-check

size-check checks the size of the image files for a given compose, and reports the results to the wiki.

You may pass --release, --milestone, and --compose to specify the compose to operate on. If you pass none of them, relval will check the 'current' compose. If you pass only some, wikitcms will try and guess what compose you meant, and the command will fail if it cannot.

Credits

The user-stats and testcase-stats sub-commands are re-implementations of work originally done by Kamil Paral and Josef Skladanka, and incorporate sections of the original implementations, which can be found in the fedora-qa git repository. Josef also suggested the report-auto capability.

The code to get the user name from ~/.fedora.cert if possible was adapted from fedora-easy-karma, by Till Maas.

License

relval is released under the GPL, version 3 or later.