sumantrom / fedora-qa / relval

Forked from fedora-qa/relval 7 years ago
Clone
README.md

relval

relval is a CLI tool, using python-wikitcms, which aids in the creation of the wiki pages that are used to track the results of Fedora release validation events, in creating statistics like the heroes of Fedora testing and test coverage statistics, and also can report test results. If you're interested in relval, you may also be interested in testdays, which is to Test Day pages as relval is to release validation pages.

Put simply, you can run relval compose --release 21 --milestone Beta --compose TC1 --username (yourfasname) and all the wiki pages that are needed for the Fedora 21 Beta TC1 release validation test event will be created (if they weren't already). The nightly sub-command does the same for nightly compose testing events, the user-stats and testcase-stats sub-commands handle statistics generation, and report-results can report test results.

Installation and use

relval is packaged in the official Fedora and EPEL 7+ repositories: to install on Fedora run dnf install relval, on RHEL / CentOS with EPEL enabled, run yum install relval. You may need to enable the updates-testing repository to get the latest version. To install on other distributions, you can run python setup.py install.

You can browse relval source, and clone with git clone https://www.happyassassin.net/cgit/relval. Tarballs are also available.

You can use the relval CLI from the tarball without installing it, as ./relval.py from the root of the tarball. You will need all its dependencies, which are listed in setup.py.

Bugs, pull requests etc.

You can file bugs as 'tasks' against relval in the Fedora QA Phabricator. You can log in with a Fedora FAS account - enter your email address as fasname@fedoraproject.org and the FAS authentication system will be used. You can submit pull requests via the same Phabricator instance using the arc diff tool - see this page for how to install and use the arc tool. Just branch off master, make your changes, and submit the arc diff from your branch.

Usage

The validation event SOP provides the correct invocation of relval to use when you simply wish to create the pages for a new compose (the most common use case).

User authentication (for commands requiring login)

The following applies for all commands that require login - anything that writes to the wiki, currently compose, nightly, report-results, report-auto, and size-check.

You may specify the FAS / Wiki user name to be used with --username. You may also create a file ~/.fedora/credentials containing one username and password pair per line, separated by a space, e.g.

username1 password1
username2 password2

If you do create this file, you should ensure it cannot be read by any other user account (chmod 0600 ~/.fedora/credentials, for e.g.)

For python-wikitcms users, note that from 1.11 onwards, the login() method of Wiki instances will use the first username and password from the credentials file if it exists and credentials are not passed as arguments.

If you do not specify the option --username (or -u), relval will attempt to guess one, as follows:

  • If the credentials file exists, the first username listed in it will be used.
  • Otherwise, it will attempt to find out the current user's system login username, and use that.
  • If all else fails, it will ask you to enter the user name.

Once the user name has been determined according to the above rules, if the credentials file exists and contains an entry for that user, the password from it will be used. Otherwise, relval will prompt you to type in the password.

Storing passwords in files comes with various possible security issues, and so the credentials file mechanism is really primarily intended for unattended, scripted operation of relval, with a dedicated account that can easily be revoked and other security precautions in place. It is not recommended for everyday interactive use.

Common options

All sub-commands honor the option --test, to operate on the staging wiki instead of the production wiki, which can be useful for testing. Please use this option if you are experimenting with the result page creation or result reporting sub-commands, especially if you also pass --force.

All options mentioned here have short names (e.g. -r for --release, but the long names are given here for clarity. Usually the short name is the first letter of the long name. The help pages (relval <sub-command> -h) list all options with both their long and short names.

compose

For validation event page creation, use the compose sub-command: relval compose. You must pass the parameters --milestone and --compose to identify the compose for which pages will be created, or --url as the location of the compose to create an event for. If --url is passed, --release, --milestone and --compose will all be ignored.

You may pass --release to specify the release to operate on; otherwise, relval will attempt to discover the 'next' release, and use that. You may pass --testtype to specify a particular 'test type' (e.g. Base or Desktop); if you pass a test type, only the page for that type (and the summary page and category pages) will be written, while if you do not, the pages for all test types will be written. You may pass --no-current to specify that the Test_Results:Current redirect pages should not be updated to point to the newly-created pages (by default, they will). You may pass --force to force the creation of pages that already exist: this applies to the results pages category page contents, and summary page, but not to the Current redirects, which will always be written if page creation succeeds (unless --no-current is passed). You may pass --download-only to specify that only the Download template (which provides the table included in the instructions section of all the results pages) should be written; this is handy if you need to create or update the Download page for an existing event.

user-stats

For user statistics generation, use relval user-stats. It has no required options.

You may pass --release to specify the release to operate on; otherwise, relval will attempt to discover the 'next' release, and use that. You may optionally specify a milestone to operate against, with --milestone (Alpha|Beta|Final) (it does not accept Branched or Rawhide, but if you do not pass --milestone at all, Branched and Rawhide result pages will be included). You may also pass the --filter option as many times as you like. If passed, only pages whose name matches any of the --filter parameters will be included. For instance, relval user-stats --release 21 --milestone Beta --filter TC3 --filter Desktop will operate against all Fedora 21 Beta pages with "TC3" or "Desktop" in their names. You may pass --bot to include 'bot' results (those from automated test systems) in the statistics; by default they are excluded.

The result will be a simple HTML page source printed directly to the console which you can save or paste into for e.g. a blog post, containing statistics on the users who contributed results to the chosen set of pages.

testcase-stats

For test coverage statistics generation, use relval testcase-stats. The parameters are the same as those for user-stats. The output will be an entire directory of HTML pages in /tmp with a top-level index.html that links to summary pages for each "test type", and detailed pages for each "unique test" that are linked from the summary pages. You can also pass --out to specify an output directory, which will be deleted if it already exists. You can simply place the entire directory on your web server in a sensible location. Note that the top-level directory will have 0700 permissions by default and you may have to change this before the content will be visible on the server.

report-results

report-results lets you...report results. It edits the result pages in the wiki for you. Why yes, a hacky TUI that pretends mediawiki is a structured data store is a deeply ridiculous thing, thank you for asking.

You may pass --release, --milestone, --compose and --testtype if you like. If you don't fully specify a compose version, it will first attempt to detect the 'current' compose and offer to let you report results against that; if you want to report against a different compose, it will prompt you for the details.

Once you've chosen a compose to report against one way or another, it will then ask you which page section to report a result in, and then which test to report a result for, then what type of result to submit, then whether you want to specify associated bug IDs and/or a comment. And then it will submit the result. Once you're done, you can submit another result for the same section, page, or test type (avoiding the need to re-input those choices).

Please do keep an eye on the actual result wiki pages and make sure the tool edited them correctly.

size-check

size-check checks the size of the image files for a given compose, and reports the results to the wiki.

You may pass --release, --milestone, and --compose to specify the compose to operate on. If you pass none of them, relval will check the 'current' compose. If you pass only some, wikitcms will try and guess what compose you meant, and the command will fail if it cannot.

Credits

The user-stats and testcase-stats sub-commands are re-implementations of work originally done by Kamil Paral and Josef Skladanka, and incorporate sections of the original implementations, which can be found in the fedora-qa git repository.

License

relval is released under the GPL, version 3 or later.