From 0edd28842d68d18ac019c64c67d7769bc6bd65be Mon Sep 17 00:00:00 2001 From: Kamil Páral Date: Nov 16 2016 13:42:54 +0000 Subject: add exported wiki pages from fedorahosted project hosting --- diff --git a/wiki/INFO b/wiki/INFO new file mode 100644 index 0000000..11ba145 --- /dev/null +++ b/wiki/INFO @@ -0,0 +1 @@ +These are exported wiki pages from soon-to-be-retired https://fedorahosted.org/autoqa/ project hosting. diff --git a/wiki/OriginalStart.txt b/wiki/OriginalStart.txt new file mode 100644 index 0000000..4061591 --- /dev/null +++ b/wiki/OriginalStart.txt @@ -0,0 +1,27 @@ += AutoQA = + +The ''autoqa'' project was a basic framework for executing a series of tests and reporting results. + +'''This project was superseded by [https://fedoraproject.org/wiki/Taskotron Taskotron], this project is no longer active.''' + += Getting involved = + * '''Source code''': See [https://fedorahosted.org/autoqa/browser] + * Browse: [http://git.fedorahosted.org/git/autoqa.git] + * Checkout: `git clone git://git.fedorahosted.org/autoqa.git autoqa` + * Tarballs: [https://fedorahosted.org/released/autoqa/] + * '''IRC''': #fedora-qa on irc.freenode.net ([http://webchat.freenode.net/?channels=fedora-qa join]) + * '''Mailing list''': + * ''Development discussion'' - https://admin.fedoraproject.org/mailman/listinfo/qa-devel + * ''Automated test results'' - https://fedorahosted.org/mailman/listinfo/autoqa-results + * '''Bugs''': Please file bugs reports and feature requests in [/autoqa/newticket Trac] (you must be logged in) + += Documentation = + * [https://fedoraproject.org/wiki/AutoQA] + * [https://fedoraproject.org/wiki/Category:AutoQA] + * [https://fedoraproject.org/wiki/Autotest] + * [wiki:rpmguard] - documentation of rpmguard test + * [wiki:autoqaconf AutoQA Configuration] - Configurable settings for AutoQA + += Release Planning = + * [wiki:Planning050series 0.5.x Release Series] + * [wiki:Planning060series 0.6.x Release Series] diff --git a/wiki/Planning050logFormatting.txt b/wiki/Planning050logFormatting.txt new file mode 100644 index 0000000..97c7a21 --- /dev/null +++ b/wiki/Planning050logFormatting.txt @@ -0,0 +1,34 @@ +This page describes the log reformatting feature for the [wiki:Planning050series 0.5.x release series]. += Objectives = + * Minimize first-result page scrolling for maintainers (doesn't apply to detailed logs) + * Provide a solution within a specific time frame (yet to be determined) + * Will explore different solutions (plaintext formatting, html output, javascript filtering etc.) + * Based on the exploration, will determine the best option in that time frame. + * Plaintext is a '''MUST''', html is a nice-to have feature; potentially for a future release + * Solution will entail 3 types of log files + * "Pretty" log - the output linked to from bodhi, designed for maintainers + * Will contain highlights and links to the actual log files but no raw log information + * "Full" log - the entire test output + * Contains the test output (similar to what we currently have) + * "Debug" log - with autotest debug messages + * Likely to be used more for triaging issues with the tests + += Specific Test Logs = +'''depcheck''' + * Extract important parts of the log for hightlights ("xxx has depsolving problems") + * Include link to full output log (without autotest debug messages) +'''upgradepath''' + * Extract lines concerning the relevant packages + * Link to full output may not be needed + += Log Formatting Mockups = +'''Richly Formatted Mockups''' + * [http://kparal.fedorapeople.org/autoqa/autoqa%20test%20output.png HTML Log Highlights Mockup] + +'''Plaintext Mockups''' + * [http://tflink.fedorapeople.org/autoqa/log_reformat/depcheck/tflink_depcheck_log_mockup.txt Plaintext depcheck log mockup] + * [http://tflink.fedorapeople.org/autoqa/log_reformat/upgradepath/tflink_upgradepath_log_mockup.txt plaintext upgradepath log mockup] + += Examples = +Within Autotest, there are already tests that produce html output + * [http://tflink.fedorapeople.org/autoqa/log_reformat/examples/autotest_htmloutput_example.png Autotest log showing html output] \ No newline at end of file diff --git a/wiki/Planning050series.txt b/wiki/Planning050series.txt new file mode 100644 index 0000000..ee92ee5 --- /dev/null +++ b/wiki/Planning050series.txt @@ -0,0 +1,29 @@ +The theme for the 0.5.0 release series is to improve the usability of AutoQA + += Objectives = + +=== Improve usability of log output === + * [wiki:Planning050logFormatting Log output feature details] +=== Reduce the number of "useless" emails sent to maintainers === + * [wiki:Planning05xEmailReduction Email reduction feature details] +=== Create documentation for current tests === +This documentation would include descriptions of: + * What the test does + * How to triage common failures + * How to read test output +=== Increase Release Frequency === +For the 0.5.x release series, we will evaluate whether or not to release on a regular basis and do the release when there is enough progress to justify the release. + * Initially, we will evaluate AutoQA for release every 2 weeks + += Progress = +== [milestone:0.5.0 0.5.0] == + * Start Date: + * End Date: TBD +'''To Do''' + [[TicketQuery(milestone=0.5.0&status=new|assigned|reopened,table)]] +'''Completed''' + [[TicketQuery(milestone=0.5.0&status=closed,table)]] + + + + diff --git a/wiki/Planning05xEmailReduction.txt b/wiki/Planning05xEmailReduction.txt new file mode 100644 index 0000000..5d1c1d7 --- /dev/null +++ b/wiki/Planning05xEmailReduction.txt @@ -0,0 +1,31 @@ +This page describes the email reduction feature of the [wiki:Planning050series 0.5.x release series]. += Objectives = + * Reduce the amount of email noise that maintainers receive from AutoQA through bodhi notifications + * Make sure that important and useful notifications are still sent + * Don't hard-code defaults + * Changes to defaults should be configurable from a config file where possible + * Configuration will be global for now, no per-user defaults + += When To Send Emails = +'''Don't Send Emails When:''' + * All tests pass + * Look for input from people who have dealt with this before + * Tests have not yet been completed + * Wait for all tests to complete before sending any notifications + * A test fails again after 3 days (current policy) + * Make this configurable on a global level + +'''Send Emails When:''' + * Update changes state + * PASS -> FAIL + * FAIL -> PASS + * Any test changes state + * This is not set in stone, make sure it's configurable as a global default + + + += Potential Features for Future Releases = + + * Notification policies configurable on a per-user basis + * Wait for 0.5.x to be completed, deal with this later if we get requests for it. + * Possibly some sort of white/black list or interface with FAS \ No newline at end of file diff --git a/wiki/Planning060series.txt b/wiki/Planning060series.txt new file mode 100644 index 0000000..c7cc54c --- /dev/null +++ b/wiki/Planning060series.txt @@ -0,0 +1,21 @@ += Objectives = +0.6.0 will consist mostly of some larger bugfixes and enhancements. We are also planning to start work on creating functional self-tests for AutoQA and the tooling required in order to run said tests. + += Release Strategy = +We are shooting to release 0.6.0 in 3 about three weeks. To that end, we have kept the feature load lighter for this release and will be providing status updates on a weekly basis in order to keep track of our progress towards the release. + += Progress = +== [milestone:0.6.0 0.6.0] == + * Start Date: 2011-07-18 + * End Date: 2011-08-08 (proposed, not set in stone) +=== Required Features === +'''TODO''' + [[TicketQuery(milestone=0.6.0&status=new|assigned|reopened&priority=major,table)]] +'''Completed''' + [[TicketQuery(milestone=0.6.0&status=closed,table&priority=major)]] + +=== 'Nice To Have' Features === +'''TODO''' + [[TicketQuery(milestone=0.6.0&status=new|assigned|reopened&priority=minor,table)]] +'''Completed''' + [[TicketQuery(milestone=0.6.0&status=closed,table&priority=minor)]] \ No newline at end of file diff --git a/wiki/RpmguardChecks.txt b/wiki/RpmguardChecks.txt new file mode 100644 index 0000000..24561a1 --- /dev/null +++ b/wiki/RpmguardChecks.txt @@ -0,0 +1,185 @@ +[[TOC]] + += rpmguard checks = + +This is the list of current rpmguard checks and a short explanation to each. You can recognize each check by a tag that is printed to the output. Here is the list of all the tags sorted alphabetically. + +=== config-file-added === + +Syntax: +{{{ +W: config-file-added FILE +}}} + +There is a new file that is marked as a configuration file. + +=== config-file-changed === + +Syntax: +{{{ +W: config-file-changed FILE +}}} + +The contents of a configuration file was modified. + +=== config-file-removed === + +Syntax: +{{{ +W: config-file-removed FILE +}}} + +A file marked as a configuration file was deleted. + +=== conflict-added === + +Syntax: +{{{ +W: conflict-added LABEL +}}} + +A new Conflicts field was added. + +=== conflict-removed === + +Syntax: +{{{ +W: conflict-removed LABEL +}}} + +An existing Conflicts field was deleted. + +=== conflict-version-lowered === + +Syntax: +{{{ +W: conflict-version-lowered CONFLICT_NAME OLD_VERSION -> NEW_VERSION +}}} + +An existing Conflicts field is a versioned field and the new version requirement is lower than the previous version requirement. + +=== doc-files-count-reduced === + +Syntax: +{{{ +W: doc-files-count-reduced OLD_COUNT -> NEW_COUNT +}}} + +The number of files marked as documentation has been reduced. + +=== executable-added === + +Syntax: +{{{ +W: executable-added FILE +}}} + +There is a new file that is executable and is located in */bin/* folder. + +=== executable-removed === + +Syntax: +{{{ +W: executable-removed FILE +}}} + +An existing file that is executable and is located in */bin/* folder has been removed. + +=== file-mode-changed === + +Syntax: +{{{ +W: file-mode-changed FILE OLD_MODE -> NEW_MODE +}}} + +The mode of an existing file was changed. The mode is displayed in octal (as used in chmod). + +=== obsolescence-added === + +Syntax: +{{{ +W: obsolescence-added LABEL +}}} + +A new Obsoletes field was added. + +=== obsolescence-removed === + +Syntax: +{{{ +W: obsolescence-removed LABEL +}}} + +An existing Obsoletes field was deleted. + +=== obsolescence-version-lowered === + +Syntax: +{{{ +W: obsolescence-version-lowered OBSOLESCENCE_NAME OLD_VERSION -> NEW_VERSION +}}} + +An existing Obsoletes field is a versioned field and the new version requirement is lower than the previous version requirement. + +=== owner-changed === + +Syntax: +{{{ +W: owner-changed FILE OLD_USER,OLD_GROUP -> NEW_USER,NEW_GROUP +}}} + +User or group of an existing file was changed. + +=== provision-added === + +Syntax: +{{{ +W: provision-added LABEL +}}} + +A new Provides field was added. + +=== provision-removed === + +Syntax: +{{{ +W: provision-removed LABEL +}}} + +An existing Provides field was deleted. + +=== provision-version-lowered === + +Syntax: +{{{ +W: provision-version-lowered PROVISION_NAME OLD_VERSION -> NEW_VERSION +}}} + +An existing Provides field is a versioned field and the new version requirement is lower than the previous version requirement. + +=== requirement-added === + +Syntax: +{{{ +W: requirement-added LABEL +}}} + +A new Requires field was added. + +=== requirement-removed === + +Syntax: +{{{ +W: requirement-removed LABEL +}}} + +An existing Requires field was deleted. + +=== requirement-version-lowered === + +Syntax: +{{{ +W: requirement-version-lowered REQUIREMENT_NAME OLD_VERSION -> NEW_VERSION +}}} + +An existing Requires field is a versioned field and the new version requirement is lower than the previous version requirement. diff --git a/wiki/autoqaconf.txt b/wiki/autoqaconf.txt new file mode 100644 index 0000000..064ccee --- /dev/null +++ b/wiki/autoqaconf.txt @@ -0,0 +1,2 @@ +This document now lives in the fedoraproject wiki: +https://fedoraproject.org/wiki/AutoQA_Configuration \ No newline at end of file diff --git a/wiki/initscripts.txt b/wiki/initscripts.txt new file mode 100644 index 0000000..0d4c3bf --- /dev/null +++ b/wiki/initscripts.txt @@ -0,0 +1,113 @@ +---- +'''THIS PAGE IS OBSOLETE, INITSCRIPTS TEST HAS BEEN REMOVED FROM AUTOQA''' +---- + += What are initscript tests = + +The goal of these tests is to check LSB compliance of initscripts - i.e. if the scripts contain [https://fedoraproject.org/wiki/Packaging/SysVInitScript#Required_Actions all required actions] and if the exit codes for [https://fedoraproject.org/wiki/Packaging/SysVInitScript#Exit_Codes_for_the_Status_Action status] and [https://fedoraproject.org/wiki/Packaging/SysVInitScript#Exit_Codes_for_non-Status_Actions non-status] actions are correct ([https://fedoraproject.org/wiki/FCNewInit/Initscripts#Init_Script_Actions another source of information]). + += How can you help? = + +== Help with testing of the tests :) == + +At the moment, we have [https://fedorahosted.org/autoqa/browser/tests/initscripts/tests some of these] tests already in AutoQA , but there are [http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=tests/initscripts/tests;hb=initscripts more] waiting to be revised and pushed to the master branch. + +We have already roughly pre-checked these tests, but they still need some deeper testing - i.e. someone needs to go through the source code, see if all the tests are correct, if the script does not leave the system in 'another' or even 'broken' state, and possibly fix the issues & provide patches so the tests can be pushed to the master branch. + += How to test the tests = + +These tests are written in BASH using [https://fedorahosted.org/beakerlib/ Beakerlib] library, which is used for writing bash-based tests in Red Hat. But don't be scared, this library is pretty self-explanatory and you can review all the tests using your BASH knowledge and this [https://fedorahosted.org/beakerlib/attachment/wiki/Download/BeakerLib%20Quick%20Reference.pdf?format=raw Beakerlib quick reference sheet]. + +So lets look at the workflow of the review. + +== Selecting the script == + +Go through the [https://fedorahosted.org/autoqa/query?status=new&status=reopened&milestone=Initscripts+review Initscript review milestone], and choose the script you would like to review. +Once you have done that, take the ticket and follow the next steps. + +== Setting up the environment == + + 1. ''Prepare a system'' -- You'll need a Fedora system to test with. While, the tests should be non-destructive, they do require installing packages that provide the respective services. Depending on your setup, you may wish to test on another system. To test with a virtual machine, you can find more information at [https://fedoraproject.org/wiki/Getting_started_with_virtualization Getting_started_with_virtualization]. + 1. ''Install initscript'' -- Depending on which test you choose, you must install the package that provides the initscript. For example, if you are testing ''mysql'', you might type: {{{yum install /etc/rc.d/init.d/mysqld}}} + 1. ''Install beakerlib'' -- Once you have your test machine, be it a virtual or bare metal, you need to install the beakerlib package. Type: {{{yum install beakerlib}}} + +== Getting the script == + +The best way to do this is using git: +{{{ +yum install git +git clone git://git.fedorahosted.org/autoqa.git autoqa +cd autoqa +git checkout -b initscripts origin/initscripts +}}} +The scripts are then in the {{{tests/initscripts/tests/}}} directory. + +Or you can just download the content of the respective directory (e.g. [http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=tests/initscripts/tests/httpd;hb=initscripts httpd] from the git repo using your web browser). + +You ''really'' need only the {{{runtest.sh}}} file, which contains the test's source code. + +== Pre-check == + +Each test should have a Setup/Cleanup phase, test the presence and function of the mandatory actions, and test the Lock/PID file behaviour. +The tests should be a variation on this [https://fedorahosted.org/autoqa/attachment/wiki/initscripts/template_runtest.sh template]. + +Stuff to do: + 1. Check if there's {{{#!/bin/bash}}} shebang at the beginning of the script. + 1. Check if the package name and service name are correct. + 1. Check if the script tests all of the [https://fedoraproject.org/wiki/Packaging/SysVInitScript#Required_Actions required actions]. + 1. Check if the tests for action exit codes are correct (see [https://fedoraproject.org/wiki/Packaging/SysVInitScript#Exit_Codes_for_the_Status_Action status] and [https://fedoraproject.org/wiki/Packaging/SysVInitScript#Exit_Codes_for_non-Status_Actions non-status] expected results). + 1. Check if there is a {{{rlServiceStop}}} in Setup and {{{rlServiceRestore}}} in Cleaup phase. + 1. If temporary user accounts are created somewhere in the script, check if they are disposed in the Cleanup phase (possibly move the user creation to Setup phase). + 1. Check if the Lock/PID file names are correct (some services may store the Lock/PID file in non-standard way - investigate in the init file itself ({{{/etc/init.d/$SERVICE}}}) + 1. If the tests creates/changes config files for the service, make sure that these get backed up and restored if present before the testrun ({{{rlFileBackup}}} {{{rlFileRestore}}}). + 1. If the script needs to delete a file, make sure it gets backed up and restored. + 1. Check for any other signs of destructive behavior (changed/deleted files, stopped/started services, ...). + +Detailed description of some of these points to be found [#Detailedpre-check here] + +== Run the script == + +Once you've finished the pre-check, install the $PACKAGE and run the runtest.sh as root (unprivileged user usually can not start/stop services and delete/create Lock/PID files). +{{{ +su root +chmod a+x runtest.sh +./runtest.sh +}}} + +Check if the script runs correctly, examine the results and make sure there are no errors. + +=== Share the result === + +Once you've finished testing, report your findings to the [https://fedorahosted.org/autoqa/query?status=new&status=assigned&status=reopened&milestone=Initscripts+review appropriate TRAC ticket]. + +If everything went correctly, congratulations! You can close the ticket. + +If problems were encountered, please add details of the failure. Bonus points for providing a patch! Patches can be attached to the relevant TRAC ticket, or send it to the [https://fedorahosted.org/mailman/listinfo/autoqa-devel autoqa-devel mailing list] with the subject {{{[PATCH][INITSCRIPTS] - #TracTicketNumber TracTicketDescription}}}. + +== Detailed pre-check == + +=== Check if the script tests all of the required actions === +=== Check if the tests for action exit codes are correct (see status and non-status expected results) === + +Each initscript should provide some certain 'actions' (e.g. start, stop, restart...), the list of required actions is here and these actions need to return certain exit codes according to and based on the type of action. + +'''Example:''' Command `service $servicename start` should return 0 if everything | went ok, and it should return 4 or 1 if you try to run the command | with non-privileged account. + +So you should check, if the test is trying to use all these actions (i.e. if there is a line like `rlRun "service $SERVICE $ACTION" "$REQUIRED EXIT CODE"`) and if these tests (the rlRuns) expect the exit code stated by the documentation. + +=== If temporary user accounts are created somewhere in the script, check if they are disposed in the Cleanup phase (possibly move the user creation to Setup phase) === + +Some tests (in fact, after the review all the tests should) have a part which tests initscript behaviour while runned as unprivileged user. These scripts create a temporary user account, but sometimes, they do it somewhere else than in the 'Setup' phase, and destroy the accound elsewhere than in Cleanup phase (see e.g. auditd test to see how it should look like in the correct state). + +So your task here is to go through the code and find the possible user creation/destruction and move it to appropriate phase. + +=== Check if the Lock/PID file names are correct (some services may store the Lock/PID file in non-standard way - investigate in the init file itself (/etc/init.d/$SERVICE) === + +Since the tests are taken from the RHEL crew, lock/pid files do not necessary need to be in the same file on Fedora. Your task here is to check, if the path to the Lock/PID file in the test is correct for the appropriate initscript. The correct path is to be found inside the initscript itself. + +=== If the tests creates/changes config files for the service, make sure that these get backed up and restored if present before the testrun (rlFileBackup rlFileRestore) === +=== Check for any other signs of destructive behavior (changed/deleted files, stopped/started services, ...) === + +Some of the tests may create/change/delete files during their execution. Since we want these test to be non-destructive you should go through the tests source code and find all (if any) occurences of this behaviour. Once found you should use rlFileBackup on the respective file _before_ it's changed/deleted and add a rlFileRestore to the cleaup phase. +You should also delete any file which is created by the test in other directory than /tmp, also in the cleanup phase. + diff --git a/wiki/rpmguard.txt b/wiki/rpmguard.txt new file mode 100644 index 0000000..9294ca1 --- /dev/null +++ b/wiki/rpmguard.txt @@ -0,0 +1,23 @@ += rpmguard = + +''rpmguard'' is a simple Python tool for checking differences between RPM packages. It is very similar to rpmdiff, but it prints only important changes, +not all of them. Therefore it can be used every time a new package is built to easily see if something hasn't gone completely wrong. + +You can read more information about rpmguard in the [https://git.fedorahosted.org/cgit/autoqa.git/plain/tests/rpmguard/README README]. + +== Download == + + * [https://git.fedorahosted.org/cgit/autoqa.git/tree/tests/rpmguard Source code browser] + * Git checkout: `git clone git://git.fedorahosted.org/autoqa.git autoqa` [[BR]] (see autoqa/tests/rpmguard) + +== Documentation == + + * [RpmguardChecks List of rpmguard checks] + +== Contact == + + * Issue tracker: + * [/autoqa/query?status=new&status=assigned&status=reopened&summary=~rpmguard&order=priority list of tickets] + * [/autoqa/newticket?keywords=rpmguard&summary=rpmguard:&component=tests create new ticket] (log in first) + * Mailing list: [https://fedorahosted.org/mailman/listinfo/autoqa-devel autoqa-devel] + * Maintainer: [https://fedoraproject.org/wiki/User:Kparal Kamil Páral]