| |
@@ -7,117 +7,242 @@
|
| |
|
| |
Description
|
| |
===========
|
| |
- When Quality Engineering requests a TC or RC they do so by filing or reopening
|
| |
- a ticket in Release Engineering pagure.
|
| |
+ All composes are defined by configuration files kept in the `pungi-fedora repository`_.
|
| |
+
|
| |
+ Composes fall into two categories. They may be release candidates created on demand
|
| |
+ or nightly composes set to run at a scheduled time each day.
|
| |
+
|
| |
+ =============== ===================== =======================
|
| |
+ Compose Name Configuration File Compose Script
|
| |
+ =============== ===================== =======================
|
| |
+ Atomic fedora-atomic.conf twoweek-nightly.sh
|
| |
+ Docker fedora-docker.conf docker-nightly.sh
|
| |
+ Cloud fedora-cloud.conf cloud-nightly.sh
|
| |
+ Modular fedora-modular.conf modular-nightly.sh
|
| |
+ Nightly fedora.conf nightly.sh
|
| |
+ =============== ===================== =======================
|
| |
+
|
| |
+ When Quality Engineering (QE) requests a Release Candidate (RC) they do so by opening
|
| |
+ an issue in the releng repository on pagure. Release candidate composes are not
|
| |
+ currently automated.
|
| |
+
|
| |
+ =============== ===================== =======================
|
| |
+ Compose Name Configuration File Compose Script
|
| |
+ =============== ===================== =======================
|
| |
+ Alpha fedora-alpha.conf release-candidate.sh
|
| |
+ Beta fedora-beta.conf release-candidate.sh
|
| |
+ GA fedora-final.conf release-candidate.sh
|
| |
+ =============== ===================== =======================
|
| |
+
|
| |
+ .. note::
|
| |
+ Fedora 26 was the last release to include an Alpha release candidate.
|
| |
+
|
| |
|
| |
Action
|
| |
======
|
| |
- Create the full product tree.
|
| |
+ The following procedures are for release candidates only. They do not apply to the
|
| |
+ scheduled nightly composes.
|
| |
|
| |
- Composing the Tree
|
| |
- ------------------
|
| |
- #. Log into a compose system: ``compose-x86-01`` or
|
| |
- ``arm01-releng00.arm.fedoraproject.org``
|
| |
+ Review Compose Tags
|
| |
+ -------------------
|
| |
+ #. List any pre-existing builds in the current compose tag
|
| |
|
| |
::
|
| |
|
| |
- $ ssh -A compose-x86-01.phx2.fedoraproject.org
|
| |
+ $ koji list-tagged f[release_version]-compose
|
| |
|
| |
- .. note::
|
| |
- ssh agent forwarding is needed to enable sshing to boxes for composing the
|
| |
- different arch trees.
|
| |
+ #. Verify pre-existing builds are in compose tags
|
| |
+
|
| |
+ The tagged builds from the previous composes should all be present in the
|
| |
+ output from the previous step. Consult the request ticket for the list
|
| |
+ of builds expected in this output.
|
| |
+
|
| |
+ .. note::
|
| |
+ The very first run of an Alpha, Beta, or GA compose should have no builds
|
| |
+ listed under the compose tag. It is important to clear pre-existing builds
|
| |
+ from the compose tag when moving between the Alpha, Beta and RC composes.
|
| |
+ Verify that these builds were removed.
|
| |
+
|
| |
+ ::
|
| |
+
|
| |
+ $ koji list-tagged f[release_version]-compose
|
| |
+ $ koji untag-build --all f[release_version]-compose [build1 build2 ...]
|
| |
+
|
| |
+ .. note::
|
| |
+ The order in which packages are added into the f[release_version]-compose tag
|
| |
+ matter. If the builds are untagged erroneously then special attention should
|
| |
+ be given to adding them back correctly.
|
| |
+
|
| |
+
|
| |
+ #. Add builds specified by QE to the current compose tag
|
| |
+
|
| |
+ ::
|
| |
+
|
| |
+ $ koji tag-build f[release_version]-compose [build1 build2 ...]
|
| |
+
|
| |
+ .. note::
|
| |
+ These steps may be completed on a local machine as long as the user has
|
| |
+ appropriate permissions in the koji tool.
|
| |
+
|
| |
+ Package Signing before the Compose
|
| |
+ ----------------------------------
|
| |
+ #. Check for unsigned packages
|
| |
+
|
| |
+ ::
|
| |
+
|
| |
+ $ koji list-tagged f[release_version]-signing-pending
|
| |
+
|
| |
+ .. note::
|
| |
+ If there are unsigned builds then wait for the automated queue to pick
|
| |
+ them up and sign them. Contact a member of the Fedora infrastructure team
|
| |
+ if the package signing has taken more than thirty minutes.
|
| |
+
|
| |
+
|
| |
+ Running the Compose
|
| |
+ -------------------
|
| |
+ #. Update the pungi-fedora config file
|
| |
+ Composes use a configuration file to construct the compose. Each
|
| |
+ compose uses its own configuration. The ``global_release`` variable
|
| |
+ should start from 1.1 and the second number should increment each time
|
| |
+ a new compose is created.
|
| |
+
|
| |
+ * Alpha - ``fedora-alpha.conf``
|
| |
+ * Beta - ``fedora-beta.conf``
|
| |
+ * RC - ``fedora-final.conf``
|
| |
+
|
| |
+ #. Log into the compose backend
|
| |
|
| |
- #. update or checkout the git repo
|
| |
+ ::
|
| |
+
|
| |
+ $ ssh compose-x86-01.phx2.fedoraproject.org
|
| |
+
|
| |
+ #. Open a screen session
|
| |
|
| |
::
|
| |
|
| |
- $ cd releng/
|
| |
- $ git pull --rebase
|
| |
+ $ screen
|
| |
+
|
| |
+ #. Obtain the pungi-fedora branch for the current compose
|
| |
|
| |
- or
|
| |
+ The first time any user account executes a compose the pungi-fedora
|
| |
+ git repository must be cloned. The compose candidate script that
|
| |
+ invokes pungi should be run from
|
| |
+ ``compose-x86-01.phx2.fedoraproject.org``.
|
| |
|
| |
::
|
| |
|
| |
- $ git clone https://git.fedorahosted.org/git/releng
|
| |
+ $ git clone ssh://git@pagure.io/pungi-fedora.git
|
| |
|
| |
- #. Sign rpms for bleed repo
|
| |
+ Enter the pungi-fedora directory.
|
| |
|
| |
::
|
| |
|
| |
- ssh releng04.phx2.fedoraproject.org
|
| |
+ $ cd pungi-fedora
|
| |
|
| |
- repeat step 2 on signing box.
|
| |
+ If the clone step above was not required then fully update the existing
|
| |
+ repository checkout from pagure.
|
| |
|
| |
::
|
| |
|
| |
- $ cd releng/scripts
|
| |
- $ NSS_HASH_ALG_SUPPORT=+MD5 ./sigulsign_unsigned.py fedora-22 -v --write-all <build nvrs>
|
| |
+ $ git fetch origin
|
| |
+ $ git checkout f[release_version]
|
| |
+ $ git pull origin f[release_version]
|
| |
+
|
| |
+ #. Run the compose
|
| |
+
|
| |
+ ::
|
| |
+
|
| |
+ $ sudo ./release-candidate.sh [Alpha|Beta|RC]-#.#
|
| |
+
|
| |
+ The numbering scheme begins with 1.1 and the second number is incremented
|
| |
+ after each compose.
|
| |
+
|
| |
+ .. note::
|
| |
+ Pungi requires numbers in the format #.# as an argument. It is because
|
| |
+ of this that composes always start with the number 1 and the second
|
| |
+ number is incremented with each compose.
|
| |
+
|
| |
+ .. note::
|
| |
+ If the compose fails with a directory missing error, then create
|
| |
+ the compose directory with ``mkdir /mnt/koji/compose/[release_version]``
|
| |
+
|
| |
+ Syncing the Compose
|
| |
+ -------------------
|
| |
+
|
| |
+ We sync the compose to ``/pub/alt/stage`` to enable faster access to new content
|
| |
+ for QA and the larger Fedora community.
|
| |
|
| |
- #. Update the bleed repo on <code>compose-x86-01</code>:
|
| |
+ #. Log into the compose backend
|
| |
|
| |
::
|
| |
|
| |
- $ ~/releng/scripts/makebleed <build nvrs>
|
| |
+ $ ssh compose-x86-01.phx2.fedoraproject.org
|
| |
+
|
| |
+ #. Open a screen session
|
| |
+
|
| |
+ ::
|
| |
|
| |
- #. Kick off compose on a compose system: ``compose-x86-01`` or
|
| |
- ``arm03-releng00.arm.fedoraproject.org``
|
| |
+ $ screen
|
| |
|
| |
- for a TC
|
| |
+ #. Check the status of the compose
|
| |
|
| |
::
|
| |
|
| |
- $ cd ~/releng/scripts
|
| |
- $ ./run-pungi 22_Beta_TC9 "" 20150409
|
| |
+ $ cat /mnt/koji/compose/[release_version]/[compose_id]/STATUS
|
| |
+
|
| |
+ Do not continue with any further steps if the output above is ``DOOMED``.
|
| |
+
|
| |
+ #. Create the directory targeted for the copy
|
| |
+ ::
|
| |
|
| |
- for a RC
|
| |
+ $ sudo -u ftpsync mkdir -p /pub/alt/stage/[release_version]_[release_label]-[#.#]
|
| |
|
| |
+ #. Locate the compose directory that will be the copy source
|
| |
::
|
| |
|
| |
- $ cd ~/releng/scripts
|
| |
- $ ./run-pungi 22_Beta _RC2 20150413
|
| |
+ $ ls /mnt/koji/compose/[release_version]/[compose_id]
|
| |
|
| |
- #. Check the compose
|
| |
+ .. note::
|
| |
+ Take care executing the synchronization if the next compose
|
| |
+ is already running. Be sure to grab the correct directory.
|
| |
|
| |
- check the tree under ``/mnt/fedora_koji/compose/<Compose>/<Release>/`` for
|
| |
- completeness
|
| |
+ If in doubt, check /mnt/koji/compose/[release_version]/[compose_id]/STATUS
|
| |
+ to be sure it is finished.
|
| |
|
| |
- #. open up the tree:
|
| |
+ #. Run the synchronization one-liner
|
| |
+
|
| |
+ The synchronization of the completed compose to the public domain is currently
|
| |
+ a one-liner shell script. Pay close attention to what needs replaced in the example
|
| |
+ below.
|
| |
|
| |
::
|
| |
|
| |
- sg releng "chmod 755 /pub/alt/stage/<Compose>/"
|
| |
+ $ for dir in Everything Cloud CloudImages Docker Labs Server Spins Workstation WorkstationOstree metadata; do sudo -u ftpsync rsync -avhH /mnt/koji/compose/26/Fedora-26-20170328.0/compose/$dir/ /pub/alt/stage/26_Alpha-1.4/$dir/ --link-dest=/pub/fedora/linux/development/26/Everything/ --link-dest=/pub/alt/stage/26_Alpha-1.1/Everything/ --link-dest=/pub/alt/stage/26_Alpha-1.2/Everything/ --link-dest=/pub/alt/stage/26_Alpha-1.3/Everything --link-dest=/pub/alt/stage/26_Alpha-1.4/Everything; done
|
| |
+
|
| |
+ .. note::
|
| |
+ This one-liner prompts for the password+token several times over the course of its runtime. If the
|
| |
+ login window is missed it will skip an entire variant. Just check the source and destination after
|
| |
+ completion and, if there is a directory missing, run the script again.
|
| |
+
|
| |
+ #. Update the issue in the releng pagure repository
|
| |
|
| |
- #. Close the ticket:
|
| |
+ Once the compose and sync is complete the issue in pagure should be updated and closed.
|
| |
|
| |
- Copy the output pasted at the end of run-pungi and paste into the ticket and
|
| |
- close it.
|
| |
+ .. admonition:: Standard Ticket Verbage
|
| |
|
| |
- Update the ticket
|
| |
- -----------------
|
| |
- The ticket should be closed when the compose has been opened up pasting in the
|
| |
- output from run-pungi
|
| |
+ Compose is done and available from https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170328.0/compose/ it has been synced to http://dl.fedoraproject.org/pub/alt/stage/26_Alpha-1.4/ rpms have all be hardlinked to /pub/fedora/linux/development/26/
|
| |
|
| |
Verification
|
| |
============
|
| |
- Verification can be done as the task steps are being performed.
|
| |
|
| |
- Image Creation
|
| |
- --------------
|
| |
- When pungi exits, you can verify that the ``development/<arch>/os/images/``
|
| |
- directory exists and has content. That path is relative to the destination
|
| |
- directory you provide pungi.
|
| |
+ The method for verifying a compose has completed is checking ``/mnt/koji/compose/[release_version]/[compose_dir]/STATUS``.
|
| |
+ Any status other than DOOMED is OK.
|
| |
|
| |
- Updating the Ticket
|
| |
- -------------------
|
| |
- One can click the link provided in the ticket update and ensure the path is
|
| |
- correct. Verifying that the ticket is closed should be pretty self evident.
|
| |
-
|
| |
- Consider Before Running
|
| |
+ Consider before Running
|
| |
=======================
|
| |
- Many things can hinder a compose, broken deps in the chroot set, broken deps
|
| |
- in the compose set, bugs in the compose software, etc... If any problem is ran
|
| |
- into along the way, it is best to alert QA via a ticket update, and then work
|
| |
- with the appropriate party to clear the obstruction.
|
| |
+ Composes and file synchronizations should be run in a screen session on a remote machine. This enables the
|
| |
+ operator to withstand network connection issues.
|
| |
+
|
| |
+ .. _pungi-fedora repository:
|
| |
+ https://pagure.io/pungi-fedora
|
| |
|
| |
- Some of these tasks take a long time to finish, so it is highly recommended
|
| |
- that you run these tasks in a screen session.
|
| |
kellin
commented 7 years ago