PR#179: Initial import of releng SOPs - infra-docs-fpo

antora.yml

file modified

+1

		`@@ -16,3 +16,4 @@`
		`- modules/ROOT/nav.adoc`
		`- modules/developer_guide/nav.adoc`
		`- modules/sysadmin_guide/nav.adoc`
		`+ - modules/release_guide/nav.adoc`

modules/release_guide/nav.adoc

file added

+10

		`@@ -0,0 +1,10 @@`
		`+ * xref:release_process.adoc[Release process]`
		`+ * xref:index.adoc[Release guide]`
		`+ ** xref:mass_rebuild.adoc[Mass Rebuild]`
		`+ ** xref:sop_file_ftbfs.adoc[File FTBFS bugs]`
		`+ ** xref:sop_mass_branching.adoc[Mass Branching]`
		`+ ** xref:sop_bodhi_activation.adoc[Updates testing activation]`
		`+ ** xref:beta_freeze.adoc[Beta freeze]`
		`+ ** xref:beta_release.adoc[Beta release]`
		`+ ** xref:final_freeze.adoc[Final freeze]`
		`+ ** xref:final_release.adoc[Final release]`

modules/release_guide/pages/_partials/attributes.adoc

file added

+5

		`@@ -0,0 +1,5 @@`
		`+ :year: 2023`
		`+ :rawhide: 39`
		`+ :branched: 38`
		`+ :current: 37`
		`+ :old_release: 36`

modules/release_guide/pages/architecture.adoc

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ [[architecture]]`
		`+ == Fedora Release Engineering Architecture`
		`+`
		`+ Here you will find an index of documentation that describes the Fedora`
		`+ Release Engineering Architecture. The goal of this is to provide an`
		`+ overview of the environment and various systems that are used to build`
		`+ it as a reference to members of the Fedora Community as well as`
		`+ materials to help prospective Release Engineering Community members.`
		`+`
		`+ === Architecture Documents`
		`+`
		`+ layered_image_build_service workflow_automation`

modules/release_guide/pages/beta_freeze.adoc

file added

+1

		`@@ -0,0 +1,1 @@`
		`+ == Fedora Beta Freeze`
		`\ No newline at end of file`

modules/release_guide/pages/beta_release.adoc

file added

+1

		`@@ -0,0 +1,1 @@`
		`+ == Fedora Beta Release`
		`\ No newline at end of file`

modules/release_guide/pages/contributing.adoc

file added

+248

		`@@ -0,0 +1,248 @@`
		`+ == Fedora Release Engineering Contributing Guide`
		`+`
		`+ Fedora Release Engineering works with many different utilities that are`
		`+ maintained in respective upstream locations. Fedora Release Engineering`
		`+ maintains an "Upstream First" policy such that if there is a bug in an`
		`+ utility or a feature needed, we pursue that upstream before entertaining`
		`+ the idea of carrying a Fedora specific patch.`
		`+`
		`+ Fedora Release Engineering also has a number of scripts and utilities`
		`+ that are strictly Fedora centric in order to automate tasks and`
		`+ processes as they relate to Fedora itself which is what is contained in`
		`+ the https://pagure.io/releng[releng git repository] hosted on`
		`+ https://pagure.io/pagure[Pagure]. If you would like to contribute to`
		`+ something in this repository, please reference the`
		+ `contributing-to-releng` section.
		`+`
		`+ === Contributing to releng`
		`+`
		`+ In order to contribute to the releng http://www.git-scm.com[git]`
		`+ repository (where the source reStructured Text version of these docs`
		`+ live), you should first have a`
		`+ https://admin.fedoraproject.org/accounts[Fedora Account System] (FAS)`
		`+ account, log into https://pagure.io[pagure.io] and the fork the`
		`+ https://pagure.io/releng[releng git repository].`
		`+`
		`+ Once you've forked the https://pagure.io/releng[releng git repository],`
		`+ you will need to set the remote upstream git clone of your fork in order`
		`+ to track the official releng repository. While not mandatory, it's`
		+ conventional to call the remote upstream the name `upstream` which can
		`+ be done with the following command while within the directory of your`
		`+ local git clone of your fork.`
		`+`
		`+ [source,bash]`
		`+ ----`
		`+ $ git remote add upstream https://pagure.io/releng.git`
		`+ ----`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ If you are not currently familiar with git, it is highly recommended to`
		`+ visit git's upstream and familiarize yourself.`
		`+`
		`+ http://www.git-scm.com/`
		`+ ====`
		`+`
		`+ ==== RelEng Developer Workflow`
		`+`
		`+ There are many options for developer workflow, but the recommended`
		`+ workflow for Fedora releng repository is known as a`
		`+ "http://www.git-scm.com/book/en/v2/Git-Branching-Branching-Workflows#Topic-Branches[Topic`
		`+ Branch]" based workflow in git nomenclature. This is how Fedora Release`
		`+ Engineering contributors can submit changes to the`
		`+ https://pagure.io/releng[releng git repository] for both code and`
		`+ documentation.`
		`+`
		`+ The general workflow is as follows:`
		`+`
		+ * Checkout `main` branch of the local git clone of your releng
		`+ repository clone.`
		`+ +`
		`+ ....`
		`+ $ git checkout main`
		`+ ....`
		`+ * Pull upstream and merge into local main to make sure that your main`
		`+ branch is in line with the latest changes from upstream. Then push it to`
		`+ your clone so that origin knows about the changes.`
		`+ +`
		`+ ....`
		`+ $ git pull --rebase upstream main`
		`+ $ git push origin main`
		`+ ....`
		`+ * Create a topic branch from main.`
		`+ +`
		`+ ....`
		`+ $ git checkout -b my-topic-branch`
		`+ ....`
		`+ * Make changes in your topic branch and commit them to your topic`
		`+ branch.`
		`+ +`
		`+ ....`
		`+ $ vim somefile.py`
		`+`
		`+ .... make some change ...`
		`+`
		`+ $ git add somefile.py`
		`+ $ git commit -s -m "awesome patch to somefile.py"`
		`+ ....`
		`+ * This step is optional but recommended in order to avoid collisions`
		`+ when submitting upstream. Here we will checkout main again and merge`
		+ `upstream/main` so that we can resolve any conflicts locally.
		`+ +`
		`+ ....`
		`+ $ git checkout main`
		`+ $ git pull --rebase upstream main`
		`+ $ git push origin main`
		`+ ....`
		`+ * Rebase on main before submitting a pull request`
		`+ +`
		`+ ....`
		`+ $ git rebase main`
		`+`
		`+ ..... Resolve any conflicts if needed ......`
		`+ ....`
		`+ * Push your topic branch to your fork's origin in pagure.`
		`+ +`
		`+ ....`
		`+ $ git push origin my-topic-branch`
		`+ ....`
		`+ * Open a pull request in Rel Eng Pagure.`
		`+ https://pagure.io/releng/pull-requests`
		`+`
		`+ ==== Developer Workflow Tips and Tricks`
		`+`
		`+ Below are some Fedora Release Engineering Developer Workflow Tips and`
		`+ Tricks used by current members of the team in order to help assist with`
		`+ development.`
		`+`
		`+ ===== pullupstream`
		`+`
		+ The following is an useful shell function to place in your `~/.bashrc`
		`+ to help automate certain aspects of the developer workflow. It will`
		`+ allow you to merge in the upstream main or devel branch into your forked`
		`+ repository for easily keeping in line with the upstream repository.`
		`+`
		+ The following is the bash function to be added to your `~/.bashrc` and
		+ make sure to `source ~/.bashrc` after adding it in order to "enable" the
		`+ function.`
		`+`
		`+ ....`
		`+ pullupstream () {`
		`+ if [[ -z "$1" ]]; then`
		`+ printf "Error: must specify a branch name (e.g. - main, devel)\n"`
		`+ else`
		`+ pullup_startbranch=$(git describe --contains --all HEAD)`
		`+ git checkout $1`
		`+ git pull --rebase upstream $1`
		`+ git push origin $1`
		`+ git checkout ${pullup_startbranch}`
		`+ fi`
		`+ }`
		`+ ....`
		`+`
		`+ With the function in place you can easily pull and merge in the releng`
		`+ main branch even while using a topic branch as follows:`
		`+`
		`+ ....`
		`+ $ git status`
		`+ On branch docs`
		`+ nothing to commit, working directory clean`
		`+`
		`+ $ pullupstream main`
		`+ Switched to branch 'main'`
		`+ Your branch is up-to-date with 'origin/main'.`
		`+ Already up-to-date.`
		`+ Everything up-to-date`
		`+ Switched to branch 'docs'`
		`+`
		`+ $ git status`
		`+ On branch docs`
		`+ nothing to commit, working directory clean`
		`+ ....`
		`+`
		`+ Now that you're back on your topic branch you can easily rebase on your`
		`+ local main branch in order to resolve any merge conflicts that may come`
		`+ up for clean pull request submission.`
		`+`
		`+ ....`
		`+ $ git rebase main`
		`+ Current branch docs is up to date.`
		`+ ....`
		`+`
		`+ === RelEng Upstream Tools`
		`+`
		`+ Fedora Release Engineering uses many tools that exist in their own`
		`+ upstream project space. These are tools that every Fedora Release`
		`+ Engineer should be familiar with and in the event there is a bug or a`
		`+ feature needed, we should participate in the respective upstream to`
		`+ resolve the issue first before considering carrying a Fedora specific`
		`+ patch.`
		`+`
		`+ ==== Tools List`
		`+`
		`+ ===== Tools Release Engineering is actively involved with upstream`
		`+`
		`+ Below are a set of tools that are centric to the Release Engineering`
		`+ team and our processes. We actively engage with upstreams of these`
		`+ projects. For these tools, we recommend the same git contribution`
		`+ workflow that is outlined above for this git repository.`
		`+`
		`+ * https://pagure.io/koji[koji] -Build System used by Fedora`
		`+ * https://pagure.io/mash[mash] -Tool that creates repositories from koji`
		`+ tags, and solves them for multilib dependencies.`
		`+ * https://pagure.io/pungi[pungi] -Fedora Compose tool`
		`+ * https://github.com/release-engineering/product-definition-center[Product`
		`+ Defintion Center (PDC)] -Repository and API for storing and querying`
		`+ product metadata`
		`+ * https://github.com/release-engineering/koji-containerbuild[koji-containerbuild]`
		`+ -Koji plugin to integrate OSBS with koji`
		`+`
		`+ ===== Tools Release Engineering is actively mostly consumers of`
		`+`
		`+ Below are the set of tools that the Release Engineering team either`
		`+ consumes directly or as the side effect of other tools in the Release`
		`+ Engineering Infrastructure. Tools here should always be engaged upstream`
		`+ in the event of a bug or enhancement needed but are not tools that the`
		`+ Release Engineering team is extremely active in their continued upstream`
		`+ development and will defer to each upstream for recommended`
		`+ contributions workflow.`
		`+`
		`+ * https://pagure.io/fedpkg[fedpkg] -command line utility for Fedora (and`
		`+ EPEL) developers. It interacts with dist-git, koji, rpmbuild, git, etc.`
		`+ * https://pagure.io/rpkg[rpkg] -library for dealing with rpm packaging`
		`+ in a git source control (used by fedpkg)`
		`+ * https://github.com/release-engineering/dist-git[dist-git] -remote Git`
		`+ repository specificaly designed to hold RPM package sources.`
		`+ * http://createrepo.baseurl.org/[creatrepo] -A python program which`
		`+ generate repodata from a set of rpm files.`
		`+ * https://github.com/rpm-software-management/createrepo_c[createrepo_c]`
		`+ -C implementation of createrepo`
		`+ * https://github.com/clalancette/oz[oz] -set of programs and classes to`
		`+ do automated installations of operating systems.`
		`+ * http://imgfac.org/[imagefactory] -imagefactory builds images for a`
		`+ variety of operating system/cloud combinations.`
		`+ * https://pagure.io/sigul[sigul] -An automated gpg signing system`
		`+ * https://github.com/rpm-software-management/mock/wiki[mock] -a tool for`
		`+ building packages in prestine buildroots`
		`+ * http://www.fedmsg.com/en/latest/[fedmsg] -Fedora Infrastructure`
		`+ Message Bus`
		`+ * https://github.com/rhinstaller/lorax[lorax] -tool to build install`
		`+ trees and images`
		`+ * http://www.openshift.org/[OpenShift] -Open Source Platform as a`
		`+ Service by Red Hat`
		`+ * https://github.com/projectatomic/osbs-client[OSBS] -set of utilities`
		`+ that turn OpenShift into a layered image build system`
		`+ * https://fedoraproject.org/wiki/Taskotron[taskotron] -a framework for`
		`+ automated task execution.`
		`+ * http://www.pulpproject.org/[pulp] -a platform for managing`
		`+ repositories of content, such as software packages, and pushing that`
		`+ content out to large numbers of consumer`
		`+ * https://github.com/pulp/crane[crane] -Crane is a small read-only web`
		`+ application that provides enough of the docker registry API to support`
		`+ "docker pull"`
		`+ * https://pagure.io/pagure[pagure] A git centered forge`
		`+ * https://github.com/projectatomic/rpm-ostree[rpm-ostree] -Store RPMs in`
		`+ OSTree repository, and atomically upgrade from commits`
		`+ * https://wiki.gnome.org/Projects/OSTree[ostree] -a tool for managing`
		`+ bootable, immutable, versioned filesystem trees.`

modules/release_guide/pages/final_freeze.adoc

file added

+1

		`@@ -0,0 +1,1 @@`
		`+ == Fedora Final Freeze`
		`\ No newline at end of file`

modules/release_guide/pages/final_release.adoc

file added

+5

		`@@ -0,0 +1,5 @@`
		`+ == Fedora Final Release`
		`+`
		`+ === Update FedoraPreviousPrevious.yaml in ansible repository`
		`+`
		`+ set the variable to False`

modules/release_guide/pages/index.adoc

file added

+150

		`@@ -0,0 +1,150 @@`
		`+ == Fedora Release Engineering`
		`+`
		`+ Contents:`
		`+`
		`+ overview philosophy contributing troubleshooting architecture sop`
		`+`
		`+ This page contains information about the Fedora Release Engineering`
		`+ team.`
		`+`
		`+ [[releng-contact-info]]`
		`+ === Contact Information`
		`+`
		+ * IRC: `#fedora-releng` on irc.libera.chat
		`+ * Mailing List: https://admin.fedoraproject.org/mailman/listinfo/rel-eng[rel-eng@lists.fedoraproject.org]`
		`+ * Issue tracker: https://pagure.io/releng/new_issue[Fedora Releng Pagure Tickets]`
		`+`
		`+ If you want the to get something done (e.g. moving packages to`
		`+ buildroots or into frozen compositions) by the ReleaseEngineering Team,`
		`+ please create a ticket in the issue tracker mentioned above. Please`
		`+ enter your FAS-username or e-mail address in the respective textbox, to`
		`+ make sure the team can contact you.`
		`+`
		`+ [[index-team-composition]]`
		`+ === Team Composition`
		`+ * https://fedoraproject.org/wiki/User:kevin[Kevin Fenzi (nirik)]`
		`+ * https://fedoraproject.org/wiki/User:sharkcz[Dan Horák (sharkcz)](secondary arches)`
		`+ * https://fedoraproject.org/wiki/User:pbrobinson[Peter Robinson(pbrobinson)]`
		`+ * https://fedoraproject.org/wiki/User:maxamillion[Adam Miller(maxamillion)]`
		`+ * https://fedoraproject.org/wiki/User:humaton[Tomas Hrcka(humaton)]`
		`+`
		`+ Release Team members are approved by FESCo. However, FESCo has delegated`
		`+ this power to the Release Team itself. If you want to join the team,`
		+ please read `join-releng`.
		`+`
		`+ === What is Fedora Release Engineering?`
		`+`
		+ For a Broad Overview, see `overview <overview>`.
		`+`
		`+ === Why we do things the way we do them`
		`+`
		`+ For information on the Fedora Release Engineering Philosophy, see`
		+ `philosophy <philosophy>`.
		`+`
		`+ === Fedora Release Engineering Leadership`
		`+`
		`+ Mohan Boddu (mboddu on IRC, FAS username mohanboddu)`
		`+`
		`+ Leadership is currently appointed by FESCo with input from the current`
		`+ release team.`
		`+`
		`+ === Things we Do`
		`+`
		`+ * {blank}`
		`+ +`
		`+ Create official Fedora releases.::`
		`+ ** {blank}`
		`+ +`
		`+ Fedora Products;;`
		`+ *** Cloud`
		`+ *** Server`
		`+ *** Workstation`
		`+ ** Fedora Spins`
		`+ * Report progress towards release from`
		`+ https://fedoraproject.org/wiki/Releases/Branched[branched] creation on.`
		`+ * Give reports to FESCo on changes to processes.`
		`+ * If something is known to be controversial, we let FESCo know before`
		`+ implementing otherwise implementation generally happens concurrently to`
		`+ reporting.`
		`+ * Set policy on freeze management`
		`+ * Administrate the build system(s)`
		`+ * Remove unmaintained packages from Fedora`
		`+ * Push updated packages`
		`+ * write and maintain tools to compose and push Fedora`
		`+`
		`+ [[join-releng]]`
		`+ === Joining Release Engineering`
		`+`
		`+ Much of rel-eng's communication is via IRC. One of the best ways to`
		`+ initially get involved is to attend one of the meetings and say that`
		`+ you're interested in doing some work during the open floor at the end of`
		`+ the meeting. If you can't make the meeting times, you can also ping one`
		`+ of us on IRC or sign up for the`
		`+ https://admin.fedoraproject.org/mailman/listinfo/rel-eng[mailing list].`
		`+`
		`+ Since release engineering needs special access to systems essential to`
		`+ Fedora people new to rel-eng will usually get access a little bit at a`
		`+ time. Typically people won't immediately be granted the ability to sign`
		`+ packages and push updates for example. A couple of tasks you could start`
		`+ out with are troubleshooting why builds are failing (and if rel-eng`
		`+ could take actions to fix it) as the requests are submitted to pagure or`
		`+ help with scripts for various rel-eng tasks.`
		`+`
		`+ There are also a number of tools that Fedora Release Engineering uses`
		`+ and relies upon, working on improving these upstream to fascilitate with`
		`+ new things that the Fedora Project is aiming to deliver is also a great`
		`+ way to get involved with Fedora Rel-Eng.`
		`+`
		`+ === How we do it`
		`+`
		+ See our `Standard Operating Procedures <sop>` for details on how we do
		`+ the things we do.`
		`+`
		`+ Most discussions regarding release engineering will happen either in`
		`+ [.title-ref]##fedora-releng# or on the releng mailing list. For`
		+ requests, please consult the `releng-contact-info`
		`+`
		`+ === Meetings`
		`+`
		`+ rel-eng holds regular meetings every Monday at 14:30 UTC in`
		`+ [.title-ref]##fedora-meeting-3# on the Libera IRC network.`
		`+`
		`+ * https://pagure.io/releng/issues?status=Open&tags=meeting[Meeting`
		`+ agendas] are created from open tickets in pagure that contain the`
		`+ meeting keyword.`
		`+`
		`+ ==== Meeting Minutes`
		`+`
		`+ Minutes are posted to the rel-eng mailing list. They are also available`
		`+ at the`
		`+ https://meetbot.fedoraproject.org/sresults/?group_id=releng&type=team[Meetbot`
		`+ team page for releng]`
		`+`
		`+ There are also`
		`+ https://fedoraproject.org/wiki/ReleaseEngineering/Meetings[historical`
		`+ Meeting Minutes for 2007-04-16 to 2009-05-04].`
		`+`
		`+ === Current activities`
		`+`
		`+ See our https://pagure.io/releng/issues[ticket queue] for the things we`
		`+ are currently working.`
		`+`
		`+ See https://fedoraproject.org/wiki/Releases[Releases] for information`
		`+ about Fedora releases, including schedules.`
		`+`
		`+ === Freeze Policies`
		`+`
		`+ * https://fedoraproject.org/wiki/Milestone_freezes[Milestone (Alpha,`
		`+ Beta, Final) freezes]`
		`+ * https://fedoraproject.org/wiki/Software_String_Freeze_Policy[String`
		`+ Freeze Policy] (Same time as Alpha Freeze)`
		`+ * https://fedoraproject.org/wiki/Changes/Policy[Change freeze policy]`
		`+ (that's 'Change' as in 'feature')`
		`+ * https://fedoraproject.org/wiki/Updates_Policy[Updates Policy] (not`
		`+ technically a freeze, but of interest)`
		`+`
		`+ === Indices and tables`
		`+`
		+ * `genindex`
		+ * `modindex`
		+ * `search`

modules/release_guide/pages/layered_image_build_service.adoc

file added

+263

		`@@ -0,0 +1,263 @@`
		`+ == Fedora Layered Image Build System`
		`+`
		`+ The Fedora Layered Image Build System aims to provide an new type of`
		`+ official binary artifact produced by Fedora. Currently, we produce two`
		`+ main types of artifacts: RPMs, and images. The RPMs are created in`
		`+ https://fedoraproject.org/wiki/Koji[Koji] from specfiles in dist-git.`
		`+ The images come in different formats, but have in common creation in`
		`+ Koji from kickstart files — this includes the official Fedora Docker`
		`+ Base Image. This change introduces a new type of image, a`
		`+ https://github.com/docker/docker/[Docker] Layered Image, which is`
		`+ created from a Dockerfile and builds on top of that base image.`
		`+`
		`+ === Layered Image Build Service Architecture`
		`+`
		`+ ....`
		`+ +------------------------------+`
		`+ \| Future Items to Integrate \|`
		`+ +------------------------------+`
		`+ \| +--------------------------+ \|`
		`+ \| \|PDC Integration \| \|`
		`+ \| +--------------------------+ \|`
		`+ \| \|New Hotness \| \|`
		`+ \| +--------------------------+ \|`
		`+ \| \|Other??? \| \|`
		`+ \| +--------------------------+ \|`
		`+ \| Magical Future \|`
		`+ \| \|`
		`+ +------------------------------+`
		`+`
		`+`
		`+`
		`+ +------------------+`
		`+ \|Fedora \|`
		`+ \|Users/Contributors\|`
		`+ +--------+---------+`
		`+ ^`
		`+ \| +----------------+`
		`+ \| \| Fedora \|`
		`+ +-------+-----------+ \| Layered Image \|`
		`+ \| Fedora Production \| \| Maintainers \|`
		`+ \| Docker Registry \| \| \|`
		`+ +-------------------+ +-------+--------+`
		`+ ^ \|`
		`+ \| V`
		`+ \| +-------------------+--+`
		`+ +-----------+------------+ \| \|`
		`+ \| RelEng Automation \| \| +------------------+ \|`
		`+ \| (Release) \| \| \|Dockerfile \| \|`
		`+ +-----------+------------+ \| +------------------+ \|`
		`+ ^ \| \|App "init" scripts\| \|`
		`+ \| \| +------------------+ \|`
		`+ \| \| \|Docs \| \|`
		`+ +---------+------------+ \| +------------------+ \|`
		`+ \| Taskotron \| \| DistGit \|`
		`+ \| Automated QA \| \| \|`
		`+ +---------+-------+----+ +-----------+----------+`
		`+ ^ ^ \|`
		`+ \| \| \|`
		`+ \| \| \|`
		`+ \| \| \|`
		`+ \| +-------------------+ \|`
		`+ \| \| \|`
		`+ [docker images] \| \|`
		`+ \| \| \|`
		`+ \| [fedmsg] \|`
		`+ +---------------+-----------+ \| \|`
		`+ \| \| \| +---------------+`
		`+ \| +----------------------+ \| \| \|`
		`+ \| \|Intermediate Docker \| \| +------+-------------------+ \|`
		`+ \| \|Images for OSBS Builds\| \| \| \| \|`
		`+ \| +----------------------+ \| \| +----------------------+ \| \|`
		`+ \| \|Layered Images \| \| \| \|containerbuild plugin \| \| \|`
		`+ \| +----------------------+ \| \| +----------------------+ \| \|`
		`+ \| \|docker\|distribution \| \| \| \|rpmbuilds \| \| \|`
		`+ \| +----------------------+ \| \| +----------------------+ \| \|`
		`+ \| Registry \| \| koji \| \|`
		`+ \| \| \| \| \|`
		`+ +-----+----+----------------+ +-----+---+----+-----------+ \|`
		`+ \| ^ \| ^ ^ \|`
		`+ \| \| \| \| \| \|`
		`+ \| \| +---------------------------+ \| \| \|`
		`+ \| \| \| [osbs-client api] \| \| \|`
		`+ \| \| \| +---------------------------+ \| \|`
		`+ \| \| \| \| \| \|`
		`+ \| \| \| \| \| \|`
		`+ V \| V \| \| \|`
		`+ +--+----+---+---+---+ \| V`
		`+ \| \| +-------------------+---+`
		`+ \| +--------------+ \| \|fedpkg container-build +`
		`+ \| \|OpenShift v3 \| \| +-----------------------+`
		`+ \| +--------------+ \|`
		`+ \| \|Atomic Reactor\| \|`
		`+ \| +--------------+ \|`
		`+ \| OSBS \|`
		`+ \| \|`
		`+ +-------------------+`
		`+`
		`+`
		`+ [--------------------- Robosignatory ------------------------------------]`
		`+ [ Every time an image is tagged or changes names, Robosignatory signs it ]`
		`+ [ ]`
		`+ [ NOTE: It's injection points in the diagram are ommitted for brevity ]`
		`+ [------------------------------------------------------------------------]`
		`+ ....`
		`+`
		`+ === Layered Image Build System Components`
		`+`
		`+ The main aspects of the Layered Image Build System are:`
		`+`
		`+ * Koji`
		`+ ** koji-containerbuild plugin`
		`+ * OpenShift Origin v3`
		`+ * Atomic Reactor`
		`+ * osbs-client tools`
		`+ * A docker registry`
		`+ ** docker-distribution`
		`+ * Taskotron`
		`+ * fedmsg`
		`+ * RelEng Automation`
		`+`
		`+ The build system is setup such that Fedora Layered Image maintainers`
		+ will submit a build to Koji via the `fedpkg container-build` command a
		+ `containers` namespace within
		`+ https://fedoraproject.org/wiki/Infrastructure/VersionControl/dist-git[DistGit].`
		`+ This will trigger the build to be scheduled in`
		`+ https://www.openshift.org/[OpenShift] via`
		`+ https://github.com/projectatomic/osbs-client[osbs-client] tooling, this`
		`+ will create a custom`
		`+ https://docs.openshift.org/latest/dev_guide/builds.html[OpenShift Build]`
		`+ which will use the pre-made buildroot Docker image that we have created.`
		`+ The https://github.com/projectatomic/atomic-reactor[Atomic Reactor]`
		+ (`atomic-reactor`) utility will run within the buildroot and prep the
		`+ build container where the actual build action will execute, it will also`
		`+ maintain uploading the`
		`+ https://fedoraproject.org/wiki/Koji/ContentGenerators[Content Generator]`
		`+ metadata back to https://fedoraproject.org/wiki/Koji[Koji] and upload`
		`+ the built image to the candidate docker registry. This will run on a`
		`+ host with iptables rules restricting access to the docker bridge, this`
		`+ is how we will further limit the access of the buildroot to the outside`
		`+ world verifying that all sources of information come from Fedora.`
		`+`
		`+ Completed layered image builds are hosted in a candidate docker registry`
		`+ which is then used to pull the image and perform tests with`
		`+ https://taskotron.fedoraproject.org/[Taskotron]. The taskotron tests are`
		`+ triggered by a http://www.fedmsg.com/en/latest/[fedmsg] message that is`
		`+ emitted from https://fedoraproject.org/wiki/Koji[Koji] once the build is`
		`+ complete. Once the test is complete, taskotron will send fedmsg which is`
		`+ then caught by the [.title-ref]#RelEng Automation# Engine that will run`
		`+ the Automatic Release tasks in order to push the layered image into a`
		`+ stable docker registry in the production space for end users to consume.`
		`+`
		`+ Note that every time the layered image tagged to a new registry,`
		`+ ultimately changing it's name,`
		`+ https://pagure.io/robosignatory[Robosignatory] will automatically sign`
		`+ the new image. This will also occur as part of the Automated Release`
		`+ process as the image will be tagged from the candidate docker registry`
		`+ into the production docker registry in order to "graduate" the image to`
		`+ stable.`
		`+`
		`+ ==== Koji`
		`+`
		`+ https://fedoraproject.org/wiki/Koji[Koji] is the Fedora Build System.`
		`+`
		`+ ==== koji-containerbuild plugin`
		`+`
		`+ The`
		`+ https://github.com/release-engineering/koji-containerbuild[koji-containerbuild]`
		`+ plugin integrates Koji and OSBS so that builds can be scheduled by koji`
		`+ and integrated into the build system with imports of metadata, logs,`
		`+ build data, and build artifacts.`
		`+`
		`+ ==== OpenShift Origin v3`
		`+`
		`+ https://www.openshift.org/[OpenShift Origin v3] is an open source`
		`+ Container Platform, built on top of http://kubernetes.io/[kubernetes]`
		`+ and https://github.com/docker/docker/[Docker]. This provides many`
		`+ aspects of the system needed including a build pipeline for Docker`
		`+ images with custom build profiles, image streams, and triggers based on`
		`+ events within the system.`
		`+`
		`+ ==== Atomic Reactor`
		`+`
		`+ https://github.com/projectatomic/atomic-reactor[Atomic Reactor] is an`
		`+ utility which allows for the building of containers from within other`
		`+ containers providing hooks to trigger automation of builds as well as`
		`+ plugins providing automatic integration many other utilities and`
		`+ services.`
		`+`
		`+ ==== osbs-client tools`
		`+`
		`+ https://github.com/projectatomic/osbs-client[osbs-client] tools allow`
		`+ for users to access the build functionality of`
		`+ https://www.openshift.org/[OpenShift Origin v3] using a simple set of`
		`+ command line utilities.`
		`+`
		`+ ==== docker-registry`
		`+`
		`+ A https://docs.docker.com/registry/[docker-registry] is a stateless,`
		`+ highly scalable server side application that stores and lets you`
		`+ distribute Docker images.`
		`+`
		`+ There are many different implementations of docker-registries, two main`
		`+ ones are supported by the Fedora Layered Image Build System.`
		`+`
		`+ ===== docker-distribution`
		`+`
		`+ The https://github.com/docker/distribution/[docker-distribution]`
		`+ registry is considered the Docker upstream "v2 registry" such that it`
		`+ was used by upstream to implement the new version 2 specification of the`
		`+ docker-registry.`
		`+`
		`+ ===== Fedora Production Registry`
		`+`
		`+ Implementation details of this are still unknown at the time of this`
		`+ writing and will be updated at a later date. For the current status and`
		`+ implementation notes please visit the`
		`+ https://fedoraproject.org/wiki/Changes/FedoraDockerRegistry[FedoraDockerRegistry]`
		`+ page.`
		`+`
		`+ ==== Taskotron`
		`+`
		`+ https://taskotron.fedoraproject.org/[Taskotron] is an automated task`
		`+ execution framework, written on top of http://buildbot.net/[buildbot]`
		`+ that currently executes many Fedora automated QA tasks and we will be`
		`+ adding the Layered Image automated QA tasks. The tests themselves will`
		`+ be held in DistGit and maintained by the Layered Image maintainers.`
		`+`
		`+ ==== RelEng Automation`
		`+`
		`+ https://pagure.io/releng-automation[RelEng Automation] is an ongoing`
		`+ effort to automate as much of the RelEng process as possible by using`
		`+ http://ansible.com/[Ansible] and being driven by`
		`+ http://www.fedmsg.com/en/latest/[fedmsg] via`
		`+ https://github.com/maxamillion/loopabull[Loopabull] to execute Ansible`
		`+ Playbooks based on fedmsg events.`
		`+`
		`+ ==== Robosignatory`
		`+`
		`+ https://pagure.io/robosignatory[Robosignatory] is a fedmsg consumer that`
		`+ automatically signs artifacts and will be used to automatically sign`
		`+ docker layered images for verification by client tools as well as end`
		`+ users.`
		`+`
		`+ ==== Future Integrations`
		`+`
		`+ In the future various other components of the`
		`+ https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]`
		`+ will likely be incorporated.`
		`+`
		`+ ===== PDC`
		`+`
		`+ https://pdc.fedoraproject.org/[PDC] is Fedora's implementation of`
		`+ https://github.com/product-definition-center/product-definition-center[Product`
		`+ Definition Center] which allows Fedora to maintain a database of each`
		`+ Compose and all of it's contents in a way that can be queried and used`
		`+ to make decisions in a programatic way.`
		`+`
		`+ ===== The New Hotness`
		`+`
		`+ https://github.com/fedora-infra/the-new-hotness[The New Hotness] is a`
		`+ http://www.fedmsg.com/en/latest/[fedmsg] consumer that listens to`
		`+ release-monitoring.org and files bugzilla bugs in response (to notify`
		`+ packagers that they can update their packages).`

modules/release_guide/pages/mass_rebuild.adoc

file added

+26

		`@@ -0,0 +1,26 @@`
		`+ = Mass rebuild information.`
		`+`
		`+ == Description`
		`+`
		`+ Periodically we do mass rebuilds of rawhide during the development cycle. This SOP will outline the steps necessary to do this.`
		`+`
		`+ == Assumptions`
		`+`
		`+ This assumes that the mass rebuild has already been approved and scheduled via release engineering and FESCo. Coordinate with infrastructure as well for any needed koji updates.`
		`+`
		`+ This also assumes that the mass rebuild does not need to be done in dependency order, and that the mass rebuild does not involve a ABI change.`
		`+`
		`+ == Considerations`
		`+`
		`+ The most important thing to keep in mind while doing a mass rebuild is to communicate clearly what actions are being performed and the status of the rebuild.`
		`+`
		`+ Check in on scripts frequently to avoid a long stalled command from adding significant delays in completing the rebuild.`
		`+`
		`+ Check with secondary arches, whether they up-to-date enough with primary, create rebuild tag and target when they are. It will then take care of rebuilds of the arch specific packages in appropriate kojis.`
		`+`
		`+ == Actions`
		`+`
		`+ === Preparatory Steps`
		`+`
		`+ The following steps may be completed in the weeks leading up to the scheduled mass rebuild.`
		`+`

modules/release_guide/pages/mass_rebuild_modules.adoc

file added

+122

		`@@ -0,0 +1,122 @@`
		`+ == Mass Rebuild of Modules`
		`+`
		`+ === Description`
		`+`
		`+ Periodically we do mass rebuilds of modules in rawhide during the`
		`+ development cycle. This SOP will outline the steps necessary to do this.`
		`+`
		`+ === Assumptions`
		`+`
		`+ This assumes that the mass rebuild has already been approved and`
		`+ scheduled via release engineering and FESCo. Coordinate with`
		`+ infrastructure as well for any needed updates.`
		`+`
		`+ === Considerations`
		`+`
		`+ * The most important thing to keep in mind while doing a mass rebuild is`
		`+ to communicate clearly what actions are being performed and the status`
		`+ of the rebuild.`
		`+ * Check in on scripts frequently to avoid a long stalled command from`
		`+ adding significant delays in completing the rebuild.`
		`+`
		`+ === Actions`
		`+`
		`+ ==== Preparatory Steps`
		`+`
		`+ The following steps should be completed after the`
		`+ https://docs.pagure.org/releng/sop_mass_rebuild_packages.html[mass`
		`+ rebuild of packages] is done.`
		`+`
		`+ . Update Scripts`
		`+`
		`+ The mass rebuild depends on two main scripts from the`
		`+ https://pagure.io/releng[releng git repository]. Each one requires some`
		`+ changes in variables for each new mass rebuild cycle.`
		`+`
		`+ ____`
		`+ * {blank}`
		`+ +`
		`+ _mass-rebuild-modules.py_::`
		`+ ** rebuildid`
		`+ * {blank}`
		`+ +`
		`+ _massrebuildsinfo.py_::`
		`+ ** module_mass_rebuild_epoch`
		`+ ** module_mass_rebuild_platform`
		`+ ____`
		`+`
		`+ Change the following items:`
		`+`
		+ * the `rebuildid` to match the release for which you are mass rebuilding
		`+ modules as per in massrebuildsinfo.py`
		+ * `module_mass_rebuild_epoch` mostly will be the epoch of mass rebuild
		`+ of packages`
		+ * `module_mass_rebuild_platform` should be the rawhide module platform
		`+`
		`+ ==== Starting the Mass Rebuild of Modules`
		`+`
		+ The `mass-rebuild-modules.py` script takes care of:
		`+`
		`+ * Discovering available available modules from PDC`
		`+ * Find the module info from mbs and check if a module build is submitted`
		`+ after the epoch date`
		`+ * Checking out modules from dist-git`
		`+ * Switching to appropriate stream`
		`+ * Find modulemd file`
		`+ * Use libmodulemd to determine if this module stream applies to this`
		`+ platform version`
		`+ * If needs rebuilding, committing the change`
		`+ * Push the commit`
		`+ * Submitting the build request through mbs`
		`+`
		`+ . Connect to the mass-rebuild Machine`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh compose-branched01.iad2.fedoraproject.org`
		`+ ....`
		`+ ____`
		`+ . Start a terminal multiplexer`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ tmux`
		`+ ....`
		`+ ____`
		`+ . Clone or checkout the latest copy of the`
		`+ https://pagure.io/releng[releng git repository].`
		`+ . Run the [.title-ref]#mass-rebuild-modules.py# script from`
		`+ _releng/scripts_`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd path/to/releng_repo/scripts`
		`+ $ ./mass-rebuild-modules.py <path_to_token_file> build --wait 2>&1 \| tee ~/massbuildmodules.out`
		`+ ....`
		`+ ____`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The token file should be located in infra's private ansible repo, or ask`
		`+ infra to get it to you using this`
		`+ https://pagure.io/fedora-infrastructure/issue/8048#comment-587789[process].`
		`+ ====`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The [.title-ref]#build# option is really important to pay attention,`
		`+ since the mass branching of modules will also use the same script, just`
		`+ changing the option to [.title-ref]#branch# and`
		`+ [.title-ref]#module_mass_branching_platform# in`
		`+ [.title-ref]#massrebuildsinfo.py#`
		`+ ====`
		`+`
		`+ ==== Post Mass Rebuild Tasks`
		`+`
		`+ Once the module mass rebuild is done, send an email to the`
		`+ devel-announce@ list`
		`+`
		`+ . Send the final notification to the`
		`+ _devel-announce@lists.fedoraproject.org_ list`

modules/release_guide/pages/mass_rebuild_packages.adoc

file added

+262

		`@@ -0,0 +1,262 @@`
		`+ == Mass Rebuild`
		`+`
		`+ === Description`
		`+`
		`+ Periodically we do mass rebuilds of rawhide during the development`
		`+ cycle. This SOP will outline the steps necessary to do this.`
		`+`
		`+ === Assumptions`
		`+`
		`+ This assumes that the mass rebuild has already been approved and`
		`+ scheduled via release engineering and FESCo. Coordinate with`
		`+ infrastructure as well for any needed koji updates.`
		`+`
		`+ This also assumes that the mass rebuild does not need to be done in`
		`+ dependency order, and that the mass rebuild does not involve a ABI`
		`+ change.`
		`+`
		`+ === Considerations`
		`+`
		`+ * The most important thing to keep in mind while doing a mass rebuild is`
		`+ to communicate clearly what actions are being performed and the status`
		`+ of the rebuild.`
		`+ * Check in on scripts frequently to avoid a long stalled command from`
		`+ adding significant delays in completing the rebuild.`
		`+ * Check with secondary arches, whether they up-to-date enough with`
		`+ primary, create rebuild tag and target when they are. It will then take`
		`+ care of rebuilds of the arch specific packages in appropriate kojis.`
		`+`
		`+ === Actions`
		`+`
		`+ ==== Preparatory Steps`
		`+`
		`+ The following steps may be completed in the weeks leading up to the`
		`+ scheduled mass rebuild.`
		`+`
		`+ . Create the Mass Rebuild Pagure Issue`
		`+ +`
		`+ ____`
		`+ Create an issue on the https://pagure.io/releng/issues[Release`
		`+ Engineering issues page] that points at the schedule for the current`
		`+ release.`
		`+`
		`+ See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue`
		`+ example].`
		`+ ____`
		`+ . Set up the Mass Rebuild Wiki Page`
		`+ +`
		`+ ____`
		`+ The mass rebuild wiki page should answer the following questions for`
		`+ maintainers:`
		`+`
		`+ * Why the mass rebuild is happening`
		`+ * How to opt out of the mass rebuild`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26`
		`+ Wiki example].`
		`+ ====`
		`+ ____`
		`+ . Send out the Mass Rebuild Notice`
		`+ +`
		`+ ____`
		`+ Send out the same information posted on the wiki to the`
		`+ [.title-ref]#devel-announce@lists.fedoraproject.org# mailing list.`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ See`
		`+ https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the`
		`+ Fedora 26 e-mail example].`
		`+ ====`
		`+ ____`
		`+ . Create a Tag to Contain the Mass Rebuild`
		`+ +`
		`+ ____`
		`+ Mass rebuilds require their own tag to contain all related builds. The`
		`+ example assumes we are doing a rebuild for Fedora 26.`
		`+`
		`+ ....`
		`+ $ koji add-tag f26-rebuild --parent f26`
		`+ ....`
		`+ ____`
		`+ . Request Package Auto-Signing for New Mass-Rebuild Tag`
		`+ +`
		`+ ____`
		`+ File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora`
		`+ Infrastructure] requesting the new mass-rebuild tag be enabled for`
		`+ package auto-signing.`
		`+ ____`
		`+ . Create the Koji Target for the Mass Rebuild`
		`+ +`
		`+ ____`
		`+ Using the same [.title-ref]#f26-rebuild# tag created in the previous`
		`+ example:`
		`+`
		`+ ....`
		`+ $ koji add-target f26-rebuild f26-build`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ koji add-target _target-name_ _buildroot-tag_ _destination-tag_`
		`+ describes the syntax format above. If the _destination-tag_ is not`
		`+ specified then it will be the same as the _target-name_.`
		`+ ====`
		`+ ____`
		`+ . Update Scripts`
		`+ +`
		`+ ____`
		`+ The mass rebuild depends on four main scripts from the`
		`+ https://pagure.io/releng[releng git repository]. Each one requires some`
		`+ changes in variables for each new mass rebuild cycle.`
		`+`
		`+ * {blank}`
		`+ +`
		`+ mass-rebuild.py::`
		`+ ** buildtag`
		`+ ** targets`
		`+ ** epoch`
		`+ ** comment`
		`+ ** target`
		`+ * {blank}`
		`+ +`
		`+ find-failures.py::`
		`+ ** buildtag`
		`+ ** desttag`
		`+ ** epoch`
		`+ * mass-tag.py`
		`+ * {blank}`
		`+ +`
		`+ need-rebuild.py::`
		`+ ** buildtag`
		`+ ** target`
		`+ ** updates`
		`+ ** epoch`
		`+ ____`
		`+`
		`+ Change the following items:`
		`+`
		`+ * the build tag, holding tag, and target tag should be updated to`
		`+ reflect the Fedora release you're building for`
		+ * the `epoch` should be updated to the point at which all features that
		`+ the mass rebuild is for have landed in the build system (and a newRepo`
		`+ task completed with those features)`
		`+ * the comment which is inserted into spec changelogs`
		`+`
		`+ ==== Starting the Mass Rebuild`
		`+`
		+ The `mass-rebuild.py` script takes care of:
		`+`
		`+ * Discovering available packages in koji`
		`+ * Trimming out packages which have already been rebuilt`
		`+ * Checking out packages from git`
		`+ * Bumping the spec file`
		`+ * Committing the change`
		`+ * git tagging the change`
		`+ * Submitting the build request to Koji`
		`+`
		`+ . Connect to the mass-rebuild Machine`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh branched-composer.phx2.fedoraproject.org`
		`+ ....`
		`+ ____`
		`+ . Start a terminal multiplexer`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ tmux`
		`+ ....`
		`+ ____`
		`+ . Clone or checkout the latest copy of the`
		`+ https://pagure.io/releng[releng git repository].`
		`+ . Run the mass-rebuild.py script from _releng/scripts_`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd path/to/releng_repo/scripts`
		`+ $ ./mass-rebuild.py 2>&1 \| tee ~/massbuild.out`
		`+ ....`
		`+ ____`
		`+`
		`+ ==== Monitoring Mass Rebuilds`
		`+`
		`+ The community has a very high interest in the status of rebuilds and`
		`+ many maintainers will want to know if their build failed right away. The`
		+ `find-failures.py` and `need-rebuild.py` scripts are designed to update
		`+ publicly available URLs for stakeholders to monitor.`
		`+`
		`+ . Connect to a Compose Machine`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh compose-x86-02.phx2.fedoraproject.org`
		`+ ....`
		`+ ____`
		`+ . Start a terminal multiplexer`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ tmux`
		`+ ....`
		`+ ____`
		`+ . Clone or checkout the latest copy of the`
		`+ https://pagure.io/releng[releng git repository]`
		`+ . {blank}`
		`+ +`
		`+ Set Up the Rebuild Failures Notification Web Site::`
		+ The `find_failures.py` script discovers attempted builds that have
		`+ failed. It lists those failed builds and sorts them by package owner.`
		`+ +`
		`+ ....`
		`+ $ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done`
		`+ ....`
		`+ . Start a second pane in the terminal emulator`
		`+ . {blank}`
		`+ +`
		`+ Set up the Site for Packages that Need Rebuilt::`
		+ The `need-rebuild.py` script discovers packages that have not yet been
		`+ rebuilt and generates an html file listing them sorted by package`
		`+ owner. This gives external stakeholders a rough idea of how much work`
		`+ is remaining in the mass rebuild.`
		`+ +`
		`+ ....`
		`+ $ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done`
		`+ ....`
		`+`
		`+ ==== Post Mass Rebuild Tasks`
		`+`
		`+ Once the mass rebuild script completes, and all the pending builds have`
		+ finished, the builds will need to be tagged. The `mass-tag.py` script
		`+ will accomplish this task. The script will:`
		`+`
		`+ * Discover completed builds`
		`+ * Trim out builds that are older than the latest build for a given`
		`+ package`
		`+ * Tag remaining builds into their final destination (without generating`
		`+ email)`
		`+`
		`+ . Clone or checkout the latest copy of the`
		`+ https://pagure.io/releng[releng git repository]`
		+ . Run the `mass-tag.py` script (requires koji kerberos authentication)
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd path/to/releng_repo/scripts`
		`+ $ ./mass-tag.py --source f36-rebuild --target f36`
		`+ ....`
		`+ ____`
		`+ . Send the final notification to the`
		`+ _devel-announce@lists.fedoraproject.org_ list`
		`+ +`
		`+ ____`
		`+ The contents should look something like this`
		`+ https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example`
		`+ email].`
		`+ ____`

modules/release_guide/pages/overview.adoc

file added

+141

		`@@ -0,0 +1,141 @@`
		`+ [[overview]]`
		`+ == Fedora Release Engineering Overview`
		`+`
		`+ [[overview-intro]]`
		`+ === Introduction`
		`+`
		`+ The development of Fedora is a very open process, involving over a`
		`+ thousand package maintainers (along with testers, translators,`
		`+ documentation writers and so forth). These maintainers are responsible`
		`+ for the bulk of Fedora distribution development. An elected`
		`+ https://fedoraproject.org/wiki/Fedora_Engineering_Steering_Committee[committee`
		`+ of people] provides some level of direction over the engineering aspects`
		`+ of the project.`
		`+`
		`+ The rapid pace of Fedora development leaves little time for polishing`
		`+ the development system into a quality release. To solve this dilemma,`
		`+ the Fedora project makes use of regular freezes and milestone (Alpha,`
		`+ Beta, Final) releases of the distribution, as well as "branching" of our`
		`+ trees to maintain different strains of development.`
		`+`
		`+ Stable branches of the Fedora tree and associated`
		`+ https://fedoraproject.org/wiki/Repositories[Repositories] are maintained`
		`+ for each Fedora release. The`
		`+ https://fedoraproject.org/wiki/Releases/Rawhide[Rawhide] rolling`
		`+ development tree is the initial entry point for all Fedora development,`
		`+ and the trunk from which all branches diverge. Releases are`
		`+ https://fedoraproject.org/wiki/Releases/Branched[Branched] from Rawhide`
		`+ some time before they are sent out as stable releases, and the milestone`
		`+ releases (Alpha, Beta and Final) are all built from this Branched tree.`
		`+`
		`+ Nightly snapshot images of various kinds are built from Rawhide and`
		`+ Branched (when it exists) and made available for download from within`
		`+ the trees on the https://mirrors.fedoraproject.org/[mirrors] or from the`
		`+ https://fedoraproject.org/wiki/Koji[Koji] build system.`
		`+`
		`+ The https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora`
		`+ Release Life Cycle] page is a good entry point for more details on these`
		`+ processes. Some other useful references regarding the Fedora release`
		`+ process include:`
		`+`
		`+ * The https://fedoraproject.org/wiki/Changes/Policy[Release planning`
		`+ process]`
		`+ * The`
		`+ https://fedoraproject.org/wiki/QA:Release_validation_test_plan[release`
		`+ validation test plan]`
		`+ * The https://fedoraproject.org/wiki/QA:Updates_Testing[updates-testing`
		`+ process], via https://fedoraproject.org/wiki/Bodhi[Bodhi] and the`
		`+ https://fedoraproject.org/wiki/Updates_Policy[Updates Policy]`
		`+ * The https://fedoraproject.org/wiki/QA:SOP_compose_request[test compose`
		`+ and release candidate system]`
		`+ * The https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[blocker`
		`+ bug process] and`
		`+ https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process[freeze`
		`+ exception bug process]`
		`+ * The [.title-ref]#Repositories#`
		`+ * The https://fedoraproject.org/wiki/Bugs_and_feature_requests[Bugzilla`
		`+ system]`
		`+`
		`+ === Final Release Checklist`
		`+`
		`+ Various tasks need to be accomplished prior to a final Fedora release.`
		`+ Release Engineering is responsible for many of them, as outlined here.`
		`+`
		`+ ==== Release Announcement`
		`+`
		`+ The https://fedoraproject.org/wiki/Docs_Project[Fedora Documentation`
		`+ Project] prepares release announcements for the final releases. A`
		+ https://bugzilla.redhat.com/bugzilla/enter_bug.cgi?product=Fedora%20Documentation&op_sys=Linux&target_milestone=---&bug_status=NEW&version=devel&component=release-notes&rep_platform=All&priority=normal&bug_severity=normal&assigned_to=relnotes%40fedoraproject.org&cc=&estimated_time_presets=0.0&estimated_time=0.0&bug_file_loc=http%3A%2F%2F&short_desc=RELNOTES%20-%20Summarize%20the%20release%20note%20suggestion%2Fcontent&comment=Provide%20details%20here.%20%20Do%20not%20change%20the%20blocking%20bug.&status_whiteboard=&keywords=&issuetrackers=&dependson=&blocked=151189&ext_bz_id=0&ext_bz_bug_id=&data=&description=&contenttypemethod=list&contenttypeselection=text%2Fplain&contenttypeentry=&maketemplate=Remember%20values%20as%20bookmarkable%20template&form_name=enter_bug[bug
		`+ needs to be filed] for this two weeks before the final release date.`
		`+`
		`+ ==== Mirror List Files`
		`+`
		`+ A new set of mirror list files need to be created for the new release.`
		`+ Email mailto:mirror-admin@fedoraproject.org[Fedora Mirror Admins] to`
		`+ have these files created. These should be created at the Final Freeze`
		`+ point but may redirect to Rawhide until final bits have been staged.`
		`+`
		`+ === Release Composing`
		`+`
		`+ Fedora “releases” can be built by anyone with a fast machine of proper`
		`+ arch and access to a package repository. All of the tools necessary to`
		`+ build a release are available from the package repository. These tools`
		`+ aim to provide a consistent way to build Fedora releases. A complete`
		`+ release can actually be built with only a couple commands, including the`
		`+ creation of network install images, offline install images ('DVDs'),`
		`+ live images, disk images, install repositories, [[FedUp]] upgrade`
		`+ images, and other bits. These commands are named pungi and`
		`+ livecd-creator.`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ There is work currently being done to replace livecd-creator with`
		`+ https://github.com/rhinstaller/lorax/blob/master/src/sbin/livemedia-creator[livemedia-creator].`
		`+ ====`
		`+`
		`+ ==== Pungi`
		`+`
		`+ https://pagure.io/pungi[Pungi] is the tool used to compose Fedora`
		`+ releases. It requires being ran in a chroot of the package set that it`
		`+ is composing. This ensures that the correct userland tools are used to`
		`+ match up with the kernel that will be used to perform the installation.`
		`+ It uses a comps file + yum repos to gather packages for the compose. The`
		`+ https://fedoraproject.org/wiki/Koji[Koji] build system provides a way to`
		`+ run tasks within chroots on the various arches, as well as the ability`
		`+ to produce yum repos of packages from specific collections.`
		`+`
		`+ ==== Livecd-creator`
		`+`
		`+ Livecd-creator is part of the`
		`+ https://fedoraproject.org/wiki/FedoraLiveCD[livecd-tools] package in`
		`+ Fedora and it is used to compose the live images as part of the Fedora`
		`+ release. It is also used to compose many of the custom`
		`+ https://spins.fedoraproject.org[Spins] or variants of Fedora.`
		`+`
		`+ === Distribution`
		`+`
		`+ Once a compose has been completed, the composed tree of release media,`
		`+ installation trees, and frozen`
		`+ https://fedoraproject.org/wiki/Repositories[Repositories] needs to be`
		`+ synchronized with the Fedora mirror system. [[MirrorManager]] has some`
		`+ more details on the mirror system. Many of the images are also offered`
		`+ via BitTorrent as an alternative method of downloading.`
		`+`
		`+ ==== Download Mirrors`
		`+`
		`+ Depends on the Fedora Mirror System and infrastructure to populate them`
		`+ privately.`
		`+`
		`+ ==== BitTorrent`
		`+`
		`+ BitTorrent is currently served by http://torrent.fedoraproject.org.`
		`+ Images are added to the system via this`
		`+ https://infrastructure.fedoraproject.org/infra/docs/docs/sysadmin-guide/sops/torrentrelease.rst[Standard`
		`+ Operating Procedure].`
		`+`
		`+ === Acknowledgements`
		`+`
		`+ This document was influenced by`
		`+ http://www.freebsd.org/doc/en_US.ISO8859-1/articles/releng/article.html[release`
		`+ engineering documents] from http://freebsd.org[FreeBSD].`

modules/release_guide/pages/philosophy.adoc

file added

+130

		`@@ -0,0 +1,130 @@`
		`+ [[philosophy]]`
		`+ == Fedora Release Engineering Philosophy`
		`+`
		`+ Being an official part of Fedora means that it is composed and supported`
		`+ by Fedora Release Engineering (releng). Releng bestows this status on`
		`+ items that successfully makes its way through our Manufacturing Train`
		`+ and satisfies all related policies. With this status the item is Open,`
		`+ Integrated, Reproducible, Auditable, Definable, and Deliverable. (in no`
		`+ particular order) This document provides definitions for each of those`
		`+ terms, why they're important, and how we guarantee them. All parts of`
		`+ Fedora should strive to be meet all parts of being official at all`
		`+ times.`
		`+`
		`+ === Open`
		`+`
		`+ It goes without saying that Fedora is built on`
		`+ https://fedoraproject.org/wiki/Foundations[the four F's]. In releng we`
		`+ are no different, we require everything be open, that is open source,`
		`+ developed in the open, available for all to look at, use and contribute`
		`+ to. All downstreams should be able to take our tooling and make their`
		`+ own derivative of Fedora, either by rebuilding everything, perhaps with`
		`+ different otions or just adding thier own marks and packages and`
		`+ recomposing. At any time anyone should be able to see how Fedora is put`
		`+ together and put together their own version of Fedora.`
		`+`
		`+ === Integrated`
		`+`
		`+ Fedora is a huge project with a massive number of ever growing`
		`+ deliverables. This means when we add new deliverables we need to have`
		`+ the composing of them tightly integrated into the process. We ship`
		`+ Fedora a whole unit, so we need to make it as a whole unit. Any new`
		`+ tooling we use needs to be consistent with the existing tooling in how`
		`+ it works. The tooling has to ensure that the output is Reproducible,`
		`+ Auditable, Definable so that it can be Deliverable.`
		`+`
		`+ === Reproducible`
		`+`
		`+ A reproducible component is one we can rebuild from scratch at any time`
		`+ with less than a day's worth of effort. It implies we can look up all of`
		`+ the source code, and know exactly what revisions to use from source`
		`+ control. We know exactly what tools are used in the build environment to`
		`+ transform the source code into product content (binaries). We also know`
		`+ how to reproduce the same build environment to ensure the tools behave`
		`+ like we expect. This aspect is why releng is in the business of`
		`+ standardizing on build tools.`
		`+`
		`+ Reproducible components are important because they make them a lot`
		`+ easier to maintain. The Security Team takes advantage of this aspect of`
		`+ an Product. Not knowing how to rebuild a subsystem in a product to apply`
		`+ security fixes, or bug fixes, makes their job much more difficult. It`
		`+ would be a significant risk to provide a product to users that we are`
		`+ incapable (or merely unsure) of how to build again. Not knowing the`
		`+ origin of source code is also a significant risk, which is why many of`
		`+ our build environments are configured to prevent tools from dynamically`
		`+ downloading content from the internet.`
		`+`
		`+ The combination of Koji and fedpkg is what enables releng to rebuild a`
		`+ component. fedpkg manages the source code in our dist-git system, and`
		`+ Koji archives details about the build environment, the tools used, logs,`
		`+ and of course the binaries themselves. The reproducibility aspect of a`
		`+ product is the primary reason we require all products be built in Koji`
		`+ if they seek to be an officially supported part of Fedora.`
		`+`
		`+ === Auditable`
		`+`
		`+ Fedora and Red Hat expect auditable output too, which means releng knows`
		`+ who built what, when, and where (and how, but that's reproducibility).`
		`+ Being able to authoritatively say that something was built within Fedora`
		`+ by people who have signed the FPCA is important for several reasons. One`
		`+ big reason is it promotes and fosters accountability within the`
		`+ comunity. It promotes ownership. Another one is more defensive: in the`
		`+ event of a security breach, we have a lot of evidence and data prepared`
		`+ to help us identify what content (if any) was compromised. If a kernel`
		`+ RPM randomly shows up, and we have no records of building it and/or`
		`+ shipping it, that should raise a lot of alarms pretty quickly!`
		`+`
		`+ Red Hat's Infosec team and Fedora Security care about this aspect`
		`+ deeply. We should never be in a position where we cannot definitively`
		`+ answer why a piece of content is available to users. This aspect is also`
		`+ a key part of the verification that is done when Fedora becomes RHEL.`
		`+ All downstream consumers of Fedora expect that they can verify the code`
		`+ and binaries that they consume from Fedora.`
		`+`
		`+ Releng tracks this data in 2 systems, 1 of which we own: Koji and Bodhi.`
		`+ Koji uses ssl certs tied to FAS and bodhi uses FAS for authentication to`
		`+ provide a strong relationship between a user and the content. Koji`
		`+ builds the content of course, the Bodhi tracks the bugs, documentation,`
		`+ and enhancements associated with the content and actually does the`
		`+ delivery. Bodhi maintains records of what was shipped when and where,`
		`+ and who pushed it.`
		`+`
		`+ === Definable`
		`+`
		`+ The ability to define and predict content is necessary as well. It is`
		`+ important to know exactly what was included in a release. It helps`
		`+ protect against shipping content that unnecessarily causes a support`
		`+ burden. This aspect of a Fedora component helps support other aspects`
		`+ like reproducibility. No need to reproduce software we do not have to`
		`+ ship, right? Ensuring the product content is lean and trim may sound`
		`+ obvious, but in the world of sprawling RPM dependencies, Maven`
		`+ artifacts, and Ruby gems, it is actually rather easy to include content`
		`+ during the course of a multi-month or multi-year development cycle.`
		`+`
		`+ Furthermore, a definable component has the changes made to it vetted and`
		`+ understood by multiple teams. They are not made in an ad-hoc manor or`
		`+ without consent from FESCo, QA, releng, and the Product Working Groups`
		`+ that contribute at the program level. This reduces change risk to the`
		`+ user, which our users and downstreams like to hear.`
		`+`
		`+ Many systems help make components definable. Releng uses Bugzilla, trac,`
		`+ blocker bugs and bodhi to track additions and changes.`
		`+`
		`+ === Deliverable`
		`+`
		+ Official parts of Fedora are eligible to be delivered to `/pub/fedora/`
		+ or `/pub/alt/releases` on https://dl.fedoraproject.org/pub/[Fedora
		`+ Download] and to get`
		`+ https://admin.fedoraproject.org/mirrormanager[mirrorlists] in`
		`+ https://github.com/fedora-infra/mirrormanager2[mirrormanager]. These`
		`+ Distribution Platforms are maintained by Fedora Infrastructure and`
		`+ releng. This is not a feature of the product content itself or how it`
		`+ was built, but rather how it was delivered to users through releng's`
		`+ processes. Those platforms are geographically replicated by the`
		`+ volunteer mirror network. They provide a reliable and durable service`
		`+ that ensures users can always reach Fedora for updates, even in the`
		`+ event of a disaster affecting our data center in phx2.`
		`+`
		`+ User support (user mailing lists and [.title-ref]##fedora#) and Fedora`
		`+ Security team depend on these services. It is vital to that critical`
		`+ security fixes and updates are always available to users.`

modules/release_guide/pages/release_process.adoc

file added

+39

		`@@ -0,0 +1,39 @@`
		`+ include::_partials/attributes.adoc[]`
		`+ = Release process.`
		`+`
		`+ == Rawhide starts Fedora Linux {branched} development`
		`+ == Mass Rebuild: RPMs first, then modules`
		`+ == Branch Fedora Linux {branched} from Rawhide`
		`+ == Rawhide starts Fedora Linux {rawhide} development`
		`+ == Update Greenwave policy product_versions to add branched release`
		`+ == Post-branch Freeze (end date approximate)`
		`+ == Bodhi updates-testing activation point`
		`+ == Beta Freeze (starts at 1400 UTC)`
		`+ == Beta Release Public Availability`
		`+ == Final Freeze (starts at 1400 UTC)`
		`+ == Final Release Public Availability (GA)`
		`+ == Fedora Linux {old_release} EOL auto closure`
		`+ == Beta release Go/No-Go Meeting`
		`+ == Final release Go/No-Go Meeting`
		`+ == Update Greenwave policy product_versions to remove EOL release`
		`+ == Notifications for retirement of packages with security issues`
		`+ == Retire packages with security issues`
		`+ == fedora-repos update with the new keys is available in updates-testing`
		`+ == Begin reminders for packages that haven't rebuilt in last 2 releases`
		`+ == Re-sign F{branched} content with F{rawhide} key`
		`+ == Retire Orphaned and Long-Time FTBFS Rawhide Packages`
		`+ == Retire FTI packages in NEW state`
		`+ == Stage Beta on master minor`
		`+ == Bit-flip Beta to mirrors`
		`+ == Notify Mirrors of Beta Release`
		`+ == Set current release to active in Package database`
		`+ == Retire all packages with broken deps`
		`+ == Retire FTI packages in NEW state`
		`+ == Enable Updates in Bodhi for the current release`
		`+ == Stage Final on master minor`
		`+ == Bit-flip Final to mirrors`
		`+ == Update PDC with F{old_release} EOL date`
		`+ == REMINDER: EOL release final update push in 1 week`
		`+ == REMINDER: EOL release final update push in 1 day`
		`+ == Notify Mirrors of Final Release`
		`+ == Remove Beta images`

modules/release_guide/pages/sop.adoc

file added

+48

		`@@ -0,0 +1,48 @@`
		`+ [[sop]]`
		`+ == Fedora Release Engineering Standard Operating Procedures`
		`+`
		`+ This page documents the Standard Operating Procedures for Fedora Release`
		`+ Engineering.`
		`+`
		`+ === Purpose`
		`+`
		`+ The SOP section contains a list of tasks Fedora Release Engineering team`
		`+ provides for the project. Current Fedora Release Engineering team`
		`+ members would can add tasks they know about and list steps to complete`
		`+ the task as well as issues to consider. This is a great way to make sure`
		`+ that individuals aren't the only ones that can fix a problem.`
		`+`
		`+ === The Community`
		`+`
		`+ The SOP section is left in the public because it is our hope that others`
		`+ in the community will add common problems, fix our steps and just in`
		`+ general look over what we're doing and help us when we're doing`
		`+ something stupid. We encourage anyone interested to go over our`
		`+ processes with a fine tooth comb. It'll make us better and we'll`
		`+ probably learn something in the process. Therefore please open a`
		`+ https://pagure.io/releng/pull-requests[pull request] to suggest`
		`+ improvements.`
		`+`
		`+ === Procedures`
		`+`
		`+ Needed: #. Composing an official release #. Making a composed release`
		`+ publicly available`
		`+`
		`+ === Standard Operating Procedures`
		`+`
		`+ sop_adding_build_targets sop_adding_content_generator`
		`+ sop_adding_new_release_engineer sop_adding_new_branch_sla`
		`+ sop_adding_packages_compose_artifact sop_adding_side_build_targets`
		`+ sop_branch_freeze sop_branching sop_breaking_development_freeze`
		`+ sop_composing_fedora sop_clean_amis sop_create_release_signing_key`
		`+ sop_deprecate_ftbfs_packages sop_end_of_life sop_eol_change`
		`+ sop_mass_branching sop_bodhi_activation sop_mass_rebuild_packages`
		`+ sop_mass_rebuild_modules sop_file_ftbfs sop_generating_openh264_composes`
		`+ sop_package_blocking sop_package_unblocking`
		`+ sop_process_dist_git_requests sop_promoting_container_content`
		`+ sop_signing_builds sop_pushing_updates sop_release_package_signing`
		`+ sop_remote_dist_git_branches sop_requesting_task_automation_users`
		`+ sop_retire_orphaned_packages sop_sigul_client_setup`
		`+ sop_stage_final_release_for_mirrors sop_unretire sop_update_critpath`
		`+ sop_update_releng_docs sop_updating_comps sop_fedora_media_writer`
		`+ sop_find_module_info sop_release_container_base_image`

modules/release_guide/pages/sop_adding_build_targets.adoc

file added

+180

		`@@ -0,0 +1,180 @@`
		`+ == Adding Build Targets`
		`+`
		`+ === Description`
		`+`
		`+ Each new release we create a build target for the next release. This SOP`
		`+ will describe the steps necessary to prepare the new build target.`
		`+`
		`+ === Action`
		`+`
		`+ Adding a build target is a complicated task. It involves updating koji,`
		`+ git, and fedora-release package.`
		`+`
		`+ [[adding_build_targets_koji]]`
		`+ ==== Koji`
		`+`
		`+ In koji a couple collection tags need to be made, and a target created`
		`+ to tie them together. We create a base collection tag named after the`
		`+ release, and a build tag to hold a few things we use in the buildroots`
		`+ that are not part of the distribution (glibc32/glibc64). Inheritance to`
		`+ the previous release is used for ownership and package data, as well as`
		`+ buildroot content data.`
		`+`
		+ The `add-tag`, `add-tag-inheritance`, `edit-tag`, and `add-target`
		`+ commands are used.`
		`+`
		`+ ....`
		`+ $ koji add-tag --help`
		`+ Usage: koji add-tag [options] name`
		`+ (Specify the --help global option for a list of other help options)`
		`+`
		`+ Options:`
		`+ -h, --help show this help message and exit`
		`+ --parent=PARENT Specify parent`
		`+ --arches=ARCHES Specify arches`
		`+`
		`+`
		`+ $ koji add-tag-inheritance --help`
		`+ Usage: koji add-tag-inheritance [options] tag parent-tag`
		`+ (Specify the --help global option for a list of other help options)`
		`+`
		`+ Options:`
		`+ -h, --help show this help message and exit`
		`+ --priority=PRIORITY Specify priority`
		`+ --maxdepth=MAXDEPTH Specify max depth`
		`+ --intransitive Set intransitive`
		`+ --noconfig Set to packages only`
		`+ --pkg-filter=PKG_FILTER`
		`+ Specify the package filter`
		`+ --force=FORCE Force adding a parent to a tag that already has that`
		`+ parent tag`
		`+`
		`+ $ koji edit-tag --help`
		`+ Usage: koji edit-tag [options] name`
		`+ (Specify the --help global option for a list of other help options)`
		`+`
		`+ Options:`
		`+ -h, --help show this help message and exit`
		`+ --arches=ARCHES Specify arches`
		`+ --perm=PERM Specify permission requirement`
		`+ --no-perm Remove permission requirement`
		`+ --lock Lock the tag`
		`+ --unlock Unlock the tag`
		`+ --rename=RENAME Rename the tag`
		`+`
		`+ $ koji add-target --help`
		`+ Usage: koji add-target name build-tag <dest-tag>`
		`+ (Specify the --help global option for a list of other help options)`
		`+`
		`+ Options:`
		`+ -h, --help show this help message and exit`
		`+ ....`
		`+`
		`+ For example if we wanted to create the Fedora 17 tags, we would do the`
		`+ following:`
		`+`
		`+ ....`
		`+ koji add-tag --parent f16-updates f17`
		`+ koji add-tag --parent f17 f17-updates`
		`+ koji add-tag --parent f17-updates f17-candidate`
		`+ koji add-tag --parent f17-updates f17-updates-testing`
		`+ koji add-tag --parent f17-updates-testing f17-updates-testing-pending`
		`+ koji add-tag --parent f17-updates f17-updates-pending`
		`+ koji add-tag --parent f17-updates f17-override`
		`+ koji add-tag --parent f17-override --arches=i686,x86_64 f17-build`
		`+ koji add-tag-inheritance --priority 1 f17-build f16-build`
		`+ koji edit-tag --perm=fedora-override f17-override`
		`+ koji edit-tag --lock f17-updates`
		`+ koji add-target f17 f17-build`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		+ The `-pending` tags are used by
		`+ https://fedoraproject.org/wiki/Bodhi[Bodhi] and`
		`+ https://fedoraproject.org/wiki/Taskotron[Taskotron] to track and test`
		`+ proposed updates. These tags are not build targets and they don't get`
		`+ made into repos, so proper inheritance isn't vital.`
		`+ ====`
		`+`
		`+ ==== Git`
		`+`
		`+ The pkgdb_sync_git_branches.py file which is hosted in Fedora`
		`+ Infrastructure ansible`
		`+ (roles/distgit/templates/pkgdb_sync_git_branches.py) needs to be updated`
		`+ for the new target for making branches.`
		`+`
		+ Update `BRANCHES` with the new branch information. The branch name maps
		`+ to the branch that is its parent.`
		`+`
		`+ ....`
		`+ BRANCHES = {'el4': 'rawhide', 'el5': 'rawhide', 'el6': 'f12',`
		`+ 'OLPC-2': 'f7',`
		`+ 'rawhide': None,`
		`+ 'fc6': 'rawhide',`
		`+ 'f7': 'rawhide',`
		`+ 'f8': 'rawhide',`
		`+ 'f9': 'rawhide',`
		`+ 'f10': 'rawhide',`
		`+ 'f11': 'rawhide',`
		`+ 'f12': 'rawhide',`
		`+ 'f13': 'rawhide', 'f14': 'rawhide'}`
		`+ ....`
		`+`
		+ and update `GITBRANCHES` with the translation from pkgdb branch string
		`+ to git branch string:`
		`+`
		`+ ....`
		`+ GITBRANCHES = {'EL-4': 'el4', 'EL-5': 'el5', 'EL-6': 'el6', 'OLPC-2': 'olpc2',`
		`+ 'FC-6': 'fc6', 'F-7': 'f7', 'F-8': 'f8', 'F-9': 'f9', 'F-10': 'f10',`
		`+ 'F-11': 'f11', 'F-12': 'f12', 'F-13': 'f13', 'F-14': 'f14', 'devel': 'rawhide'}`
		`+ ....`
		`+`
		`+ The genacls.pkgdb file also needs to be updated for active branches to`
		+ generate ACLs for. Update the `ACTIVE` variable. genacls.pkgdb lives in
		`+ puppet (modules/gitolite/files/distgit/genacls.pkgdb). The format is`
		`+ pkgdb branch string to git branch string (until pkgdb uses git branch`
		`+ strings):`
		`+`
		`+ ....`
		`+ ACTIVE = {'OLPC-2': 'olpc2/', 'EL-4': 'el4/', 'EL-5': 'el5/',`
		`+ 'EL-6': 'el6/', 'F-11': 'f11/', 'F-12': 'f12/', 'F-13': 'f13/',`
		`+ 'F-14': 'f14/', 'devel': 'rawhide'}`
		`+ ....`
		`+`
		`+ ==== fedora-release`
		`+`
		+ Currently the fedora-release package provides the `%{?dist}` definitions
		`+ used during building of packages. When a new target is created,`
		`+ fedora-release must be built for the collection with new dist`
		`+ definitions.`
		`+`
		`+ ==== Comps`
		`+`
		`+ * In the comps module in Fedora Hosted git`
		`+ (ssh://git.fedorarhosted.org/git/comps.git), create and add a new comps`
		`+ file based on the previous release. (Just copying it should be fine.)`
		+ Add the new file to `po/POTFILES.in`.
		`+ * When rawhide is retargeted in koji to point to the new release, update`
		+ the `Makefile` to target comps-rawhide.xml at the new version.
		+ * Don't forget to `git push` your changes after committing.
		`+`
		`+ === Verification`
		`+`
		`+ Given the complexity of all the changes necessary to create a new build`
		`+ target, the best way to verify is to attempt a build. Given that`
		`+ fedora-release has to be built before anything else so that dist tags`
		`+ translate correctly it is a good test case. For example, this was used`
		`+ to test the new Fedora 15 target:`
		`+`
		`+ * Use pkgdb to request an F-15 branch of fedora-release`
		`+ * Use pkgdb2branch.py to actually make the branch`
		`+ * Update fedora-release clone`
		`+ * Adjust .spec file in rawhide for new dist defines`
		`+ * commit/build`
		`+ * Track build in koji to ensure proper tagging is used`
		`+`
		`+ What this won't test is translations of dist at tag time given that`
		+ `fedora-release` doesn't use dist in it's Release. Verifying with a
		`+ second package that uses dist is a good idea.`

modules/release_guide/pages/sop_adding_content_generator.adoc

file added

+58

		`@@ -0,0 +1,58 @@`
		`+ == Adding new koji content generator`
		`+`
		`+ === Description`
		`+`
		`+ Koji added support for content generators some time ago. Basic premise`
		`+ of content generators is that it lets us create build systems for new`
		`+ types of content without affecting or changing core Koji code and in`
		`+ some way simplify integration with rest of the release toolchain. More`
		`+ information about content generators, background, requirements and more`
		`+ can be found in Koji`
		`+ https://docs.pagure.org/koji/content_generators/[content generator`
		`+ documentation]`
		`+`
		`+ For content generator to be able to create/import builds into Koji`
		`+ following prerequisites have to be met:`
		`+`
		`+ * Koji has to recognize the content generator type`
		`+ * User doing the content generator import has to have permissions for`
		`+ this action`
		`+ * Any new content types have to be defined in Koji`
		`+`
		`+ === Questions to ask`
		`+`
		`+ There are some questions that should be answered before the content`
		`+ generator is enabled/added to Koji`
		`+`
		`+ * Where is the content generator service running, what is its support`
		`+ status etc?`
		`+ * Is new type of content being added or is the content generator`
		`+ providing different way to build content Koji already knows about?`
		`+ * What is the expected size of content that will be imported into Koji?`
		`+ * Does the content generator follow each of the requirements for writing`
		`+ it from Koji documentation referenced above?`
		`+`
		`+ === Adding a new content generator into koji`
		`+`
		`+ First we create the content generator and give a user permission to do`
		`+ imports for it:`
		`+`
		`+ ....`
		`+ koji grant-cg-access <username> <content_generator_name> --new`
		`+ ....`
		`+`
		`+ In many cases the content generator will be adding content with new`
		`+ content type. This can be achieved simply by running:`
		`+`
		`+ ....`
		`+ koji call addBType <content type>`
		`+ ....`
		`+`
		`+ ==== Explanation`
		`+`
		`+ * username - is a name of user which will be doing the imports. In most`
		`+ cases this will be a service-level account`
		`+ * content generator name - this name has to be provided by the content`
		`+ generator development team`
		`+ * --new - this switch ensures we create the content generator if it`
		`+ doesn't exist`

modules/release_guide/pages/sop_adding_new_branch_sla.adoc

file added

+28

		`@@ -0,0 +1,28 @@`
		`+ == Adding New Branch SLAs`
		`+`
		`+ === Description`
		`+`
		`+ In the ArbitraryBranching model, packagers can choose whatever SLAs they`
		`+ want for the branches of their packages, but they must choose from a`
		`+ subset of pre-defined SLAs stored in PDC, maintained by releng.`
		`+`
		`+ This SOP describes the steps necessary for a release engineer to create`
		`+ a new SLA.`
		`+`
		`+ === Action`
		`+`
		`+ Adding a new SLA is simple. It involves running a script in the releng`
		`+ repo, with an authorized token. There is a token available on`
		`+ [.title-ref]#pdc-backend01# in the [.title-ref]#/etc/pdc.d/# directory.`
		`+`
		`+ ....`
		`+ $ ./scripts/pdc/insert-sla.py`
		`+ Name of the SLA: wild_and_cavalier`
		`+ Description of the SLA: Anything goes! This branch may rebase at any time. No stability guarantees provided.`
		`+ ....`
		`+`
		`+ === Verification`
		`+`
		`+ Verifying that the SLA is present is simple: visit the`
		`+ https://pdc.fedoraproject.org/rest_api/v1/component-branch-slas/[appropriate`
		`+ PDC endpoint] and verify that your newly-added SLA is present.`

modules/release_guide/pages/sop_adding_new_release_engineer.adoc

file added

+160

		`@@ -0,0 +1,160 @@`
		`+ == Adding a New Release Engineer`
		`+`
		`+ === Description`
		`+`
		`+ People volunteer (or get assigned) to doing Fedora release engineering`
		`+ from time to time. This SOP seeks to describe the process to add a new`
		`+ release engineer so that they have the rights to accomplish their tasks,`
		`+ know where to find the tasks, and are introduced to the existing`
		`+ members. There are several groups that manage access to the respective`
		`+ systems:`
		`+`
		+ * `cvsadmin`: Admin group for pkgdb2 (allows to retire/orphan all
		`+ packages etc), allows git write access via SSH to all packages in`
		`+ dist-git`
		+ * `gitreleng`: Allows write access to release engineering git repo
		+ * `signers`: Membership is necessary to use
		`+ https://fedoraproject.org/wiki/Sigul_Client_Setup_SOP[sigul].`
		+ * `sysadmin`: Allows SSH access to bastion, the SSH gateway to the PHX2
		`+ data center. SSH access to several other internal systems is only`
		`+ possible from here.`
		+ * `sysadmin-cvs`: Allows shell access to pkgs01 (pkgs.fedoraproject.org)
		+ * `sysadmin-releng`: Allows SSH access to autosign01, koji03, koji04,
		`+ releng04, relepel01 from bastion`
		`+`
		`+ === Action`
		`+`
		`+ A new release engineer will access rights in a variety of systems, as`
		`+ well as be introduced to the releng group.`
		`+`
		`+ ==== Git`
		`+`
		`+ Fedora Release Engineering maintains a git repo of scripts. This can be`
		`+ found in https://pagure.io/pagure[Pagure] at`
		`+ ssh://git@pagure.io/releng.git. Write access to this group is controlled`
		`+ by access to the 'gitreleng' FAS group. The new member's FAS username`
		`+ will need to be added to this group.`
		`+`
		`+ https://pagure.io/releng`
		`+`
		+ `FIXME: walkthrough group addition`
		`+`
		`+ ==== FAS`
		`+`
		`+ There is a releng group in FAS that release engineers are added to in`
		`+ order to grant them various rights within the Fedora infrastructure. The`
		`+ new member's FAS username will need to be added to this group.`
		`+`
		+ `FIXME: walkthrough group addition`
		`+`
		`+ ==== Koji`
		`+`
		`+ In order to perform certain (un)tagging actions a release engineer must`
		`+ be an admin in koji. To grant a user admin rights one uses the`
		+ `grant-permission` command in koji.
		`+`
		`+ ....`
		`+ $ koji grant-permission --help`
		`+ Usage: koji grant-permission <permission> <user> [<user> ...]`
		`+ (Specify the --help global option for a list of other help options)`
		`+`
		`+ Options:`
		`+ -h, --help show this help message and exit`
		`+ ....`
		`+`
		`+ For example if we wanted to grant npetrov admin rights we would issue:`
		`+`
		`+ ....`
		`+ $ koji grant-permission admin npetrov`
		`+ ....`
		`+`
		`+ ==== Sigul`
		`+`
		`+ Sigul is our signing server system. They need to bet setup as a signer`
		`+ if they are going to be signing packages for a release.`
		`+`
		`+ For information on how to setup Sigul, please see:`
		`+ https://fedoraproject.org/wiki/Sigul_Client_Setup_SOP[sigul]`
		`+`
		`+ ==== RelEng Docs Page`
		`+`
		`+ The new release engineer should be added to the`
		`+ ref:link:index-team-composition[Release Engineering membership list]`
		`+`
		`+ ==== rel-eng email list`
		`+`
		`+ Release engineering ticket spam and discussion happens on our`
		`+ https://admin.fedoraproject.org/mailman/listinfo/rel-eng[Mailing List].`
		`+ New releng people need to subscribe.`
		`+`
		`+ ==== IRC`
		`+`
		`+ We ask that release engineers idle in [.title-ref]##fedora-releng# on`
		`+ Libera to be available for queries by other infrastructure admins.`
		`+ Idling on [.title-ref]##fedora-admin# on Libera is optional. It is noisy`
		`+ little bit but people sometimes ask releng stuff.`
		`+`
		`+ ==== New member announcement`
		`+`
		`+ When a new releng member starts, we announce it to the email list. This`
		`+ lets the other admins know to expect a new name to show up in tickets`
		`+ and on IRC.`
		`+`
		`+ === Verification`
		`+`
		`+ ==== Git`
		`+`
		`+ You can verify group membership by sshing to a system that is setup with`
		+ FAS and using `getent` to verify membership in the gitreleng group:
		`+`
		`+ ....`
		`+ $ ssh fedorapeople.org getent group gitreleng`
		`+ gitreleng:x:101647:ausil,dwa,jwboyer,kevin,notting,pbabinca,sharkcz,skvidal,spot`
		`+ ....`
		`+`
		`+ You can verify that the new user is in the above list.`
		`+`
		`+ ==== FAS`
		`+`
		`+ You can verify group membership by sshing to a system that is setup with`
		+ FAS and using `getent` to verify membership in the releng group:
		`+`
		`+ ....`
		`+ $ ssh fedorapeople.org getent group releng`
		`+ releng:x:101737:atowns,ausil,dwa,jwboyer,kevin,lmacken,notting,pbabinca,spot`
		`+ ....`
		`+`
		`+ You can verify that the new user is in the above list.`
		`+`
		`+ ==== Koji`
		`+`
		`+ To verify that the releng member is an admin koji use the`
		+ `list-permissions` koji command:
		`+`
		`+ ....`
		`+ $ koji list-permissions --user npetrov`
		`+ admin`
		`+ ....`
		`+`
		`+ This shows that npetrov is an admin.`
		`+`
		`+ ==== Sigul`
		`+`
		+ * `FIXME`
		`+`
		`+ ==== Wiki Page`
		`+`
		`+ Verification is easy. Just look at the page.`
		`+`
		`+ ==== releng mailing list`
		`+`
		`+ Verify by asking the user if they got the announcement email`
		`+`
		`+ ==== Announcement email`
		`+`
		`+ See above`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ * Make sure the releng person has a solid grasp of the tasks we do and`
		`+ where to get help if stuck`

modules/release_guide/pages/sop_adding_packages_compose_artifact.adoc

file added

+81

		`@@ -0,0 +1,81 @@`
		`+ == Adding a package to a Release Artifact`
		`+`
		`+ === Description`
		`+`
		`+ In the event that a Fedora contributor would like to have a package`
		`+ added to an Artifact of a Compose (such as an installer ISO image, a`
		`+ liveCD, Cloud Image, Vagrant, Docker, etc.) that is slated for Release,`
		`+ the following procedures must be followed due to the interdependence of`
		`+ different components within the distro layout.`
		`+`
		`+ ==== Background`
		`+`
		`+ First, some information on where this all comes from and how things fit`
		`+ together.`
		`+`
		`+ There is the concept of the "Install Tree" which is the collection of`
		`+ packages available at install time. This is a vast sub-set of the whole`
		`+ of the Fedora Package Collection and it is the pool of possible packages`
		`+ that is available to end users who choose to customize their install`
		`+ from the https://fedoraproject.org/wiki/Anaconda[Anaconda] installer. It`
		`+ is also the pool of possible packages that is available to the`
		`+ https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart files`
		`+ that are used to generate various components of the compose via`
		`+ https://pagure.io/pungi[pungi] which then produces the Release`
		`+ Artifacts.`
		`+`
		`+ The Install Tree itself is defined by the`
		`+ https://pagure.io/fedora-comps[comps] groups so in order to add a net`
		`+ new package to one of the Release Artifacts, the package must be placed`
		`+ in an appropriate https://pagure.io/fedora-comps[comps] xml file. For`
		`+ more information on what specifically defines the "appropriate`
		`+ [.title-ref]#comps_# xml file" and what kinds of approvals or review`
		`+ might be needed for adding new packages, please see`
		`+ https://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups[this`
		`+ HowTo].`
		`+`
		`+ === Action`
		`+`
		`+ We will need to edit the comps file specific to the Fedora release we`
		`+ would like to target. For example, if we were to target Fedora 25 we`
		+ would edit `comps-f25.xml.in` found within the
		`+ https://pagure.io/fedora-comps[comps] git repository and this should be`
		`+ modified based on the`
		`+ https://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups#How_to_edit_comps[How`
		`+ to edit comps] procedure.`
		`+`
		`+ If the package that was added is a part of a pre-existing comps group`
		`+ that is already used in the target Release Artifact's`
		`+ https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart file`
		`+ then we are done.`
		`+`
		`+ However, if there is a new comps group added then we need to include`
		`+ that new comps group in the respective`
		`+ https://pagure.io/fedora-kickstarts[fedora-kickstarts] kickstart file`
		`+ similar to the following.`
		`+`
		`+ ....`
		`+ %packages`
		`+ @mynewcompsgroup`
		`+ ....`
		`+`
		`+ Next we will need to tell https://pagure.io/pungi[pungi] Variants data`
		`+ about the new group and it's relationship to the corresponding`
		`+ https://sgallagh.wordpress.com/2016/03/18/sausage-factory-multiple-edition-handling-in-fedora/[Variant].`
		`+ This information is held in the https://pagure.io/pungi-fedora[Fedora`
		`+ Pungi Configs] https://pagure.io/[pagure] git forge repository. The file`
		+ needed to be edited is `variants-fedora.xml` and can be viewed from a
		`+ web browser`
		`+ https://pagure.io/pungi-fedora/blob/master/f/variants-fedora.xml[here].`
		`+`
		`+ Once this has been completed, we're all done.`
		`+`
		`+ === Verification`
		`+`
		`+ Verify that the next compose is successful and that the change made`
		`+ didn't cause any issues. This can be done from the`
		`+ https://pdc.fedoraproject.org/compose/[Fedora Product Definition Center]`
		`+ which is a central store of information about Composes and their`
		`+ resulting artifacts.`
		`+`
		`+ === Consider Before Running`

modules/release_guide/pages/sop_adding_side_build_targets.adoc

file added

+208

		`@@ -0,0 +1,208 @@`
		`+ == Adding Side Build Tags`
		`+`
		`+ === Description`
		`+`
		`+ Bigger Features can take a while to stabilize and land or need a large`
		`+ number of packages to be built against each other, this is easiest`
		`+ served by having a separate build tag for the development work. This SOP`
		`+ will describe the steps necessary to prepare the new build target.`
		`+`
		`+ === Action`
		`+`
		`+ Engineers should be aware that adding side build targets incurs extra`
		`+ newRepo tasks in the koji.`
		`+`
		`+ ==== Research Tag`
		`+`
		`+ . Verify whether a tag already exists.`
		`+ +`
		`+ The typical tag format is _PRODUCT_-_DESCRIPTOR_. The _DESCRIPTOR_`
		`+ should be something brief that clearly shows why the tag exists.`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Don't think too hard about what makes a good descriptor. The descriptor`
		`+ for the XFCE version 4.8 side-build target was _xfce48_. KDE often`
		`+ simply uses _kde_ as its descriptor. Use best judgement and if in doubt`
		+ ask in IRC on `#fedora-releng`.
		`+ ====`
		`+ +`
		`+ EPEL6`
		`+`
		`+ ....`
		`+ $ koji taginfo epel6-kde`
		`+ ....`
		`+ +`
		`+ EPEL7`
		`+`
		`+ ....`
		`+ $ koji taginfo epel7-kde`
		`+ ....`
		`+ +`
		`+ Fedora`
		`+`
		`+ ....`
		`+ $ koji taginfo f28-kde`
		`+ ....`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ If the tag already exists, continue searching for an available tag by`
		+ appending `-2` and incrementing the number by one until an available tag
		+ is found. For example, if `f28-kde` already exists then search for
		+ `f28-kde-2`, `f28-kde-3`, and so on until a suitable tag is found.
		`+ ====`
		`+ . Determine the appropriate architectures.`
		`+ +`
		`+ EPEL6`
		`+`
		`+ ....`
		`+ $ koji taginfo dist-6E-epel-build`
		`+ ....`
		`+ +`
		`+ EPEL7`
		`+`
		`+ ....`
		`+ $ koji taginfo epel7-build`
		`+ ....`
		`+ +`
		`+ Fedora`
		`+`
		`+ ....`
		`+ $ koji taginfo f28-build`
		`+ ....`
		`+`
		`+ ==== Create Side Build Target`
		`+`
		`+ . Create the proper tag.`
		`+ +`
		`+ Note the slightly different syntax depending on which product needs the`
		`+ side-build target and the comma delimited list of architectures based on`
		`+ the information from a previous step.`
		`+ +`
		`+ EPEL6`
		`+`
		`+ ....`
		`+ $ koji add-tag epel6-kde --parent=dist-6E-epel-build --arches=i686,x86_64,ppc64`
		`+ ....`
		`+ +`
		`+ EPEL7`
		`+`
		`+ ....`
		`+ $ koji add-tag epel7-kde --parent=epel7-build --arches=aarch64,x86_64,ppc64,ppc64le`
		`+ ....`
		`+ +`
		`+ Fedora`
		`+`
		`+ ....`
		`+ $ koji add-tag f28-kde --parent=f28-build --arches=armv7hl,i686,x86_64,aarch64,ppc64,ppc64le,s390x`
		`+ ....`
		`+ . Create the target.`
		`+ +`
		`+ EPEL6`
		`+`
		`+ ....`
		`+ $ koji add-target epel6-kde epel6-kde`
		`+ ....`
		`+ +`
		`+ EPEL7`
		`+`
		`+ ....`
		`+ $ koji add-target epel7-kde epel7-kde`
		`+ ....`
		`+ +`
		`+ Fedora`
		`+`
		`+ ....`
		`+ $ koji add-target f28-kde f28-kde`
		`+ ....`
		`+ . Find the taskID for the newRepo task associated with the newly created`
		`+ target.`
		`+ +`
		`+ ....`
		`+ $ koji list-tasks --method=newRepo`
		`+ ID Pri Owner State Arch Name`
		`+ 25101143 15 kojira OPEN noarch newRepo (f28-kde)`
		`+ ....`
		`+ . Ensure the newRepo task is being run across all architectures.`
		`+ +`
		`+ ....`
		`+ $ koji watch-task 25101143`
		`+ Watching tasks (this may be safely interrupted)...`
		`+ 25101143 newRepo (f28-kde): open (buildvm-14.phx2.fedoraproject.org)`
		`+ 25101154 createrepo (i386): closed`
		`+ 25101150 createrepo (ppc64le): closed`
		`+ 25101152 createrepo (ppc64): closed`
		`+ 25101151 createrepo (aarch64): closed`
		`+ 25101149 createrepo (armhfp): closed`
		`+ 25101153 createrepo (s390x): open (buildvm-ppc64le-04.ppc.fedoraproject.org)`
		`+ 25101148 createrepo (x86_64): open (buildvm-aarch64-21.arm.fedoraproject.org)`
		`+ ....`
		`+ . Request Package Auto-Signing for New Tag`
		`+ +`
		`+ File a ticket in https://pagure.io/fedora-infrastructure/issues[pagure`
		`+ infrastructure] requesting the new tag be enabled for package`
		`+ auto-signing.`
		`+ . Update the Pagure Issue`
		`+ +`
		`+ Update the issue according to the following template which assumes a`
		`+ side target was made for KDE under Fedora 28. _TAG_NAME_ has been`
		`+ created:`
		`+ +`
		`+ ____`
		`+ $ koji add-tag f28-kde --parent=f28-build`
		`+ --arches=armv7hl,i686,x86_64,aarch64,ppc64,ppc64le,s390x`
		`+`
		`+ $ koji add-target f28-kde f28-kde`
		`+`
		`+ You can do builds with:`
		`+`
		`+ $ fedpkg build --target=f28-kde`
		`+`
		`+ Let us know when you are done and we will move all the builds into f28.`
		`+ ____`
		`+`
		`+ === Cleanup`
		`+`
		`+ Fedora Release Engineering is responsible for merging the packages from`
		`+ the side-build target and tag back into the main tag. The requestor will`
		`+ update the original ticket when ready for the procedure outlined below.`
		`+`
		`+ . Remove the target`
		`+ +`
		`+ ....`
		`+ $ koji remove-target <SIDE_TAG_NAME>`
		`+ ....`
		`+ . Merge side build back to main target.`
		`+ +`
		`+ Get the latest checkout from https://pagure.io/releng/[Fedora Release`
		`+ Engineering Repository] and run the [.title-ref]#mass-tag.py# from the`
		`+ scripts directory.`
		`+ +`
		`+ ....`
		`+ $ ./mass-tag.py --source <SIDE_TAG_NAME> --target <MAIN_TAG_NAME> > mass_tag.txt`
		`+ ....`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The _MAIN_TAG_NAME_ for Fedora is typically the pending subtag, e.g.`
		+ `f28-pending` when bodhi is not managing updates. After bodhi is enabled
		+ and managing updates then merge into `f28-updates-candidate`.
		`+ ====`
		`+ . Paste Output to the Original Ticket`
		`+ +`
		`+ Paste the output from mass-tag.py into the pagure/releng ticket to show`
		`+ what packages were merged and what packages need rebuilding for those`
		`+ who work on the buildroot.`
		`+`
		`+ Tags are never removed.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ * Is the amount of work to be done worth the cost of newRepo tasks.`
		`+ * If there is only a small number of packages overrides may be better.`
		`+ * Is there a mass-rebuild going on? no side tags are allowed while a`
		`+ mass rebuild is underway`

modules/release_guide/pages/sop_bodhi_activation.adoc

file added

+124

		`@@ -0,0 +1,124 @@`
		`+ == Bodhi Activation Point`
		`+`
		`+ === Description`
		`+`
		`+ Bodhi must be activated after two weeks of`
		`+ https://docs.pagure.org/releng/sop_mass_branching.html[Mass Branching]`
		`+ at 14:00 UTC.`
		`+`
		`+ === Action`
		`+`
		`+ ==== Making koji changes`
		`+`
		`+ Make the following koji tag changes`
		`+`
		`+ ....`
		`+ $ koji remove-tag-inheritance f33-updates-candidate f33`
		`+ $ koji remove-tag-inheritance f33-updates-testing f33`
		`+ $ koji remove-tag-inheritance f33-updates-pending f33`
		`+ $ koji remove-tag-inheritance f33-override f33`
		`+ $ koji add-tag-inheritance f33-updates-candidate f33-updates`
		`+ $ koji add-tag-inheritance f33-updates-testing f33-updates`
		`+ $ koji add-tag-inheritance f33-updates-pending f33-updates`
		`+ $ koji add-tag-inheritance f33-override f33-updates`
		`+ $ koji edit-tag --perm=admin f33`
		`+ ....`
		`+`
		`+ ==== Update bodhi rpm release`
		`+`
		`+ Set the bodhi rpm to release to not to automatically create the update`
		`+ and also bodhi knows to compose the updates`
		`+`
		`+ ....`
		`+ $ bodhi releases edit --name "F33" --stable-tag f33-updates --testing-repository updates-testing --package-manager dnf --no-create-automatic-updates --composed-by-bodhi --state frozen`
		`+ ....`
		`+`
		`+ ==== Add the modular release`
		`+`
		`+ Run the following command on your own workstation to add the modular`
		`+ release`
		`+`
		`+ ....`
		`+ $ bodhi releases create --name F33M --long-name "Fedora 33 Modular" --id-prefix FEDORA-MODULAR --version 33 --branch f33m --dist-tag f33-modular --stable-tag f33-modular-updates --testing-tag f33-modular-updates-testing --candidate-tag f33-modular-updates-candidate --pending-stable-tag f33-modular-updates-pending --pending-testing-tag f33-modular-updates-testing-pending --pending-signing-tag f33-modular-signing-pending --override-tag f33-modular-override --state pending --user mohanboddu`
		`+ ....`
		`+`
		`+ [WARNING]`
		`+ .Warning`
		`+ ====`
		`+ Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in`
		`+ Bodhi, it is critical that Bodhi processes be restarted any time`
		+ `bodhi releases create` or `bodhi releases edit` are used.
		`+ ====`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Add the container and flatpak releases if they weren't already added to`
		`+ bodhi`
		`+ ====`
		`+`
		`+ === Update vars`
		`+`
		`+ Update the _FedoraBranchedBodhi_ and _RelEngFrozen_ vars in infra`
		`+ ansible`
		`+`
		`+ === Update all relevant projects in ansible`
		`+ As in https://pagure.io/fedora-infra/ansible/pull-request/1327[this Ansible Pull request] create changes for the {branched} release`
		`+ ==== Run the playbooks`
		`+`
		`+ ....`
		`+ $ rbac-playbook openshift-apps/greenwave.yml`
		`+ $ rbac-playbook openshift-apps/bodhi.yml`
		`+ $ rbac-playbook groups/bodhi-backend.yml`
		`+ $ rbac-playbook groups/releng-compose.yml`
		`+ $ rbac-playbook manual/autosign.yml`
		`+ ....`
		`+`
		`+ Greenwave runs in OpenShift (as implied by the playbook paths), and so`
		`+ the change will not be live right away when the playbook finishes. You`
		`+ can monitor`
		`+ https://greenwave-web-greenwave.app.os.fedoraproject.org/api/v1.0/policies`
		`+ to wait for the new policy to appear (it should take a few minutes).`
		`+`
		`+ ==== Restart bodhi services`
		`+`
		`+ Restart bodhi services to understand the bodhi new release on`
		`+ bodhi-backend01 (Look at warning in`
		`+ https://docs.pagure.org/releng/sop_bodhi_activation.html#action and the`
		`+ bug is https://github.com/fedora-infra/bodhi/issues/2177)`
		`+`
		`+ ....`
		`+ $ sudo systemctl restart bodhi-celery`
		`+ $ sudo systemctl restart fm-consumer@config`
		`+ $ sudo systemctl restart koji-sync-listener`
		`+ ....`
		`+`
		`+ ==== Send Announcement`
		`+`
		`+ Email devel-announce and test-announce lists about Bodhi Activation.`
		`+ Please find the body of the email in templates dir in https://pagure.io/releng/blob/main/f/mail-templates/04-beta-freeze.txt[releng repository]`
		`+`
		`+`
		`+ === Verification`
		`+`
		`+ Compare koji tagging structure with older release`
		`+`
		`+ ....`
		`+ $ koji list-tag-inheritance <branched_release> --reverse`
		`+ $ koji list-tag-inheritance <latest_stable_release> --reverse`
		`+ ....`
		`+`
		`+ Compare the bodhi release with older release`
		`+`
		`+ ....`
		`+ $ bodhi releases info <branched_release>`
		`+ $ bodhi releases info <latest_stable_release>`
		`+ ....`
		`+`
		`+ Check for other variants like modular, container and flatpaks`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ No considerations at this time. The docs git repository is simply a`
		`+ static html hosting space and we can just re-render the docs and push to`
		`+ it again if necessary.`

modules/release_guide/pages/sop_branch_freeze.adoc

file added

+33

		`@@ -0,0 +1,33 @@`
		`+ == Branching Freeze`
		`+`
		`+ === Introduction/Background`
		`+`
		`+ When the next release is branched from rawhide, it initially composes`
		`+ much like rawhide with nightly composes and no updates process.`
		`+`
		`+ Once the Bodhi is activated, we will push updates to the branched and`
		`+ the nightly composes will start to differ. But two weeks before the`
		`+ scheduled release of either Beta or GA, we will start the freeze for`
		`+ that release and stop pushing updates.`
		`+`
		`+ * Send announcement to devel-announce mailing list noting that the alpha`
		`+ change freeze is going to happen at least one day in advance.`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ For updates pushers: In Change freeze only updates that fix accepted`
		`+ blockers or Freeze break bugs are allowed into the main tree. Please`
		`+ coordinate with QA for any stable updates pushes. Otherwise ONLY push`
		`+ updates-testing.`
		`+ ====`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ For Final/GA release: During Final freeze, we dont want to block any`
		`+ packages in koji as it will effect the RC composes. So, please update`
		`+ the`
		`+ https://pagure.io/releng/blob/master/f/scripts/block_retired.py[block_retired.py]`
		`+ script and remove the branched release reference.`
		`+ ====`

modules/release_guide/pages/sop_branching.adoc

file added

+33

		`@@ -0,0 +1,33 @@`
		`+ == Branching`
		`+`
		`+ === Description`
		`+`
		`+ This SOP covers how to make git and pkgdb branches for packages, either`
		`+ for new packages that have passed review, or for existing packages that`
		`+ need a new branch (e.g. EPEL). Release Engineering has written a script`
		`+ to automate this process.`
		`+`
		`+ === Normal Action (automated)`
		`+`
		`+ . On your local system (not on an infrastructure hosted system), be sure`
		`+ you have the following packages installed:`
		`+ * python-bugzilla`
		`+ * python-configobj`
		`+ * python-fedora`
		`+ . Run "bugzilla login" and successfully receive an Authorization cookie.`
		`+ . {blank}`
		`+ +`
		`+ Clone the fedora-infrastructure tools repository:::`
		`+ ....`
		`+ git clone https://pagure.io/releng.git`
		`+ ....`
		`+ . In scripts/process-git-requests, run "process-git-requests". Answer`
		`+ the prompts.`
		`+`
		`+ === Manual Action`
		`+`
		`+ ==== Creating a new branch for an existing package`
		`+`
		+ . ssh into `pkgs.fedoraproject.org`
		+ . `pkgdb-client edit -u $YOURUSERNAME -b $NEWBRANCH --master=devel $NAMEOFPACKAGE`
		+ . `pkgdb2branch.py $NAMEOFPACKAGE`

modules/release_guide/pages/sop_breaking_development_freeze.adoc

file added

+51

		`@@ -0,0 +1,51 @@`
		`+ == Breaking Development Freeze`
		`+`
		+ `FIXME: NEED TO FIGURE OUT HOW TO FEDORA-VERSION-NEXT`
		`+`
		`+ === Description`
		`+`
		`+ Packages which require an exception to freeze policies must be run`
		`+ through this SOP.`
		`+`
		`+ The following freeze policies are set for the following significant`
		`+ release milestones:`
		`+`
		`+ * https://fedoraproject.org/wiki/Software_String_Freeze_Policy[String`
		`+ Freeze]`
		`+ * https://fedoraproject.org/wiki/Milestone_freezes[Beta Freeze]`
		`+ * https://fedoraproject.org/wiki/Milestone_freezes[Final Freeze]`
		`+`
		`+ See https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora`
		`+ Release Life Cycle] for a summary of all the freezes, dates, and`
		`+ exception handling, or the release engineering`
		`+ [https://fedorapeople.org/groups/schedule/f-\{\{FedoraVersionNumbernext}}-releng-tasks.html`
		`+ calendar for the current release].`
		`+`
		`+ === Action`
		`+`
		`+ The commands to tag a package properly once it has been accepted:`
		`+`
		`+ ....`
		`+ $ koji move-pkg --force dist-f{{FedoraVersionNumber\|next}}-updates-candidate dist-f{{FedoraVersionNumber\|next}} <PKGNAME>`
		`+ $ koji tag-pkg --force f{{FedoraVersionNumber\|next}}-<RELEASE> <PKGNAME>`
		`+ ....`
		`+`
		`+ Where <PKGNAME> is the package name, and <RELEASE> is the first release`
		`+ in which the package should land (e.g. alpha, beta, final).`
		`+`
		`+ === Verification`
		`+`
		+ The `koji` client reports success or failure. For secondary
		`+ verification, run these commands:`
		`+`
		`+ ....`
		`+ $ koji latest-pkg dist-f{{FedoraVersionNumber\|next}} <PKGNAME>`
		`+ $ koji latest-pkg dist-f{{FedoraVersionNumber\|next}}-updates-candidate <PKGNAME>`
		`+ ....`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ * Change agrees with stated policies (see links above)`
		`+ * Approval for change has been granted under`
		`+ https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[Blocker Bug`
		`+ Process] or [.title-ref]#Freeze Exception Bug Process#`

modules/release_guide/pages/sop_clean_amis.adoc

file added

+59

		`@@ -0,0 +1,59 @@`
		`+ == Clean AMIs Process`
		`+`
		`+ === Description`
		`+`
		`+ The Fedora AMIs are uploaded on a daily basis to Amazon Web Services.`
		`+ Over time the number of AMIs piles up and have to be removed manually.`
		`+ Manual removal comes with it's own set of issues where missing to delete`
		`+ the AMIs is a viable issue.`
		`+`
		`+ The goal of the script is to automate the process and continue regular`
		`+ removal of the AMIs. The report of the script is pushed to a`
		`+ https://pagure.io/ami-purge-report[Pagure repo]`
		`+`
		`+ === Action`
		`+`
		`+ There is a script in the https://pagure.io/releng[Fedora RelEng repo]`
		+ named `clean-amis.py` under the `scripts` directory.
		`+`
		`+ The script runs as a cron job within the Fedora Infrastructure to delete`
		`+ the old AMIs. The permission of the selected AMIs are changed to`
		`+ private. This is to make sure that if someone from the community raises`
		`+ an issue we have the option to get the AMI back to public. After 10`
		`+ days, if no complaints are raised the AMIs are deleted permanently.`
		`+`
		`+ The complete process can be divided in couple of parts:`
		`+`
		`+ * Fetching the data from datagrepper. Based on the [.title-ref]#--days#`
		`+ param, the script starts fetching the fedmsg messages from datagrepper`
		`+ for the specified timeframe i.e. for lasts [.title-ref]#n# days, where`
		`+ [.title-ref]#n# is the value of [.title-ref]#--days# param. The queried`
		`+ fedmsg topic [.title-ref]#fedimg.image.upload#.`
		`+ * Selection of the AMIs: After the AMIs are parsed from datagrepper. The`
		`+ AMIs are filtered to remove Beta, Two-week Atomic Host and GA released`
		`+ AMIs. Composes with [.title-ref]#compose_type# set to`
		`+ [.title-ref]#nightly# are picked up for deletion. Composes which contain`
		`+ date in the [.title-ref]#compose label# are also picked up for deletion.`
		`+ GA composes also have the compose_type set to production. So to`
		`+ distinguish then we filter them if the compose_label have date in them.`
		`+ The GA composes dont have date whereas they have the version in format`
		`+ of X.Y`
		`+ * Updated permissions of AMIs The permissions of the selected AMIs are`
		`+ changed to private.`
		`+ * Deletion of AMIs After 10 days, the private AMIs are deleted.`
		`+`
		`+ In order to change the permissions of the AMIs use the command given`
		`+ below, add [.title-ref]#--dry-run# argument test the command works.`
		`+ Adding [.title-ref]#--dry-run# argument will print the AMIs to console.`
		`+`
		`+ ....`
		`+ AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} PAGURE_ACCESS_TOKEN={{ ami_purge_report_api_key }} ./clean-amis.py --change-perms --days 7 --permswaitperiod 5`
		`+ ....`
		`+`
		`+ In order to delete the AMIs whose launch permissions have been removed,`
		`+ add [.title-ref]#--dry-run# argument test the command works. Adding`
		`+ [.title-ref]#--dry-run# argument will print the AMIs to console.`
		`+`
		`+ ....`
		`+ AWS_ACCESS_KEY={{ ec2_image_delete_access_key_id }} AWS_SECRET_ACCESS_KEY={{ ec2_image_delete_access_key }} PAGURE_ACCESS_TOKEN={{ ami_purge_report_api_key }} ./clean-amis.py --delete --days 17 --deletewaitperiod 10`
		`+ ....`

modules/release_guide/pages/sop_composing_fedora.adoc

file added

+622

		`@@ -0,0 +1,622 @@`
		`+ == Composing Fedora`
		`+`
		`+ === Description`
		`+`
		`+ All composes are defined by configuration files kept in the`
		`+ https://pagure.io/pungi-fedora[pungi-fedora repository].`
		`+`
		`+ Composes fall into two categories. They may be release candidates`
		`+ created on demand or nightly composes set to run at a scheduled time`
		`+ each day.`
		`+`
		`+ [cols=",,",options="header",]`
		`+ \|===`
		`+ \|Compose Name \|Configuration File \|Compose Script`
		`+ \|Docker \|fedora-docker.conf \|docker-nightly.sh`
		`+ \|Cloud \|fedora-cloud.conf \|cloud-nightly.sh`
		`+ \|Modular \|fedora-modular.conf \|modular-nightly.sh`
		`+ \|Nightly \|fedora.conf \|nightly.sh`
		`+ \|===`
		`+`
		`+ When Quality Engineering (QE) requests a Release Candidate (RC) they do`
		`+ so by opening an issue in the releng repository on pagure. Release`
		`+ candidate composes are not currently automated.`
		`+`
		`+ [cols=",,",options="header",]`
		`+ \|===`
		`+ \|Compose Name \|Configuration File \|Compose Script`
		`+ \|Beta \|fedora-beta.conf \|release-candidate.sh`
		`+ \|GA \|fedora-final.conf \|release-candidate.sh`
		`+ \|===`
		`+`
		`+ === Action`
		`+`
		`+ The following procedures are for release candidates only. They do not`
		`+ apply to the scheduled nightly composes.`
		`+`
		`+ ==== Review Compose Tags`
		`+`
		`+ . List any pre-existing builds in the current compose tag`
		`+ +`
		`+ ....`
		`+ $ koji list-tagged f[release_version]-compose`
		`+ ....`
		`+ . Verify pre-existing builds are in compose tags`
		`+ +`
		`+ The tagged builds from the previous composes should all be present in`
		`+ the output from the previous step. Consult the request ticket for the`
		`+ list of builds expected in this output.`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The very first run of an Beta, or GA compose should have no builds`
		`+ listed under the compose tag. It is important to clear pre-existing`
		`+ builds from the compose tag when moving between the Beta and RC`
		`+ composes. Verify that these builds were removed.`
		`+`
		`+ ....`
		`+ $ koji list-tagged f[release_version]-compose`
		`+ $ koji untag-build --all f[release_version]-compose [build1 build2 ...]`
		`+ ....`
		`+ ====`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The order in which packages are added into the`
		`+ f[release_version]-compose tag matter. If the builds are untagged`
		`+ erroneously then special attention should be given to adding them back`
		`+ correctly.`
		`+ ====`
		`+ . Add builds specified by QE to the current compose tag`
		`+ +`
		`+ ....`
		`+ $ koji tag-build f[release_version]-compose [build1 build2 ...]`
		`+ ....`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ These steps may be completed on a local machine as long as the user has`
		`+ appropriate permissions in the koji tool.`
		`+ ====`
		`+`
		`+ ==== Package Signing before the Compose`
		`+`
		`+ . Check for unsigned packages`
		`+ +`
		`+ ....`
		`+ $ koji list-tagged f[release_version]-signing-pending`
		`+ ....`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ If there are unsigned builds then wait for the automated queue to pick`
		`+ them up and sign them. Contact a member of the Fedora infrastructure`
		`+ team if the package signing has taken more than thirty minutes.`
		`+ ====`
		`+`
		`+ ==== Running the Compose`
		`+`
		`+ . Update the pungi-fedora config file Composes use a configuration file`
		`+ to construct the compose. Each compose uses its own configuration. The`
		+ `global_release` variable should start from 1.1 and the second number
		`+ should increment each time a new compose is created.`
		+ * Beta - `fedora-beta.conf`
		+ * RC - `fedora-final.conf`
		`+ . Log into the compose backend`
		`+ +`
		`+ ....`
		`+ $ ssh compose-x86-01.phx2.fedoraproject.org`
		`+ ....`
		`+ . Open a screen session`
		`+ +`
		`+ ....`
		`+ $ screen`
		`+ ....`
		`+ . Obtain the pungi-fedora branch for the current compose`
		`+ +`
		`+ The first time any user account executes a compose the pungi-fedora git`
		`+ repository must be cloned. The compose candidate script that invokes`
		+ pungi should be run from `compose-x86-01.iad2.fedoraproject.org`.
		`+ +`
		`+ ....`
		`+ $ git clone ssh://git@pagure.io/pungi-fedora.git`
		`+ ....`
		`+ +`
		`+ Enter the pungi-fedora directory.`
		`+ +`
		`+ ....`
		`+ $ cd pungi-fedora`
		`+ ....`
		`+ +`
		`+ If the clone step above was not required then fully update the existing`
		`+ repository checkout from pagure.`
		`+ +`
		`+ ....`
		`+ $ git fetch origin`
		`+ $ git checkout f[release_version]`
		`+ $ git pull origin f[release_version]`
		`+ ....`
		`+ . Run the compose`
		`+ +`
		`+ ....`
		`+ $ sudo ./release-candidate.sh [Beta\|RC]-#.#`
		`+ ....`
		`+ +`
		`+ The numbering scheme begins with 1.1 and the second number is`
		`+ incremented after each compose.`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Pungi requires numbers in the format #.# as an argument. It is because`
		`+ of this that composes always start with the number 1 and the second`
		`+ number is incremented with each compose.`
		`+ ====`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ If the compose fails with a directory missing error, then create the`
		+ compose directory with `mkdir /mnt/koji/compose/[release_version]`
		`+ ====`
		`+`
		`+ ==== Syncing the Compose`
		`+`
		+ We sync the compose to `/pub/alt/stage` to enable faster access to new
		`+ content for QA and the larger Fedora community.`
		`+`
		`+ . Log into the compose backend`
		`+ +`
		`+ ....`
		`+ $ ssh compose-x86-01.iad2.fedoraproject.org`
		`+ ....`
		`+ . Open a screen session`
		`+ +`
		`+ ....`
		`+ $ screen`
		`+ ....`
		`+ . Check the status of the compose`
		`+ +`
		`+ ....`
		`+ $ cat /mnt/koji/compose/[release_version]/[compose_id]/STATUS`
		`+ ....`
		`+ +`
		+ Do not continue with any further steps if the output above is `DOOMED`.
		`+ . Create the directory targeted for the copy :`
		`+ +`
		`+ ....`
		`+ $ sudo -u ftpsync mkdir -m 750 -p /pub/alt/stage/[release_version]_[release_label]-[#.#]`
		`+ ....`
		`+ . Locate the compose directory that will be the copy source :`
		`+ +`
		`+ ....`
		`+ $ ls /mnt/koji/compose/[release_version]/[compose_id]`
		`+ ....`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Take care executing the synchronization if the next compose is already`
		`+ running. Be sure to grab the correct directory.`
		`+`
		`+ If in doubt, check`
		`+ /mnt/koji/compose/[release_version]/[compose_id]/STATUS to be sure it is`
		`+ finished.`
		`+ ====`
		`+ . Run the synchronization one-liner`
		`+ +`
		`+ The synchronization of the completed compose to the public domain is`
		`+ currently a one-liner shell script. Pay close attention to what needs`
		`+ replaced in the example below.`
		`+ +`
		`+ ....`
		`+ $ sudo -u ftpsync sh -c 'for dir in Everything Cloud Container Kinoite Labs Modular Server Silverblue Spins Workstation metadata; do rsync -avhH /mnt/koji/compose/31/Fedora-31-20190911.0/compose/$dir/ /pub/alt/stage/31_Beta-1.1/$dir/ --link-dest=/pub/fedora/linux/development/31/Everything/ --link-dest=/pub/alt/stage/31_Beta-1.1/Everything/; done'`
		`+ ....`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ If multiple composes are run like 1.2, 1.3, add multiple --link-dest`
		`+ arguments above with multiple composes`
		`+ ====`
		`+ . Set the permissions of the synced compose :`
		`+ +`
		`+ ....`
		`+ $ sudo -u ftpsync chmod 755 /pub/alt/stage/[release_version]_[release_label]-[#.#]`
		`+ ....`
		`+ . Update the issue in the releng pagure repository`
		`+ +`
		`+ Once the compose and sync is complete the issue in pagure should be`
		`+ updated and closed.`
		`+ +`
		`+ Standard Ticket Verbage`
		`+`
		`+ Compose is done and available from`
		`+ https://kojipkgs.fedoraproject.org/compose/26/Fedora-26-20170328.0/compose/`
		`+ it has been synced to`
		`+ http://dl.fedoraproject.org/pub/alt/stage/26_Alpha-1.4/ rpms have all be`
		`+ hardlinked to /pub/fedora/linux/development/26/`
		`+`
		`+ ===== Verification`
		`+`
		`+ The method for verifying a compose has completed is checking`
		+ `/mnt/koji/compose/[release_version]/[compose_dir]/STATUS`. Any status
		`+ other than DOOMED is OK.`
		`+`
		`+ === Pre-Release Work`
		`+`
		`+ ==== Pushing Updates to Stable`
		`+`
		`+ When the release is signed off on Thursday after the Go/No-Go meeting,`
		`+ push the freeze and blocker to stable updates`
		`+`
		`+ Generally the updates are requested stable by QA. If they are not`
		`+ available, you can request them by following`
		`+`
		`+ ....`
		`+ $ bodhi updates request <updateid> stable`
		`+ ....`
		`+`
		`+ Once the updates are requested stable, please push them to stable by`
		`+ following the`
		`+ https://docs.pagure.org/releng/sop_pushing_updates.html#pushing-stable-updates-during-freeze[bodhi`
		`+ push to stable sop]`
		`+`
		`+ ==== koji tag changes`
		`+`
		`+ Once the updates are pushed stable, we need to clone the koji tag for`
		`+ beta release or lock the koji tag for final release.`
		`+`
		`+ ===== For Beta Release`
		`+`
		`+ ....`
		`+ $ koji clone-tag --all --latest-only f31 f31-Beta`
		`+ $ koji clone-tag --all --latest-only f31-modular f31-Beta-modular`
		`+ ....`
		`+`
		`+ ===== For Final Release`
		`+`
		`+ ....`
		`+ $ koji edit-tag --lock f31`
		`+ $ koji edit-tag --lock f31-modular`
		`+ ....`
		`+`
		`+ ==== Bodhi Changes`
		`+`
		+ Set the bodhi release to `current`
		`+`
		`+ ....`
		`+ $ bodhi releases edit --name F31 --state current`
		`+ ....`
		`+`
		`+ === Changes for Final Release`
		`+`
		`+ Once Final is GO, we need to perform different changes as that of Beta`
		`+ release.`
		`+`
		`+ ==== Last Branched Compose`
		`+`
		`+ Manually run a branched compose so that the GOLD content is same as the`
		`+ nightly compose. This also helps in updating the silverblue refs as that`
		`+ of the GOLD content.`
		`+`
		`+ ==== Update silverblue refs`
		`+`
		`+ Please update the refs as per the following commands on`
		`+ [.title-ref]#bodhi-backend01.phx2.fedoraproject.org#`
		`+`
		`+ Run the following commands from`
		`+ [.title-ref]#/mnt/koji/compose/ostree/repo# and`
		`+ [.title-ref]#/mnt/koji/ostree/repo/#`
		`+`
		`+ ....`
		`+ $ sudo -u ftpsync ostree refs --create=fedora/31/x86_64/updates/silverblue fedora/31/x86_64/silverblue`
		`+ $ sudo -u ftpsync ostree refs --create=fedora/31/aarch64/updates/silverblue fedora/31/aarch64/silverblue`
		`+ $ sudo -u ftpsync ostree refs --create=fedora/31/ppc64le/updates/silverblue fedora/31/ppc64le/silverblue`
		`+`
		`+ $ sudo ostree refs --delete fedora/31/x86_64/silverblue`
		`+ $ sudo ostree refs --delete fedora/31/aarch64/silverblue`
		`+ $ sudo ostree refs --delete fedora/31/ppc64le/silverblue`
		`+`
		`+ $ sudo -u ftpsync ostree refs --alias --create=fedora/31/x86_64/silverblue fedora/31/x86_64/updates/silverblue`
		`+ $ sudo -u ftpsync ostree refs --alias --create=fedora/31/aarch64/silverblue fedora/31/aarch64/updates/silverblue`
		`+ $ sudo -u ftpsync ostree refs --alias --create=fedora/31/ppc64le/silverblue fedora/31/ppc64le/updates/silverblue`
		`+ ....`
		`+`
		`+ Run the following command only from [.title-ref]#/mnt/koji/ostree/repo/#`
		`+`
		`+ ....`
		`+ $ sudo ostree summary -u`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Before pushing the updates to fxx-updates, run the last branched compose`
		`+ so that both branched and rc composes have the same content. Once the`
		`+ branched compose is done, then update the silverblue refs as mentioned`
		`+ above. If the order is changed, that will screw up the refs`
		`+ ====`
		`+`
		`+ ==== Disable Branched Compose`
		`+`
		`+ Now that we have a final GOLD compose, we dont need nightly branched`
		`+ composes anymore. This is disabled in`
		`+ https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/releng[releng`
		`+ role] in infra ansible repo and then running the playbook.`
		`+`
		`+ ....`
		`+ $ sudo rbac-playbook groups/releng-compose.yml`
		`+ ....`
		`+`
		`+ ==== Lift RelEng freeze`
		`+`
		`+ Lift the RelEng Freeze so that the updates will be pushed to stable.`
		`+ This is done by editing`
		`+ https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/vars/all/RelEngFrozen.yaml[RelEngFrozen`
		`+ variable] in infra ansible repo and then run the bodhi playbook.`
		`+`
		`+ ....`
		`+ $ sudo rbac-playbook groups/bodhi-backend.yml`
		`+ ....`
		`+`
		`+ ==== Other Changes`
		`+`
		`+ These changes include enabling nightly container and cloud composes,`
		`+ other variable changes in infra ansible repo, bodhi pungi config`
		`+ changes, updates sync changes and others.`
		`+`
		`+ Run the appropriate playbooks after the following changes`
		`+`
		`+ ....`
		`+ diff --git a/roles/releng/files/branched b/roles/releng/files/branched`
		`+ index 966f5c3..1c0454f 100644`
		`+ --- a/roles/releng/files/branched`
		`+ +++ b/roles/releng/files/branched`
		`+ @@ -1,3 +1,3 @@`
		`+ # branched compose`
		`+ -MAILTO=releng-cron@lists.fedoraproject.org`
		+ -15 7 * * * root TMPDIR=`mktemp -d /tmp/branched.XXXXXX` && cd $TMPDIR && git clone https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && /usr/local/bin/lock-wrapper branched-compose "PYTHONMALLOC=debug LANG=en_US.UTF-8 ./nightly.sh" && sudo -u ftpsync /usr/local/bin/update-fullfiletimelist -l /pub/fedora-secondary/update-fullfiletimelist.lock -t /pub fedora fedora-secondary
		`+ +#MAILTO=releng-cron@lists.fedoraproject.org`
		+ +#15 7 * * * root TMPDIR=`mktemp -d /tmp/branched.XXXXXX` && cd $TMPDIR && git clone https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && /usr/local/bin/lock-wrapper branched-compose "PYTHONMALLOC=debug LANG=en_US.UTF-8 ./nightly.sh" && sudo -u ftpsync /usr/local/bin/update-fullfiletimelist -l /pub/fedora-secondary/update-fullfiletimelist.lock -t /pub fedora fedora-secondary
		`+ diff --git a/roles/releng/files/cloud-updates b/roles/releng/files/cloud-updates`
		`+ index a0ffbe8..287d57d 100644`
		`+ --- a/roles/releng/files/cloud-updates`
		`+ +++ b/roles/releng/files/cloud-updates`
		`+ @@ -6,6 +6,6 @@ MAILTO=releng-cron@lists.fedoraproject.org`
		`+ MAILTO=releng-cron@lists.fedoraproject.org`
		+ 15 7 * * * root TMPDIR=`mktemp -d /tmp/CloudF29.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f29 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+`
		`+ -#Fedora 28 Cloud nightly compose`
		`+ -#MAILTO=releng-cron@lists.fedoraproject.org`
		+ -#15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF28.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f28 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+ +#Fedora 31 Cloud nightly compose`
		`+ +MAILTO=releng-cron@lists.fedoraproject.org`
		+ +15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+ diff --git a/roles/releng/files/container-updates b/roles/releng/files/container-updates`
		`+ index d763149..5446840 100644`
		`+ --- a/roles/releng/files/container-updates`
		`+ +++ b/roles/releng/files/container-updates`
		`+ @@ -1,6 +1,6 @@`
		`+ -#Fedora 28 Container Updates nightly compose`
		`+ -#MAILTO=releng-cron@lists.fedoraproject.org`
		+ -#45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF28.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f28 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+ +#Fedora 31 Container Updates nightly compose`
		`+ +MAILTO=releng-cron@lists.fedoraproject.org`
		+ +45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+`
		`+ # Fedora 30 Container Updates nightly compose`
		`+ MAILTO=releng-cron@lists.fedoraproject.org`
		`+ diff --git a/vars/all/00-FedoraCycleNumber.yaml b/vars/all/00-FedoraCycleNumber.yaml`
		`+ index 22476b0..4bd0d46 100644`
		`+ --- a/vars/all/00-FedoraCycleNumber.yaml`
		`+ +++ b/vars/all/00-FedoraCycleNumber.yaml`
		`+ @@ -1 +1 @@`
		`+ -FedoraCycleNumber: 30`
		`+ +FedoraCycleNumber: 31`
		`+`
		`+ diff --git a/vars/all/FedoraBranched.yaml b/vars/all/FedoraBranched.yaml`
		`+ index 42ac534..0bbcc1d 100644`
		`+ --- a/vars/all/FedoraBranched.yaml`
		`+ +++ b/vars/all/FedoraBranched.yaml`
		`+ @@ -1 +1 @@`
		`+ -FedoraBranched: True`
		`+ +FedoraBranched: False`
		`+`
		`+ diff --git a/vars/all/FedoraPreviousPrevious.yaml b/vars/all/FedoraPreviousPrevious.yaml`
		`+ index a8e3d3b..a061e04 100644`
		`+ --- a/vars/all/FedoraPreviousPrevious.yaml`
		`+ +++ b/vars/all/FedoraPreviousPrevious.yaml`
		`+ @@ -1 +1 @@`
		`+ -FedoraPreviousPrevious: False`
		`+ +FedoraPreviousPrevious: True`
		`+ diff --git a/vars/all/Frozen.yaml b/vars/all/Frozen.yaml`
		`+ index 97d3bc3..7578a88 100644`
		`+ --- a/vars/all/Frozen.yaml`
		`+ +++ b/vars/all/Frozen.yaml`
		`+ @@ -1 +1 @@`
		`+ -Frozen: True`
		`+ +Frozen: False`
		`+`
		`+`
		`+ diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2`
		`+ index 688bade..28b524a 100644`
		`+ --- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2`
		`+ +++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2`
		`+ @@ -179,8 +179,8 @@ ostree = {`
		`+ # In the case of testing, also inject the last stable updates`
		`+ "https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/updates/f[[ release.version_int ]]-updates/compose/Everything/$basearch/os/",`
		`+ [% endif %]`
		`+ - # For f31 the compose location is going to be under /compose/branched/`
		`+ - [% if release.version_int == 31 %]`
		`+ + # For F32 the compose location is going to be under /compose/branched/`
		`+ + [% if release.version_int == 32 %]`
		`+ "https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/branched/latest-Fedora-[[ release.version_int ]]/compose/Everything/$basearch/os/"`
		`+ [% else %]`
		`+ "https://kojipkgs{{ env_suffix }}.fedoraproject.org/compose/[[ release.version_int ]]/latest-Fedora-[[ release.version_int ]]/compose/Everything/$basearch/os/"`
		`+`
		`+ diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2`
		`+ index 28b524a..640ddf0 100644`
		`+ --- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2`
		`+ +++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2`
		`+ @@ -193,8 +193,8 @@ ostree = {`
		`+ "ostree_ref": "fedora/[[ release.version_int ]]/${basearch}/testing/silverblue",`
		`+ [% endif %]`
		`+ "tag_ref": False,`
		`+ - "arches": ["x86_64"],`
		`+ - "failable": ["x86_64"]`
		`+ + "arches": ["x86_64", "ppc64le", "aarch64" ],`
		`+ + "failable": ["x86_64", "ppc64le", "aarch64" ]`
		`+ },`
		`+ ]`
		`+ }`
		`+`
		`+`
		`+ diff --git a/roles/bodhi2/backend/files/new-updates-sync b/roles/bodhi2/backend/files/new-updates-sync`
		`+ index d08c893..2d0fb4d 100755`
		`+ --- a/roles/bodhi2/backend/files/new-updates-sync`
		`+ +++ b/roles/bodhi2/backend/files/new-updates-sync`
		`+ @@ -25,8 +25,9 @@ RELEASES = {'f31': {'topic': 'fedora',`
		`+ 'modules': ['fedora', 'fedora-secondary'],`
		`+ 'repos': {'updates': {`
		`+ 'from': 'f31-updates',`
		`+ - 'ostrees': [{'ref': 'fedora/31/x86_64/updates/silverblue',`
		`+ - 'dest': OSTREEDEST}],`
		`+ + 'ostrees': [{'ref': 'fedora/31/%(arch)s/updates/silverblue',`
		`+ + 'dest': OSTREEDEST,`
		`+ + 'arches': ['x86_64', 'ppc64le', 'aarch64']}],`
		`+ 'to': [{'arches': ['x86_64', 'armhfp', 'aarch64', 'source'],`
		`+ 'dest': os.path.join(FEDORADEST, '31', 'Everything')},`
		`+ {'arches': ['ppc64le', 's390x'],`
		`+ @@ -34,8 +35,9 @@ RELEASES = {'f31': {'topic': 'fedora',`
		`+ ]},`
		`+ 'updates-testing': {`
		`+ 'from': 'f31-updates-testing',`
		`+ - 'ostrees': [{'ref': 'fedora/31/x86_64/testing/silverblue',`
		`+ - 'dest': OSTREEDEST}],`
		`+ + 'ostrees': [{'ref': 'fedora/31/%(arch)s/testing/silverblue',`
		`+ + 'dest': OSTREEDEST,`
		`+ + 'arches': ['x86_64', 'ppc64le', 'aarch64']}],`
		`+ 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'],`
		`+ 'dest': os.path.join(FEDORADEST, 'testing', '31', 'Everything')},`
		`+ {'arches': ['ppc64le', 's390x'],`
		`+`
		`+`
		`+ diff --git a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json`
		`+ index aac977e..9e0cbf2 100644`
		`+ --- a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json`
		`+ +++ b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json`
		`+ @@ -12,14 +12,14 @@`
		`+ "version": "devel"`
		`+ },`
		`+ {`
		`+ - "allow_retire": true,`
		`+ + "allow_retire": false,`
		`+ "branchname": "f31",`
		`+ "date_created": "2014-05-14 12:36:15",`
		`+ "date_updated": "2018-08-14 17:07:23",`
		`+ "dist_tag": ".fc31",`
		`+ "koji_name": "f31",`
		`+ "name": "Fedora",`
		`+ - "status": "Under Development",`
		`+ + "status": "Active",`
		`+ "version": "31"`
		`+ },`
		`+ {`
		`+ ....`
		`+`
		`+ ==== Bodhi config`
		`+`
		`+ ==== After Beta`
		`+`
		`+ ....`
		`+ diff --git a/vars/all/FedoraBranchedBodhi.yaml b/vars/all/FedoraBranchedBodhi.yaml`
		`+ index 606eb2e..ca2ba61 100644`
		`+ --- a/vars/all/FedoraBranchedBodhi.yaml`
		`+ +++ b/vars/all/FedoraBranchedBodhi.yaml`
		`+ @@ -3,4 +3,4 @@`
		`+ #prebeta: After bodhi enablement/beta freeze and before beta release`
		`+ #postbeta: After beta release and before final release`
		`+ #current: After final release`
		`+ -FedoraBranchedBodhi: prebeta`
		`+ +FedoraBranchedBodhi: postbeta`
		`+ ....`
		`+`
		`+ ==== After Final`
		`+`
		`+ ....`
		`+ diff --git a/vars/all/FedoraBranchedBodhi.yaml b/vars/all/FedoraBranchedBodhi.yaml`
		`+ index 380f61d..76ba14d 100644`
		`+ --- a/vars/all/FedoraBranchedBodhi.yaml`
		`+ +++ b/vars/all/FedoraBranchedBodhi.yaml`
		`+ @@ -1,2 +1,2 @@`
		`+ #options are: prebeta, postbeta, current`
		`+ -FedoraBranchedBodhi: postbeta`
		`+ +FedoraBranchedBodhi: current`
		`+ ....`
		`+`
		`+ ==== Mirroring`
		`+`
		`+ Run [.title-ref]#stage-release.sh# script from`
		`+ https://pagure.io/releng[releng repo] in pagure on`
		`+ [.title-ref]#bodhi-backend01.phx2.fedoraproject.org#, this will sign the`
		`+ checksums and will put the content on mirrors.`
		`+`
		`+ ===== For Beta`
		`+`
		`+ ....`
		`+ $ sh scripts/stage-release.sh 32_Beta Fedora-32-20200312.0 32_Beta-1.2 fedora-32 1`
		`+ ....`
		`+`
		`+ ===== For Final`
		`+`
		`+ ....`
		`+ $ sh scripts/stage-release.sh 32 Fedora-32-20200415.0 32_RC-1.1 fedora-32 0`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Make sure to grab the directory size usage numbers which is used to send`
		`+ an email to [.title-ref]#mirror-admin@lists.fedoraproject.org# list.`
		`+ ====`
		`+`
		`+ ==== Sync the signed checksums to stage`
		`+`
		`+ We need to sync the signed checksums to /pub/alt/stage/ by running the`
		`+ following command`
		`+`
		`+ ....`
		+ $ for dir in Everything Cloud Container Labs Server Spins Workstation Silverblue Kinoite metadata; do sudo -u ftpsync rsync -avhH /mnt/koji/compose/37/Fedora-37-20221105.0/compose/$dir/ /pub/alt/stage/37_RC-1.7/$dir/ --link-dest=/pub/fedora/linux/releases/37/Everything/ --link-dest=/pub/alt/stage/37_RC-1.2/Everything/ --link-dest=/pub/alt/stage/37_RC-1.3/Everything --link-dest=/pub/alt/stage/37_RC-1.4/Everything --link-dest=/pub/alt/stage/37_RC-1.5/Everything --link-dest=/pub/alt/stage/37_RC-1.6/Everything --link-dest=/pub/alt/stage/37_RC-1.7/Everything; done
		`+ ....`
		`+`
		`+ === Move development to release folder with mirrormanager`
		`+`
		`+ Two weeks after the release move bits from development to release`
		`+ directory`
		`+`
		`+ . {blank}`
		`+ +`
		`+ ssh to the mm-backend01.iad2.fedoraproject.org::`
		`+ ::;;`
		`+ $ ssh mm-backend01.iad2.fedoraproject.org`
		`+ . {blank}`
		`+ +`
		`+ get root::`
		`+ ::;;`
		`+ $ sudo su`
		`+ . {blank}`
		`+ +`
		`+ run the mm2_move-devel-to-release::`
		`+ ::;;`
		`+ $ mm2_move-devel-to-release --version=35 --category='Fedora Linux'`
		`+`
		`+ === Consider before Running`
		`+`
		`+ Composes and file synchronizations should be run in a screen session on`
		`+ a remote machine. This enables the operator to withstand network`
		`+ connection issues.`

modules/release_guide/pages/sop_create_release_signing_key.adoc

file added

+536

		`@@ -0,0 +1,536 @@`
		`+ == Create Release Signing Key`
		`+`
		`+ === Description`
		`+`
		`+ At the beginning of each release under development a new package signing`
		`+ key is created for it. This key is used to prove the authenticity of`
		`+ packages built by Fedora and distributed by Fedora. This key will be`
		`+ used to sign all packages for the public test and final releases.`
		`+`
		`+ === Action`
		`+`
		`+ ==== Sigul`
		`+`
		`+ Sigul is the signing server which holds our keys. In order to make use`
		`+ of a new key, the key will have to be created and access to the key will`
		+ have to be granted. The `new-key`, `grant-key-access`, and
		+ `change-passphrase` commands are used.
		`+`
		`+ ....`
		`+ $ sigul new-key --help`
		`+ usage: client.py new-key [options] key`
		`+`
		`+ Add a key`
		`+`
		`+ options:`
		`+ -h, --help show this help message and exit`
		`+ --key-admin=USER Initial key administrator`
		`+ --name-real=NAME_REAL`
		`+ Real name of key subject`
		`+ --name-comment=NAME_COMMENT`
		`+ A comment about of key subject`
		`+ --name-email=NAME_EMAIL`
		`+ E-mail of key subject`
		`+ --expire-date=YYYY-MM-DD`
		`+ Key expiration date`
		`+`
		`+ $ sigul grant-key-access --help`
		`+ usage: client.py grant-key-access key user`
		`+`
		`+ Grant key access to a user`
		`+`
		`+ options:`
		`+ -h, --help show this help message and exit`
		`+`
		`+ $ sigul change-passphrase --help`
		`+ usage: client.py change-passphrase key`
		`+`
		`+ Change key passphrase`
		`+`
		`+ options:`
		`+ -h, --help show this help message and exit`
		`+ ....`
		`+`
		`+ For example if we wanted to create the Fedora 23 signing key, we would`
		`+ do the following:`
		`+`
		`+ . Log into a system configured to run sigul client.`
		`+ . Create the key using a strong passphrase when prompted`
		`+ +`
		`+ ....`
		`+ $ sigul new-key --key-admin ausil --key-type gnupg \`
		`+ --gnupg-name-real Fedora \`
		`+ --gnupg-name-comment 23 \`
		`+ --gnupg-name-email fedora-23-primary@fedoraproject.org fedora-23`
		`+ ....`
		`+ +`
		`+ For EPEL`
		`+ +`
		`+ ....`
		`+ $ sigul new-key --key-admin ausil --key-type gnupg \`
		`+ --gnupg-name-real "Fedora EPEL" \`
		`+ --gnupg-name-comment 7 \`
		`+ --gnupg-name-email epel@fedoraproject.org epel-7`
		`+ ....`
		`+ . Wait a while for entropy. This can take several minutes.`
		`+ . For Fedora, also create the IMA signing key`
		`+ +`
		`+ ....`
		`+ $ sigul new-key --key-admin ausil --key-type ECC fedora-23-ima`
		`+ ....`
		`+ . Grant key access to Fedora Account holders who will be signing`
		`+ packages and protect it with a temporary a passphrase. For example,`
		+ `CHANGEME.`. Do the same with the -ima key for Fedora.
		`+ +`
		`+ ....`
		`+ $ sigul grant-key-access fedora-23 kevin`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ IMPORTANT: Grant the access to autopen user as it's required for`
		`+ robosignatory autosigning and then restart robosignatory service`
		`+ ====`
		`+`
		`+ . Provide the key name and temporary passphrase to signers. If they`
		`+ don't respond, revoke access until they are ready to change their`
		`+ passphrase. Signers can change their passphrase using the`
		+ `change-passphrase` command:
		`+ +`
		`+ ....`
		`+ $ sigul change-passphrase fedora-23`
		`+ ....`
		`+ . When your sigul cert expires, you will need to run:`
		`+ +`
		`+ ....`
		`+ certutil -d ~/.sigul -D -n sigul-client-cert`
		`+ ....`
		`+ +`
		`+ to remove the old cert, then`
		`+ +`
		`+ ....`
		`+ sigul_setup_client`
		`+ ....`
		`+ +`
		`+ to add a new one.`
		`+`
		`+ ==== fedora-repos`
		`+`
		`+ The fedora-repos package houses a copy of the public key information.`
		`+ This is used by rpm to verify the signature on files encountered.`
		`+ Currently the fedora-repos package has a single key file named after the`
		`+ version of the key and the arch the key is for. To continue our example,`
		+ the file would be named `RPM-GPG-KEY-fedora-27-primary` which is the
		`+ primary arch key for Fedora 27. To create this file, use the`
		+ `get-public-key` command from sigul:
		`+`
		`+ ....`
		`+ $ sigul get-public-key fedora-27 > RPM-GPG-KEY-fedora-27-primary`
		`+ ....`
		`+`
		`+ Add this file to the repo and update the archmap file for the new`
		`+ release.`
		`+`
		`+ ....`
		`+ $ git add RPM-GPG-KEY-fedora-27-primary`
		`+ ....`
		`+`
		`+ Then make a new fedora-repos build for rawhide`
		+ (`FIXME: this should be its own SOP`)
		`+`
		`+ ==== getfedora.org`
		`+`
		`+ getfedora.org/keys lists information about all of our keys. We need to`
		`+ let the websites team know we have created a new key so that they can`
		`+ add it to the list.`
		`+`
		`+ We do this by filing an issues in their pagure instance`
		`+ https://pagure.io/fedora-websites/ we should point them at this SOP`
		`+`
		`+ ===== Web team SOP`
		`+`
		`+ ....`
		`+ # from git repo root`
		`+ cd fedoraproject.org/`
		`+ curl $KEYURL > /tmp/newkey`
		`+ $EDITOR update-gpg-keys # Add key ID of recently EOL'd version to obsolete_keys`
		`+ ./update-gpg-key /tmp/newkey`
		`+ gpg static/fedora.gpg # used to verify the new keyring`
		`+ # it should look something like this:`
		`+ # pub 4096R/57BBCCBA 2009-07-29 Fedora (12) <fedora@fedoraproject.org>`
		`+ # pub 4096R/E8E40FDE 2010-01-19 Fedora (13) <fedora@fedoraproject.org>`
		`+ # pub 4096R/97A1071F 2010-07-23 Fedora (14) <fedora@fedoraproject.org>`
		`+ # pub 1024D/217521F6 2007-03-02 Fedora EPEL <epel@fedoraproject.org>`
		`+ # sub 2048g/B6610DAF 2007-03-02 [expires: 2017-02-27]`
		`+ # it must only have the two supported versions of fedora, rawhide and EPEL`
		`+ # also verify that static/$NEWKEY.txt exists`
		`+ $EDITOR data/content/{keys,verify}.html # see git diff 1840f96~ 1840f96`
		`+ ....`
		`+`
		`+ ==== sigulsign_unsigned`
		`+`
		+ `sigulsign_unsigned.py` is the script Release Engineers use to sign
		`+ content in koji. This script has a hardcoded list of keys and aliases to`
		`+ the keys that needs to be updated when we create new keys.`
		`+`
		+ Add the key details to the `KEYS` dictionary near the top of the
		+ `sigulsign_unsigned.py` script. It lives in Release Engineering's git
		+ repo at `ssh://git@pagure.io/releng.git` in the `scripts` directory. You
		`+ will need to know the key ID to insert the correct information:`
		`+`
		`+ ....`
		`+ $ gpg <key block from sigul get-public-key>`
		`+ ....`
		`+`
		`+ ==== Public Keyservers`
		`+`
		`+ We upload the key to the public key servers when we create the keys. To`
		`+ do this, we need to get the ascii key block from sigul, determine the`
		`+ key ID, import they key into our local keyring, and then upload it to`
		`+ the key servers.`
		`+`
		`+ ....`
		`+ $ sigul get-public-key fedora-13 > fedora-13`
		`+ $ gpg fedora-13 (The ID is the "E8E40FDE" part of 4096R/E8E40FDE)`
		`+ $ gpg --import fedora-13`
		`+ $ gpg --send-keys E8E40FDE`
		`+ ....`
		`+`
		`+ ==== pungi-fedora`
		`+`
		`+ The nightly compose configs come from the pungi-fedora project on`
		`+ https://pagure.io We need to create a pull request to pull in the new`
		`+ key.`
		`+`
		`+ ....`
		`+ $ git clone ssh://git@pagure.io/<your fork path>/pungi-fedora.git`
		`+ $ cd pungi-fedora`
		`+ $ vim *conf`
		`+ <set key value in sigkeys = line >`
		`+ $ git commit -m 'Add new key'`
		`+ $ git push`
		`+ $ file a Pull Request`
		`+ ....`
		`+`
		`+ ==== Koji`
		`+`
		`+ Koji has a garbage collection utility that will find builds that meet`
		`+ criteria to be removed to save space. Part of that criteria has to do`
		`+ with whether or not the build has been signed with a key. If the`
		`+ collection utility doesn't know about a key it will ignore the build.`
		`+ Thus as we create new keys we need to inform the utility of these keys`
		`+ or else builds can pile up. The configuration for the garbage collection`
		`+ lives within ansible.`
		`+`
		`+ On a clone of the infrastructure ansible git repo edit the`
		`+ roles/koji_hub/templates/koji-gc.conf.j2 file:`
		`+`
		`+ ....`
		`+ diff --git a/roles/koji_hub/templates/koji-gc.conf.j2 b/roles/koji_hub/templates/koji-gc.conf.j2`
		`+ index 9ecb750..9c48a8e 100644`
		`+ --- a/roles/koji_hub/templates/koji-gc.conf.j2`
		`+ +++ b/roles/koji_hub/templates/koji-gc.conf.j2`
		`+ @@ -35,6 +35,7 @@ key_aliases =`
		`+ 81B46521 fedora-24`
		`+ FDB19C98 fedora-25`
		`+ 64DAB85D fedora-26`
		`+ + F5282EE4 fedora-27`
		`+ 217521F6 fedora-epel`
		`+ 0608B895 fedora-epel-6`
		`+ 352C64E5 fedora-epel-7`
		`+ @@ -52,6 +53,7 @@ unprotected_keys =`
		`+ fedora-24`
		`+ fedora-25`
		`+ fedora-26`
		`+ + fedora-27`
		`+ fedora-extras`
		`+ redhat-beta`
		`+ fedora-epel`
		`+ @@ -91,6 +93,7 @@ policy =`
		`+ sig fedora-24 && age < 12 weeks :: keep`
		`+ sig fedora-25 && age < 12 weeks :: keep`
		`+ sig fedora-26 && age < 12 weeks :: keep`
		`+ + sig fedora-27 && age < 12 weeks :: keep`
		`+ sig fedora-epel && age < 12 weeks :: keep`
		`+ sig fedora-epel-6 && age < 12 weeks :: keep`
		`+ sig fedora-epel-7 && age < 12 weeks :: keep`
		`+ ....`
		`+`
		`+ In this case the fedora-epel key was added to the list of key aliases,`
		`+ then referenced in the list of unprotected_keys, and finally a policy`
		`+ was created for how long to keep builds signed with this key.`
		`+`
		`+ Once you've made your change commit and push. The buildsystem will pick`
		`+ up this change the next time puppet refreshes.`
		`+`
		`+ === Verification`
		`+`
		`+ We can verify that the key was created in sigul, the correct users have`
		`+ access to the key, the key was added to the fedora-release package, that`
		`+ the website was updated with the right key, that sigulsign_unsigned was`
		`+ properly updated, and that the key was successfully updated to the`
		`+ public key servers.`
		`+`
		`+ ==== sigul`
		`+`
		+ Use the `list-keys` command to verify that the key was indeed added to
		`+ sigul:`
		`+`
		`+ ....`
		`+ $ sigul list-keys`
		`+ Administrator's password:`
		`+ fedora-10`
		`+ fedora-10-testing`
		`+ fedora-11`
		`+ fedora-12`
		`+ fedora-13`
		`+ ....`
		`+`
		`+ Our new key should be on the list. This command expects your`
		`+ administrative password.`
		`+`
		+ Use the `list-key-users` command to verify all the signers have access:
		`+`
		`+ ....`
		`+ $ sigul list-key-users fedora-13`
		`+ Key passphrase:`
		`+ jkeating`
		`+ jwboyer`
		`+ ....`
		`+`
		`+ This command expects your key passphrase for the key in question.`
		`+`
		`+ ==== fedora-release`
		`+`
		`+ To verify that the key was added to this package correctly, download the`
		`+ latest build from koji and run rpm2cpio on it, then run gpg on the key`
		`+ file:`
		`+`
		`+ ....`
		`+ $ koji download-build --arch noarch --latest f27 fedora-repos`
		`+ fedora-repos-rawhide-27-0.1.noarch.rpm \| 7.3 kB 00:00:00`
		`+ fedora-repos-27-0.1.noarch.rpm \| 87 kB 00:00:00`
		`+ $ rpmdev-extract fedora-repos-27-0.1.noarch.rpm`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-27-fedora`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-10-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-11-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-12-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-arm`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-mips`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-arm`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-15-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-arm`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-16-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-arm`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-17-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-arm`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-18-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-19-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-20-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-aarch64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-ppc64le`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-21-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-aarch64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-ppc64le`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-aarch64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-ppc64le`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-s390`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-23-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-aarch64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-ppc64le`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-aarch64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-ppc64le`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-aarch64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-ppc64le`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-aarch64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-armhfp`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-ppc64le`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-s390x`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-7-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-primary-original`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-i386`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ia64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ppc`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-ppc64`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-primary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-primary-original`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-secondary`
		`+ fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-9-x86_64`
		`+ fedora-repos-27-0.1.noarch/etc/yum.repos.d`
		`+ fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-cisco-openh264.repo`
		`+ fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-updates-testing.repo`
		`+ fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora-updates.repo`
		`+ fedora-repos-27-0.1.noarch/etc/yum.repos.d/fedora.repo`
		`+`
		`+ $ gpg2 fedora-repos-27-0.1.noarch/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-27-primary`
		`+ pub rsa4096 2017-02-21 [SCE]`
		`+ 860E19B0AFA800A1751881A6F55E7430F5282EE4`
		`+ uid Fedora 27 (27) <fedora-27@fedoraproject.org>`
		`+ pub 4096R/E8E40FDE 2010-01-19 Fedora (13) <fedora@fedoraproject.org>`
		`+ ....`
		`+`
		`+ You may wish to do this in a tempoary directory to make cleaning it up`
		`+ easy.`
		`+`
		`+ ==== getfedora.org`
		`+`
		`+ One can simply browse to https://getfedora.org/keys to verify that the`
		`+ key has been uploaded.`
		`+`
		`+ ==== sigulsign_unsigned`
		`+`
		`+ The best way to test whether or not the key has been added correctly is`
		`+ to sign a package using the key, like our newly built fedora-repos`
		`+ package.`
		`+`
		`+ ....`
		`+ $ ./sigulsign_unsigned.py fedora-13 fedora-release-13-0.3`
		`+ Passphrase for fedora-13:`
		`+ ....`
		`+`
		`+ The command should exit cleanly.`
		`+`
		`+ ==== Public key servers`
		`+`
		`+ One can use the <code>search-keys</code> command from gpg to locate the`
		`+ key on the public server:`
		`+`
		`+ ....`
		`+ $ gpg2 --search-keys "Fedora (13)"`
		`+ gpg: searching for "Fedora (13)" from hkp server subkeys.pgp.net`
		`+ (1) Fedora (13) <fedora@fedoraproject.org>`
		`+ 4096 bit RSA key E8E40FDE, created: 2010-01-19`
		`+ ...`
		`+ ....`
		`+`
		`+ ==== Koji`
		`+`
		`+ Log into koji02.phx2.fedoraproject.org by way of`
		`+ bastion.fedoraproject.org.`
		`+`
		+ Verify that `/etc/koji-gc/koji-gc.conf` has the new key in it.
		`+`
		`+ === Consider Before Running`
		`+`
		`+ Nothing at this time.`

modules/release_guide/pages/sop_deprecate_ftbfs_packages.adoc

file added

+123

		`@@ -0,0 +1,123 @@`
		`+ == Deprecate FTBFS Packages`
		`+`
		`+ === Description`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ FTBFS = "Fails To Build From Source"`
		`+ ====`
		`+`
		`+ Every release prior to the Feature Freeze we deprecate all packages that`
		`+ https://fedoraproject.org/wiki/Fails_to_build_from_source[FTBFS]. This`
		`+ keeps out software that no longer builds from source, and prevents`
		`+ future problems down the road.`
		`+`
		`+ === Action`
		`+`
		`+ The FTBFS process takes place in stages:`
		`+`
		`+ . Detecting a list of FTBFS packages and the dependencies that will be`
		`+ broken if they are removed.`
		`+ . Sending the list of potential deprecated FTBFS packages to`
		`+ devel@lists.fedoraproject.org for community review and removal from the`
		`+ FTBFS list by fixing the package.`
		`+ . Removing packages confirmed as FTBFS from the Fedora package`
		`+ repositories.`
		`+`
		`+ ==== Detecting FTBFS`
		`+`
		`+ We will remove packages that have failed to build for at least two`
		`+ release cycles. For example, in preparation for Fedora 21 branching,`
		`+ packages which FTBFS since the Fedora 19 cycle (i.e. packages that have`
		`+ a dist tag of fc18 or earlier) will be considered candidates for`
		`+ removal. Adjust`
		`+ https://pagure.io/releng/blob/main/f/scripts/find_FTBFS.py[find_FTBFS.py]`
		`+ and run it to get a list of candidate packages.`
		`+`
		`+ Given a candidate list from above, rel-eng should attempt to build each`
		`+ of the candidate packages using koji. Should package building now`
		`+ succeed, the package may be removed from the candidate list.`
		`+`
		`+ ==== Announcing Packages to be Deprecated`
		`+`
		`+ Email the output to the development list`
		+ (`devel@lists.fedodraproject.org`) at least a week before the feature
		`+ freeze. This gives maintainers an opportunity to fix packages that are`
		`+ important to them. Follow-up on the list where necessary.`
		`+`
		`+ ==== Retiring FTBFS packages`
		`+`
		`+ Once maintainers have been given an opportunity to pick up and fix FTBFS`
		+ packages, the remaining packages are `retired` by blocking them, and
		+ creating the `dead.package` file in git.
		`+`
		`+ ===== GIT and Package DB`
		`+`
		`+ Required permissions: provenpackage for GIT, cvsadmin for Package DB.`
		`+`
		+ We just have to remove the existing files from the `rawhide` branch and
		+ replace them with a `dead.package` file whose contents describe why the
		`+ package is dead. Also the package needs to be marked as retired in`
		`+ PackageDB. Fedpkg takes care of this:`
		`+`
		`+ For example, if we wished to clean up git for the roxterm package we`
		`+ would:`
		`+`
		`+ ....`
		`+ $ fedpkg clone roxterm`
		`+ $ cd roxterm`
		`+ $ fedpkg retire "Retired on $(date -I), because it failed to build for two releases (FTBFS Cleanup)."`
		`+ ....`
		`+`
		`+ ===== Koji`
		`+`
		`+ Required permissions: admin in koji if the automatic blocking fails.`
		`+`
		`+ Blocking should happen automatically a few minutes after the packags was`
		+ retired in PackageDB. If it does not, use the `block-pkg` `koji` command
		`+ is used to do the blocking.`
		`+`
		`+ Koji accepts multiple package names as input and thus we can use the`
		`+ FTBFS package list as input. Deprecated packages are only blocked from`
		+ the latest `f##` tag. For example, if we wanted to `deprecate` (block)
		+ `sbackup, roxterm,` and `uisp` from rawhide during the development of
		`+ Fedora 21 we would run the following command:`
		`+`
		`+ ....`
		`+ $ koji block-pkg f21 sbackup roxterm uisp`
		`+ ....`
		`+`
		`+ ===== Bugs`
		`+`
		`+ This procedure probably leaves open bugs for the deprecated packages`
		`+ behind. It is not within the scope of releng to take care of these. If`
		`+ bugs are closed, only bugs targeted at Rawhide should be affected, since`
		`+ other branches might still be maintained.`
		`+`
		`+ === Verification`
		`+`
		`+ To verify that the packages were blocked correctly we can use the`
		+ `latest-pkg` `koji` action.
		`+`
		`+ ....`
		`+ $ koji latest-pkg dist-f16 wdm`
		`+ ....`
		`+`
		+ This should return nothing, as the `wdm` package is blocked.
		`+`
		`+ Also check that package DB shows that the package is retired and that`
		`+ the rawhide branch contains only a dead.package file.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ Generally we block anything that doesn't leave broken dependencies. If`
		`+ there are packages whose removal would result in broken dependencies a`
		`+ second warning should be sent to devel@lists.fedoraproject.org and to`
		`+ <package>-owner@fedoraproject.org for each dependent package.`
		`+`
		`+ Allow another couple of days for maintainers to take notice and fix`
		`+ these package so the package repository can be maintained without broken`
		`+ dependencies or needing to deprecate the package. It is not good to have`
		`+ broken package dependencies in our package repositories so every effort`
		`+ should be made to find owners or to fix the broken dependencies.`

modules/release_guide/pages/sop_end_of_life.adoc

file added

+541

		`@@ -0,0 +1,541 @@`
		`+ == End Of Life`
		`+`
		`+ === Description`
		`+`
		`+ Each release of Fedora is maintained as laid out in the`
		`+ https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule[maintenance`
		`+ schedule]. At the conclusion of the maintenance period, a Fedora release`
		+ enters `end of life` status. This procedure describes the tasks
		`+ necessary to move a release to that status.`
		`+`
		`+ === Actions`
		`+`
		`+ ==== Set date`
		`+`
		`+ * {blank}`
		`+ +`
		`+ Releng responsibilities:::`
		`+ ** Follow guidelines of`
		`+ https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule[maintenance`
		`+ schedule]`
		`+ ** Take into account any infrastructure or other supporting project`
		`+ resource contention`
		`+ ** Announce the closure of the release to the package maintainers.`
		`+`
		`+ ==== Reminder announcement`
		`+`
		`+ Send an email to devel@, devel-announce@, test-announce@, announce@`
		`+ lists as remainder about the release EOL.`
		`+`
		`+ ===== Announcement Content`
		`+`
		`+ ....`
		`+ Hello all,`
		`+`
		`+ Fedora 31 will go end of life for updates and support on 24th of`
		`+ November 2020. No further updates, including security updates, will be`
		`+ available for Fedora 31 after the said date. All the updates of Fedora`
		`+ 31 being pushed to stable will be stopped as well.`
		`+`
		`+ Fedora 32 will continue to receive updates until approximately one`
		`+ month after the release of Fedora 34. The maintenance schedule of`
		`+ Fedora releases is documented on the Fedora Project wiki [0]. The`
		`+ fedora Project wiki also contains instructions [1] on how to upgrade`
		`+ from a previous release of Fedora to a version receiving updates.`
		`+`
		`+ Regards,`
		`+ Mohan Boddu.`
		`+`
		`+ [0]https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule`
		`+ [1]https://fedoraproject.org/wiki/Upgrading?rd=DistributionUpgrades`
		`+ ....`
		`+`
		`+ ==== Koji tasks`
		`+`
		`+ * Disable builds by removing targets`
		`+`
		`+ ....`
		`+ $ koji remove-target f31`
		`+ $ koji remove-target f31-candidate`
		`+ $ koji remove-target f31-container-candidate`
		`+ $ koji remove-target f31-flatpak-candidate`
		`+ $ koji remove-target f31-infra`
		`+ $ koji remove-target f31-coreos-continuous`
		`+ $ koji remove-target f31-rebuild`
		`+ $ koji remote-target <side-targets> #any side targets that are still around`
		`+ ....`
		`+`
		`+ * Purge from disk the signed copies of rpms that are signed with the`
		`+ EOL'd release key. To acheive this, add the release key to`
		`+ koji_cleanup_signed.py script in https://pagure.io/releng[releng] repo`
		`+ and the script on compose-branched01.iad2.fedoraproject.org`
		`+`
		`+ ....`
		`+ ./scripts/koji_cleanup_signed.py`
		`+ ....`
		`+`
		`+ ==== PDC tasks`
		`+`
		`+ * Set PDC active value for the release to False`
		`+`
		`+ ....`
		`+ curl -u: -H 'Authorization: Token <token>' -H 'Accept: application/json' -H 'Content-Type:application/json' -X PATCH -d '{"active":"false"}' https://pdc.fedoraproject.org/rest_api/v1/releases/fedora-31/`
		`+ ....`
		`+`
		`+ * Set the EOL dates in PDC for all the components to the release EOL`
		`+ date if they are not already set. Run the following script from`
		`+ https://pagure.io/releng[releng] repo`
		`+`
		`+ ....`
		`+ python scripts/pdc/adjust-eol-all.py <token> f31 2020-11-24`
		`+ ....`
		`+`
		`+ ==== Bodhi tasks`
		`+`
		`+ * Run the following bodhi commands to set the releases state to`
		`+ archived`
		`+`
		`+ ....`
		`+ $ bodhi releases edit --name "F31" --state archived`
		`+ $ bodhi releases edit --name "F31M" --state archived`
		`+ $ bodhi releases edit --name "F31C" --state archived`
		`+ $ bodhi releases edit --name "F31F" --state archived`
		`+ ....`
		`+`
		`+ [WARNING]`
		`+ .Warning`
		`+ ====`
		`+ Due to a https://github.com/fedora-infra/bodhi/issues/2177[bug] in`
		`+ Bodhi, it is critical that Bodhi processes be restarted any time`
		+ `bodhi releases create` or `bodhi releases edit` are used.
		`+ ====`
		`+`
		`+ * On bodhi-backend01.iad2.fedoraproject.org, run the following commands`
		`+`
		`+ ....`
		`+ $ sudo systemctl restart fm-consumer@config.service`
		`+ $ sudo systemctl restart bodhi-celery.service`
		`+ ....`
		`+`
		`+ ==== Fedora Infra Ansible Changes`
		`+`
		`+ * We need to make changes to bodhi, koji, mbs, releng, autosign roles in`
		`+ ansible repo.`
		`+`
		`+ [source,diff]`
		`+ ----`
		`+ From 73dc8a1042a190f1b88bf78e110d44753cfa7962 Mon Sep 17 00:00:00 2001`
		`+ From: Mohan Boddu <mboddu@bhujji.com>`
		`+ Date: Nov 24 2020 17:19:23 +0000`
		`+ Subject: F31 EOL`
		`+`
		`+`
		`+ Signed-off-by: Mohan Boddu <mboddu@bhujji.com>`
		`+`
		`+ ---`
		`+`
		`+ diff --git a/roles/bodhi2/backend/files/new-updates-sync b/roles/bodhi2/backend/files/new-updates-sync`
		`+ index a143047..d8c8a73 100755`
		`+ --- a/roles/bodhi2/backend/files/new-updates-sync`
		`+ +++ b/roles/bodhi2/backend/files/new-updates-sync`
		`+ @@ -113,50 +113,6 @@ RELEASES = {'f33': {'topic': 'fedora',`
		`+ 'dest': os.path.join(FEDORAALTDEST, 'testing', '32', 'Modular')}`
		`+ ]}}`
		`+ },`
		`+ - 'f31': {'topic': 'fedora',`
		`+ - 'version': '31',`
		`+ - 'modules': ['fedora', 'fedora-secondary'],`
		`+ - 'repos': {'updates': {`
		`+ - 'from': 'f31-updates',`
		`+ - 'ostrees': [{'ref': 'fedora/31/%(arch)s/updates/silverblue',`
		`+ - 'dest': OSTREEDEST,`
		`+ - 'arches': ['x86_64', 'ppc64le', 'aarch64']}],`
		`+ - 'to': [{'arches': ['x86_64', 'armhfp', 'aarch64', 'source'],`
		`+ - 'dest': os.path.join(FEDORADEST, '31', 'Everything')},`
		`+ - {'arches': ['ppc64le', 's390x'],`
		`+ - 'dest': os.path.join(FEDORAALTDEST, '31', 'Everything')}`
		`+ - ]},`
		`+ - 'updates-testing': {`
		`+ - 'from': 'f31-updates-testing',`
		`+ - 'ostrees': [{'ref': 'fedora/31/%(arch)s/testing/silverblue',`
		`+ - 'dest': OSTREEDEST,`
		`+ - 'arches': ['x86_64', 'ppc64le', 'aarch64']}],`
		`+ - 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'],`
		`+ - 'dest': os.path.join(FEDORADEST, 'testing', '31', 'Everything')},`
		`+ - {'arches': ['ppc64le', 's390x'],`
		`+ - 'dest': os.path.join(FEDORAALTDEST, 'testing', '31', 'Everything')}`
		`+ - ]}}`
		`+ - },`
		`+ - 'f31m': {'topic': 'fedora',`
		`+ - 'version': '31m',`
		`+ - 'modules': ['fedora', 'fedora-secondary'],`
		`+ - 'repos': {'updates': {`
		`+ - 'from': 'f31-modular-updates',`
		`+ - 'ostrees': [],`
		`+ - 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'],`
		`+ - 'dest': os.path.join(FEDORADEST, '31', 'Modular')},`
		`+ - {'arches': ['ppc64le', 's390x'],`
		`+ - 'dest': os.path.join(FEDORAALTDEST, '31', 'Modular')}`
		`+ - ]},`
		`+ - 'updates-testing': {`
		`+ - 'from': 'f31-modular-updates-testing',`
		`+ - 'ostrees': [],`
		`+ - 'to': [{'arches': ['x86_64', 'aarch64', 'armhfp', 'source'],`
		`+ - 'dest': os.path.join(FEDORADEST, 'testing', '31', 'Modular')},`
		`+ - {'arches': ['ppc64le', 's390x'],`
		`+ - 'dest': os.path.join(FEDORAALTDEST, 'testing', '31', 'Modular')}`
		`+ - ]}}`
		`+ - },`
		`+ 'epel8': {'topic': 'epel',`
		`+ 'version': '8',`
		`+ 'modules': ['epel'],`
		`+ diff --git a/roles/bodhi2/backend/tasks/main.yml b/roles/bodhi2/backend/tasks/main.yml`
		`+ index a4b2a2b..d84f86a 100644`
		`+ --- a/roles/bodhi2/backend/tasks/main.yml`
		`+ +++ b/roles/bodhi2/backend/tasks/main.yml`
		`+ @@ -76,7 +76,7 @@`
		`+ # bodhi2/backend/files/koji_sync_listener.py`
		`+ # This cronjob runs only once a day. The listener script runs reactively.`
		`+ cron: name="owner-sync" minute="15" hour="4" user="root"`
		`+ - job="/usr/local/bin/lock-wrapper owner-sync '/usr/local/bin/owner-sync-pagure f34 f34-container f34-modular f33 f33-container f33-modular f33-flatpak f32 f32-container f32-modular f32-flatpak f31 f31-container f31-flatpak f31-modular epel8 epel8-playground epel8-modular epel7 dist-6E-epel module-package-list modular'"`
		`+ + job="/usr/local/bin/lock-wrapper owner-sync '/usr/local/bin/owner-sync-pagure f34 f34-container f34-modular f33 f33-container f33-modular f33-flatpak f32 f32-container f32-modular f32-flatpak epel8 epel8-playground epel8-modular epel7 dist-6E-epel module-package-list modular'"`
		`+ cron_file=update-koji-owner`
		`+ when: env == "production"`
		`+ tags:`
		`+ diff --git a/roles/bodhi2/backend/templates/koji_sync_listener.toml b/roles/bodhi2/backend/templates/koji_sync_listener.toml`
		`+ index 753adc0..41954ca 100644`
		`+ --- a/roles/bodhi2/backend/templates/koji_sync_listener.toml`
		`+ +++ b/roles/bodhi2/backend/templates/koji_sync_listener.toml`
		`+ @@ -48,10 +48,6 @@ taglist = [`
		`+ "f32-container",`
		`+ "f32-modular",`
		`+ "f32-flatpak",`
		`+ - "f31",`
		`+ - "f31-container",`
		`+ - "f31-flatpak",`
		`+ - "f31-modular",`
		`+ "epel8",`
		`+ "epel8-playground",`
		`+ "epel8-modular",`
		`+ diff --git a/roles/koji_hub/templates/hub.conf.j2 b/roles/koji_hub/templates/hub.conf.j2`
		`+ index 2f8b716..4816dba 100644`
		`+ --- a/roles/koji_hub/templates/hub.conf.j2`
		`+ +++ b/roles/koji_hub/templates/hub.conf.j2`
		`+ @@ -187,6 +187,5 @@ sidetag =`
		`+ tag f34-build :: allow`
		`+ tag f33-build :: allow`
		`+ tag f32-build :: allow`
		`+ - tag f31-build :: allow`
		`+ tag eln-build :: allow`
		`+ all :: deny`
		`+ diff --git a/roles/mbs/common/files/default-modules.production/platform-f31.yaml b/roles/mbs/common/files/default-modules.production/platform-f31.yaml`
		`+ deleted file mode 100644`
		`+ index 0608f93..0000000`
		`+ --- a/roles/mbs/common/files/default-modules.production/platform-f31.yaml`
		`+ +++ /dev/null`
		`+ @@ -1,28 +0,0 @@`
		`+ -data:`
		`+ - description: Fedora 31 traditional base`
		`+ - license:`
		`+ - module: [MIT]`
		`+ - name: platform`
		`+ - profiles:`
		`+ - buildroot:`
		`+ - rpms: [bash, bzip2, coreutils, cpio, diffutils, fedora-release, findutils, gawk,`
		`+ - glibc-minimal-langpack, grep, gzip, info, make, patch, redhat-rpm-config,`
		`+ - rpm-build, sed, shadow-utils, tar, unzip, util-linux, which, xz]`
		`+ - srpm-buildroot:`
		`+ - rpms: [bash, fedora-release, fedpkg-minimal, glibc-minimal-langpack, gnupg2,`
		`+ - redhat-rpm-config, rpm-build, shadow-utils]`
		`+ - stream: f31`
		`+ - summary: Fedora 31 traditional base`
		`+ - context: 00000000`
		`+ - version: 1`
		`+ - xmd:`
		`+ - mbs:`
		`+ - buildrequires: {}`
		`+ - commit: f31`
		`+ - requires: {}`
		`+ - koji_tag: module-f31-build`
		`+ - mse: TRUE`
		`+ - virtual_streams: [fedora]`
		`+ -document: modulemd`
		`+ -version: 1`
		`+ -`
		`+ diff --git a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json`
		`+ index d2f9a89..0eae499 100644`
		`+ --- a/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json`
		`+ +++ b/roles/pkgdb-proxy/files/pkgdb-gnome-software-collections.json`
		`+ @@ -41,7 +41,7 @@`
		`+ "dist_tag": ".fc31",`
		`+ "koji_name": "f31",`
		`+ "name": "Fedora",`
		`+ - "status": "Active",`
		`+ + "status": "EOL",`
		`+ "version": "31"`
		`+ },`
		`+ {`
		`+ diff --git a/roles/releng/files/cloud-updates b/roles/releng/files/cloud-updates`
		`+ index 0a37b49..ebb807c 100644`
		`+ --- a/roles/releng/files/cloud-updates`
		`+ +++ b/roles/releng/files/cloud-updates`
		`+ @@ -7,5 +7,5 @@ MAILTO=releng-cron@lists.fedoraproject.org`
		+ 15 7 * * * root TMPDIR=`mktemp -d /tmp/CloudF32.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f32 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+`
		`+ # Fedora 31 Cloud nightly compose`
		`+ -MAILTO=releng-cron@lists.fedoraproject.org`
		+ -15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+ +#MAILTO=releng-cron@lists.fedoraproject.org`
		+ +#15 8 * * * root TMPDIR=`mktemp -d /tmp/CloudF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./cloud-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+ diff --git a/roles/releng/files/container-updates b/roles/releng/files/container-updates`
		`+ index d3a0e14..6932bec 100644`
		`+ --- a/roles/releng/files/container-updates`
		`+ +++ b/roles/releng/files/container-updates`
		`+ @@ -1,6 +1,6 @@`
		`+ #Fedora 31 Container Updates nightly compose`
		`+ -MAILTO=releng-cron@lists.fedoraproject.org`
		+ -45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+ +#MAILTO=releng-cron@lists.fedoraproject.org`
		+ +#45 5 * * * root TMPDIR=`mktemp -d /tmp/containerF31.XXXXXX` && pushd $TMPDIR && git clone -n https://pagure.io/pungi-fedora.git && cd pungi-fedora && git checkout f31 && LANG=en_US.UTF-8 ./container-nightly.sh RC-$(date "+\%Y\%m\%d").0 && popd && rm -rf $TMPDIR
		`+`
		`+ # Fedora 33 Container Updates nightly compose`
		`+ MAILTO=releng-cron@lists.fedoraproject.org`
		`+ diff --git a/roles/robosignatory/templates/robosignatory.toml.j2 b/roles/robosignatory/templates/robosignatory.toml.j2`
		`+ index 41ab24c..60295c1 100644`
		`+ --- a/roles/robosignatory/templates/robosignatory.toml.j2`
		`+ +++ b/roles/robosignatory/templates/robosignatory.toml.j2`
		`+ @@ -92,30 +92,6 @@ handlers = ["console"]`
		`+`
		`+ # Temporary tags`
		`+`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f31-kde"`
		`+ - to = "f31-kde"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-31', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('3c3359c4', 'd300e724') }}"`
		`+ -`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f31-gnome"`
		`+ - to = "f31-gnome"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-31', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('3c3359c4', 'd300e724') }}"`
		`+ -`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f31-python"`
		`+ - to = "f31-python"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-31', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('3c3359c4', 'd300e724') }}"`
		`+ -`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f30-kde"`
		`+ - to = "f30-kde"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-30', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('cfc659b9', 'd300e724') }}"`
		`+ -`
		`+ # Infra tags`
		`+`
		`+ [[consumer_config.koji_instances.primary.tags]]`
		`+ @@ -143,12 +119,6 @@ handlers = ["console"]`
		`+ keyid = "{{ (env == 'production')\|ternary('47dd8ef9', 'd300e724') }}"`
		`+`
		`+ [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f31-infra-candidate"`
		`+ - to = "f31-infra-stg"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-infra', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('47dd8ef9', 'd300e724') }}"`
		`+ -`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ from = "f32-infra-candidate"`
		`+ to = "f32-infra-stg"`
		`+ key = "{{ (env == 'production')\|ternary('fedora-infra', 'testkey') }}"`
		`+ @@ -170,18 +140,6 @@ handlers = ["console"]`
		`+ # Gated coreos-pool tag`
		`+`
		`+ [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f30-coreos-signing-pending"`
		`+ - to = "coreos-pool"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-30', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('cfc659b9', 'd300e724') }}"`
		`+ -`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f31-coreos-signing-pending"`
		`+ - to = "coreos-pool"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-31', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('3c3359c4', 'd300e724') }}"`
		`+ -`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ from = "f32-coreos-signing-pending"`
		`+ to = "coreos-pool"`
		`+ key = "{{ (env == 'production')\|ternary('fedora-32', 'testkey') }}"`
		`+ @@ -297,19 +255,6 @@ handlers = ["console"]`
		`+ keyid = "{{ (env == 'production')\|ternary('12c944d0', 'd300e724') }}"`
		`+ type = "modular"`
		`+`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f31-signing-pending"`
		`+ - to = "f31-updates-testing-pending"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-31', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('3c3359c4', 'd300e724') }}"`
		`+ -`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f31-modular-signing-pending"`
		`+ - to = "f31-modular-updates-testing-pending"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-31', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('3c3359c4', 'd300e724') }}"`
		`+ - type = "modular"`
		`+ -`
		`+ #epel8 modular tags`
		`+ [[consumer_config.koji_instances.primary.tags]]`
		`+ from = "epel8-modular-signing-pending"`
		`+ @@ -417,12 +362,6 @@ handlers = ["console"]`
		`+ key = "{{ (env == 'production')\|ternary('fedora-32', 'testkey') }}"`
		`+ keyid = "{{ (env == 'production')\|ternary('12c944d0', 'd300e724') }}"`
		`+`
		`+ - [[consumer_config.koji_instances.primary.tags]]`
		`+ - from = "f31-openh264"`
		`+ - to = "f31-openh264"`
		`+ - key = "{{ (env == 'production')\|ternary('fedora-31', 'testkey') }}"`
		`+ - keyid = "{{ (env == 'production')\|ternary('3c3359c4', 'd300e724') }}"`
		`+ -`
		`+ # f33-rebuild`
		`+ [[consumer_config.koji_instances.primary.tags]]`
		`+ from = "f33-rebuild"`
		`+ ----`
		`+`
		`+ * Run the associated playbooks on _batcave_`
		`+`
		`+ ....`
		`+ $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/bodhi-backend.yml`
		`+ $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/koji-hub.yml`
		`+ $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/mbs.yml`
		`+ $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/releng-compose.yml`
		`+ $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies -t pkgdb2`
		`+ $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/manual/autosign.yml`
		`+ $ sudo ansible-playbook /srv/web/infra/ansible/playbooks/openshift-apps/bodhi.yml`
		`+ ....`
		`+`
		`+ ==== MBS Platform Retirement`
		`+`
		`+ * To retire the platform in mbs, run the following command on`
		`+ mbs-backend01.iad2.fedoraproject.org`
		`+`
		`+ ....`
		`+ $ sudo mbs-manager retire platform:f31`
		`+ ....`
		`+`
		`+ ==== Final announcement`
		`+`
		`+ * Send the final announcement to devel@, devel-announce@,`
		`+ test-announce@, announce@ lists`
		`+`
		`+ ===== Announcement content`
		`+`
		`+ ....`
		`+ Hello all,`
		`+`
		`+ As of the 24th of November 2020, Fedora 31 has reached its end of life`
		`+ for updates and support. No further updates, including security`
		`+ updates, will be available for Fedora 31. All the updates that are`
		`+ currently in testing won't get pushed to stable. Fedora 32 will`
		`+ continue to receive updates until approximately one month after the`
		`+ release of Fedora 34. The maintenance schedule of Fedora releases is`
		`+ documented on the Fedora Project wiki [0]. The Fedora Project wiki`
		`+ also contains instructions [1] on how to upgrade from a previous`
		`+ release of Fedora to a version receiving updates.`
		`+`
		`+ Mohan Boddu.`
		`+`
		`+ [0] https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule`
		`+ [1] https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/`
		`+ ....`
		`+`
		`+ ===== Update eol wiki page`
		`+`
		`+ https://fedoraproject.org/wiki/End_of_life update with release and`
		`+ number of days.`
		`+`
		`+ ===== Update FedoraPreviousPrevious.yaml in ansible repository`
		`+`
		`+ set the variable to False`
		`+`
		`+ ===== Move the EOL release to archive`
		`+`
		`+ . log into to bodhi-backend01 and become root`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh bodhi-backend01.iad2.fedoraproject.org`
		`+ $ sudo su`
		`+ $ su - ftpsync`
		`+ ....`
		`+ ____`
		`+ . then change into the releases directory.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd /pub/fedora/linux/releases`
		`+ ....`
		`+ ____`
		`+ . check to see that the target directory doesnt already exist.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ls /pub/archive/fedora/linux/releases/`
		`+ ....`
		`+ ____`
		`+ . do a recursive rsync to update any changes in the trees since the`
		`+ previous copy.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ rsync -avAXSHP ./35/ /pub/archive/fedora/linux/releases/35/`
		`+ ....`
		`+ ____`
		`+ . we now do the updates and updates/testing in similar ways.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd ../updates/`
		`+ $ rsync -avAXSHP 35/ /pub/archive/fedora/linux/updates/35/`
		`+ $ cd testing`
		`+ $ rsync -avAXSHP 35/ /pub/archive/fedora/linux/updates/testing/35/`
		`+ ....`
		`+ ____`
		`+ . do the same with fedora-secondary.`
		`+ . announce to the mirror list this has been done and that in 2 weeks you`
		`+ will move the old trees to archives.`
		`+ . in two weeks, log into mm-backend01 and run the archive script`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ sudo -u mirrormanager mm2_move-to-archive --originalCategory="Fedora Linux" --archiveCategory="Fedora Archive" --directoryRe='/35/Everything'`
		`+ ....`
		`+ ____`
		`+ . if there are problems, the postgres DB may have issues and so you need`
		`+ to get a DBA to update the backend to fix items.`
		`+ . wait an hour or so then you can remove the files from the main tree.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh bodhi-backend01`
		`+ $ cd /pub/fedora/linux`
		`+ $ cd releases/35`
		`+ $ ls # make sure you have stuff here`
		`+ $ rm -rf *`
		`+ $ ln ../20/README .`
		`+ $ cd ../../updates/35`
		`+ $ ls #make sure you have stuff here`
		`+ $ rm -rf *`
		`+ $ ln ../20/README .`
		`+ $ cd ../testing/35`
		`+ $ ls # make sure you have stuff here`
		`+ $ rm -rf *`
		`+ $ ln ../20/README .`
		`+ ....`
		`+ ____`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ * Resource contention in infrastructure, such as outages`
		`+ * Extenuating circumstances for specific planned updates, if any`
		`+ * Send the reminder announcement, if it isn't sent already`

modules/release_guide/pages/sop_eol_change.adoc

file added

+98

		`@@ -0,0 +1,98 @@`
		`+ == Adjust EOLs and SLs on branches`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ This SOP is about adjust EOLs and SLs on so-called "arbitrary" branches.`
		`+ Unretiring a package _also_ involves changing the EOL on a branch, but`
		`+ you won't find information on that here. See the unretirement SOP for`
		`+ more info there.`
		`+ ====`
		`+`
		`+ === Description`
		`+`
		`+ With "arbitrary branching", modules can include streams for an RPM that`
		`+ are not directly associated with a Fedora release. Modules _themselves_`
		`+ can have branches not directly associated with a Fedora release. For`
		`+ instance, our [.title-ref]#python3# module has a [.title-ref]#3.6#`
		`+ branch. The SLs on that module branch all go EOL on 2018-06-01.`
		`+ @torsava, one of the`
		`+ https://src.fedoraproject.org/modules/python3k[python3 module`
		`+ maintainers], may request that this branch have its EOL extended until`
		`+ 2018-12-01.`
		`+`
		`+ When a maintainer wants to change EOL on a rpm, module, or container`
		`+ branch, they need to file a https://pagure.io/releng/issues[releng`
		`+ ticket] requesting it. They have no way to do it on their own. Releng`
		`+ must review the request, and then process it.`
		`+`
		`+ === Policy`
		`+`
		`+ Here are some _policy_ guidelines to help you (releng) make decisions`
		`+ about these tickets`
		`+`
		`+ * Clarify. Does the maintainer want the EOL lengthed for an rpm? Or for`
		`+ a module? Or for a container? If they just say "increase the EOL for`
		`+ [.title-ref]#httpd#, please", it is not clear which thing they're really`
		`+ talking about.`
		`+ * Expect that maintainers generally know _when_ their EOL should go`
		`+ until. You don't need to go and research upstream mailing lists to`
		`+ figure out what makes sense. Politely asking the maintainer for some`
		`+ background information on _why_ the EOL should be changed is good to`
		`+ record in the ticket for posterity. Bonus points if they can provide`
		`+ references to upstream discussions that make the request make sense.`
		`+ * EOLs should _almost always_ only be extended into the future.`
		`+ Shortening an EOL should only be done with care. There might be modules`
		`+ out there that depend on an rpm branch with a conflicting EOL of their`
		`+ own. If a _shorter_ EOL is requested for an rpm branch, you should`
		`+ verify that no modules that depend on it have a conflicting EOL. If a`
		`+ _shorter_ EOL is requested for a module branch, you should verify that`
		`+ no other modules require it that have a conflicting EOL.`
		`+ * EOLs should not be arbitrary dates. At Flock 2017, we`
		`+ https://pagure.io/fedrepo_req/issue/100[decided on using two standard`
		`+ dates] for EOLs to help make things less crazy. You should use December`
		`+ 1st or June 1st of any given year for all EOL dates.`
		`+ * Many branches will _often_ have multiple SLs all with the _same_ EOL.`
		`+ I.e., the branch is fully supported until such time as it is totally`
		`+ retired. There is no gray area. However, it is _possible_ to have`
		`+ branches with piecemeal SLs and EOLs. A branch may support`
		`+ [.title-ref]#bug_fixes# until time X but will further support`
		`+ [.title-ref]#security_fixes# until time Y. This is nicely flexible for`
		`+ the maintainers, but also introduces complexity. If a maintainer`
		`+ requests piecemeal SL EOLs, ask to make sure they really want this kind`
		`+ of complexity.`
		`+`
		`+ === Action`
		`+`
		`+ We have a script in the releng repo:`
		`+`
		`+ ....`
		`+ $ PYTHONPATH=scripts/pdc python3 scripts/pdc/adjust-eol.py -h`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Run it with [.title-ref]#python3#. It imports [.title-ref]#fedrepo_req#`
		`+ which is python3 by default. Installing [.title-ref]#python2#`
		`+ dependencies should be possible when needed.`
		`+ ====`
		`+`
		`+ Here is an example of using it to increase the SL of the`
		`+ [.title-ref]#3.6# branch of the [.title-ref]#python3# module (not the`
		`+ rpm branch):`
		`+`
		`+ ....`
		`+ $ PYTHONPATH=scripts/pdc python3 scripts/pdc/adjust-eol.py \`
		`+ fedora \`
		`+ SOME_TOKEN \`
		`+ python3 \`
		`+ module \`
		`+ 3.6 \`
		`+ 2018-12-01`
		`+ Connecting to PDC args.server 'fedora' with token 'a9a1e4cbca122c21580d1fe4646e603a770c5280'`
		`+ Found 2 existing SLs in PDC.`
		`+ Adjust eol of (module)python3#3.6 security_fixes:2018-06-01 to 2018-12-01? [y/N]: y`
		`+ Adjust eol of (module)python3#3.6 bug_fixes:2018-06-01 to 2018-12-01? [y/N]: y`
		`+ Set eol to 2018-12-01 on ['(module)python3#3.6 security_fixes:2018-06-01', '(module)python3#3.6 bug_fixes:2018-06-01']`
		`+ ....`

modules/release_guide/pages/sop_fedora_media_writer.adoc

file added

+156

		`@@ -0,0 +1,156 @@`
		`+ == Fedora Media Writer Building and Signing`
		`+`
		`+ === Description`
		`+`
		`+ Whenever a new version of Fedora Media Writer is available, it is`
		`+ required to build and code sign it.`
		`+`
		`+ === Action`
		`+`
		`+ ==== Windows`
		`+`
		`+ {empty}#. Get a windows code signing key from private ansible`
		`+ repository. It is in the ansible-private/files/code-signing/ directory`
		`+`
		`+ . Convert the .cert formatted certificate to .pxf format:`
		`+ +`
		`+ ....`
		`+ $ openssl pkcs12 -export -in code-signing.crt -inkey code-signing.key -out authenticode.pfx`
		`+ ....`
		`+ . Clone the Fedora Media Writer git repo:`
		`+ +`
		`+ ....`
		`+ $ git clone https://github.com/MartinBriza/MediaWriter.git`
		`+ ....`
		`+ . Checkout the release tag for which the executable to be created:`
		`+ +`
		`+ ....`
		`+ $ git checkout tags/<release_number>`
		`+ ....`
		`+ . The script to build and sign the executable is available at`
		`+ dist/win/build.sh`
		`+ . Get the mingw-mediawriter from koji. Make sure the version is the one`
		`+ that needs building and signing.`
		`+ . Install the remaining packages that are listed under PACKAGES variable`
		`+ in build.sh script.`
		`+ . Export CERTPATH to the location where the .pfx file is located and`
		`+ make sure its named as authenticode.pfx and export CERTPASS to the file`
		`+ that contains the password used in creating .pvk file.`
		`+ . Run the build.sh script:`
		`+ +`
		`+ ....`
		`+ $ ./build.sh`
		`+ ....`
		`+`
		`+ === Verification`
		`+`
		`+ The FedoraMediaWriter-win32-<release_number>.exe is located under`
		`+ dist/win/ directory.`
		`+`
		`+ ==== OS X:`
		`+`
		`+ ==== Build:`
		`+`
		`+ . install xcode 8.1 from apple store.`
		`+ . {blank}`
		`+ +`
		`+ install qt for mac from:::`
		`+ http://download.qt.io/official_releases/qt/5.7/5.7.0/qt-opensource-mac-x64-clang-5.7.0.dmg`
		`+ . Open a terminal and run the following commands`
		`+ +`
		`+ ::::`
		`+ $ git clone https://github.com/MartinBriza/MediaWriter $ cd`
		`+ MediaWriter $ mkdir build $ cd build $`
		`+ $QT_PREFIX/$QT_VERSION/clang64/bin/qmake .. $ make $ cp`
		`+ build/helper/mac/helper.app/Contents/MacOS/helper`
		`+ build/app/FedoraMediaWriter.app/Contents/MacOS/ $ cd build/app $`
		`+ $QT_PREFIX/$QT_VERSION/clang_64/bin/macdeployqt "Fedora Media`
		`+ Writer.app" -executable="Fedora Media`
		`+ Writer.app/Contents/MacOS/helper" -qmldir="../../app"`
		`+`
		`+ ==== Prepare certificates`
		`+`
		`+ This only needs to happen once per build machine, and prepares the`
		`+ certificates by requesting them from Apple.`
		`+`
		`+ . Install Xcode from the Mac App store`
		`+ . Start Xcode`
		`+ . Press Command-, (or in the menu bar click Xcode -> Preferences)`
		`+ . Go to Accounts tab`
		`+ . Click the plus button and sign in`
		`+ . Select the new account`
		`+ . Select the correct team`
		`+ . Click View Details`
		`+ . Under "Signing Identities", find "Developer ID Application"`
		`+ . Click Create`
		`+ . Wait until the button disappears`
		`+ . Done`
		`+`
		`+ ==== Sign and DMG`
		`+`
		`+ . Open a terminal`
		`+ . cd to the root directory of the FMW project`
		`+ . Run the following bash script:`
		`+ +`
		`+ ____`
		`+ ....`
		`+ #/bin/bash`
		`+`
		`+ security -v unlock-keychain login.keychain`
		`+`
		`+ # First sign all dynamic libraries (dylib's)`
		`+ cd "build/app/Fedora Media Writer.app"`
		`+ for dylib in $(find . -name "*dylib")`
		`+ do`
		`+ codesign -s "Developer ID Application: Fedora Gilmore" -v $dylib`
		`+ done`
		`+ # Now sign framework bundles`
		`+ for framework in $(find . -name "*framework")`
		`+ do`
		`+ codesign -s "Developer ID Application: Fedora Gilmore" -v $framework`
		`+ done`
		`+`
		`+ # Sign the two binaries`
		`+ codesign -s "Developer ID Application: Fedora Gilmore" -v Contents/MacOS/helper`
		`+ codesign -s "Developer ID Application: Fedora Gilmore" -v "Contents/MacOS/Fedora Media Writer"`
		`+`
		`+ # Sign the app bundle`
		`+ codesign -s "Developer ID Application: Fedora Gilmore" -v .`
		`+`
		`+ # Create the dmg`
		`+ cd ..`
		`+ rm -f FedoraMediaWriter-osx-*.dmg`
		`+`
		`+ hdiutil create -srcfolder "Fedora Media Writer.app" -format UDCO -imagekey zlib-level=9 -scrub \`
		`+ -volname FedoraMediaWriter-osx FedoraMediaWriter-osx-$(git describe --tags).dmg`
		`+ ....`
		`+ ____`
		`+`
		`+ ==== Account Email(OS X)`
		`+`
		`+ ____`
		`+ ::::`
		`+ releng@fedoraproject.org`
		`+ ____`
		`+`
		`+ ==== Account Holders(OS X)`
		`+`
		`+ . Primary: Dennis Gilmore(ausil)`
		`+ . Backup: Kevin Fenzi(kevin)`
		`+ . Manager/bill-payer: Paul Frields(pfrields)`
		`+`
		`+ === Sync binaries to the web`
		`+`
		`+ copy both files to /srv/web/fmw on sundries01 create symlinks to the`
		`+ FedoraMediaWriter-win32-latest.exe and FedoraMediaWriter-osx-latest.dmg`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ Nothing yet.`
		`+`
		`+ === Issue with signing`
		`+`
		`+ If the build is done but it is not signed then try editing the`
		+ `build.sh` and add -askpass argument for all the osslsigncode commands
		`+ and run the script, when it asks for the password you can enter the`
		`+ password that was used in creating .pvk file.`

modules/release_guide/pages/sop_file_ftbfs.adoc

file added

+76

		`@@ -0,0 +1,76 @@`
		`+ == File FTBFS`
		`+`
		`+ === Description`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ FTBFS = "Fails To Build From Source"`
		`+ ====`
		`+`
		`+ After every mass rebuild, we file FTBFS bugs for the packages that`
		`+ failed to build during mass rebuild.`
		`+`
		`+ This should be run after the`
		`+ https://docs.pagure.org/releng/sop_mass_rebuild_packages.html#post-mass-rebuild-tasks[mass`
		`+ rebuild builds are merged into main tag].`
		`+`
		`+ === Action`
		`+`
		`+ The FTBFS bugs are filed in bugzilla.`
		`+`
		`+ . {blank}`
		`+ +`
		`+ Create a bugzilla bug for FTBFS::`
		`+ * use the https://bugzilla.redhat.com/show_bug.cgi?id=1750908[previous`
		`+ FTBFS bugzilla bug example] if its not created`
		`+ . {blank}`
		`+ +`
		`+ Set alias for RAWHIDEFTBFS::`
		`+ * remove RAWHIDEFTBFS alias from the previous FTBFS bugzilla`
		`+ * set RAWHIDEFTBFS alias on the new rawhide version FTBFS bugzilla`
		`+ * set the alias on RAWHIDEFailsToInstall bugzilla in same fashion`
		`+ . {blank}`
		`+ +`
		`+ Install [.title-ref]#python-bugzilla-cli# on your local machine if its`
		`+ not installed::`
		`+ ....`
		`+ $ sudo dnf install python-bugzilla-cli`
		`+ ....`
		`+ . {blank}`
		`+ +`
		`+ Update the [.title-ref]#massrebuildsinfo.py#::`
		`+ * epoch`
		`+ * buildtag`
		`+ * destag`
		`+ * tracking_bug`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Most of these values are already updated during mass rebuild, only one`
		`+ that might need updating is [.title-ref]#tracking_bug#`
		`+ ====`
		`+ . {blank}`
		`+ +`
		`+ Update the [.title-ref]#mass_rebuild_file_bugs.py#::`
		`+ * rebuildid`
		`+ . {blank}`
		`+ +`
		`+ Login into bugzilla in the terminal using [.title-ref]#bugzilla login#`
		`+ command::`
		`+ ....`
		`+ $ bugzilla login`
		`+ ....`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Login as [.title-ref]#releng@fedoraproject.org#`
		`+ ====`
		`+ . {blank}`
		`+ +`
		`+ Run [.title-ref]#mass_rebuild_file_bugs.py# locally::`
		`+ ....`
		`+ $ python mass_rebuild_file_bugs.py`
		`+ ....`

modules/release_guide/pages/sop_find_module_info.adoc

file added

+28

		`@@ -0,0 +1,28 @@`
		`+ == Finding Module Information`
		`+`
		`+ === Description`
		`+`
		`+ When users submit builds to the Module Build Service (MBS), it in turn`
		`+ submits builds to Koji. Sometimes, you are looking at a koji build, and`
		`+ you want to know what module-build it is a part of.`
		`+`
		`+ === Caveat`
		`+`
		`+ It requires that the build has been completed and has been tagged, until`
		`+ https://pagure.io/fm-orchestrator/issue/375 is complete.`
		`+`
		`+ === Setup`
		`+`
		`+ Run the following:`
		`+`
		`+ ....`
		`+ $ sudo dnf install python-arrow python-requests koji`
		`+ ....`
		`+`
		`+ === Action`
		`+`
		`+ Run the following:`
		`+`
		`+ ....`
		`+ $ scripts/mbs/koji-module-info.py $BUILD_ID`
		`+ ....`

modules/release_guide/pages/sop_generating_openh264_composes.adoc

file added

+185

		`@@ -0,0 +1,185 @@`
		`+ == Generating Openh264 Composes`
		`+`
		`+ === Description`
		`+`
		`+ Openh264 repos are a special case and we need to generate the composes`
		`+ for it in a different way. We use ODCS to generate the private compose`
		`+ and send the rpms to Cisco to publish them on their CDN. We publish the`
		`+ repodata on our side.`
		`+`
		`+ [WARNING]`
		`+ .Warning`
		`+ ====`
		`+ We do not have all the appropriate legal rights to distribute these`
		`+ packages, so we need to be extra carefull to make sure they are never`
		`+ distributed via our build system or websites`
		`+ ====`
		`+`
		`+ === Action`
		`+`
		`+ ==== Permissions needed`
		`+`
		`+ You will need some ODCS permissions in order to request private composes`
		`+ and composes from tags. You can set this in infra/ansible in`
		`+ inventory/group_vars/odcs in the odcs_allowed_clients_users variable.`
		`+ See other releng users entries for format.`
		`+`
		`+ ==== Get the odcs token`
		`+`
		`+ In order to generate an odcs compose, you need a openidc token.`
		`+`
		+ Run the odcs-token.py under `scripts/odcs/` from pagure releng
		`+ repository to generate the token.`
		`+`
		`+ ....`
		`+ $ ./odcs-token.py`
		`+ ....`
		`+`
		`+ ==== Make sure rpms are written out with the right signature`
		`+`
		`+ ....`
		`+ $ koji write-signed-rpm eb10b464 openh264-2.2.0-1.fc38`
		`+ ....`
		`+`
		`+ Where the key for that branch is listed, then the open264 package and`
		`+ version.`
		`+`
		`+ ==== Generate a private odcs compose`
		`+`
		`+ With the token generated above, generate the odcs private compose`
		`+`
		`+ ....`
		`+ $ python odcs-private-compose.py <token> <koji_tag> <signingkeyid>`
		`+ ....`
		`+`
		+ `koji_tag`: fxx-openh264 (Openh264 builds are tagged to fxx-openh264
		`+ tags where [.title-ref]#xx# represents the fedora release)`
		`+`
		+ `signingkeyid`: The short hash of the key for this Fedora branch.
		`+`
		+ The composes are stored under `/srv/odcs/private/` dir on
		+ `odcs-backend-releng01.iad2.fedoraproject.org`
		`+`
		`+ ==== Pull the compose to your local machine`
		`+`
		`+ We need to extract the rpms and tar them to send them to Cisco. In order`
		`+ to that, first of all we need to pull the compose to our local machine.`
		`+`
		`+ ===== Move the compose to your home dir on odcs-backend-releng01.iad2.fedoraproject.org`
		`+`
		`+ Since the compose is owned by [.title-ref]#odcs-server# pull it into`
		`+ your home dir`
		`+`
		`+ ....`
		`+ $ mkdir ~/32-openh264`
		`+ $ sudo rsync -avhHP /srv/odcs/private/odcs-3835/ ~/32-openh264/`
		`+ $ sudo chown -R mohanboddu:mohanboddu ~/32-openh264/`
		`+ ....`
		`+`
		`+ ===== Sync the compose to your local machine`
		`+`
		`+ Pull in the compose from your home dir on odcs releng backend to your`
		`+ local machine into a temp working dir`
		`+`
		`+ ....`
		`+ $ mkdir openh264-20200813`
		`+ $ scp -rv odcs-backend-releng01.iad2.fedoraproject.org:/home/fedora/mohanboddu/32-openh264/ openh264-20200813/`
		`+ ....`
		`+`
		`+ ===== Make the changes needed`
		`+`
		`+ Please follow the following commands to make the necessary tar files to`
		`+ send to Cisco`
		`+`
		`+ ....`
		`+ $ cd openh264-20200813`
		`+ $ mkdir 32-rpms`
		`+ # Copy rpms including devel rpms`
		`+ $ cp -rv 32-openh264/compose/Temporary/////*rpm 32-rpms/`
		`+ # Copy debuginfo rpms`
		`+ $ cp -rv 32-openh264/compose/Temporary//////rpm 32-rpms/`
		`+ # copy the src.rpm`
		`+ $ cp -rv 32-openh264/compose/Temporary/////*src.rpm 32-rpms/`
		`+ $ cd 32-rpms`
		`+ # Create the tar file with the rpms`
		`+ $ tar -cJvf ../fedora-32-openh264-rpms.tar.xz *rpm`
		`+ ....`
		`+`
		`+ We need to send this tar file to Cisco along with the list of rpms in`
		`+ each tarball.`
		`+`
		`+ ===== Syncing the compose to sundries01`
		`+`
		`+ Once we get a confirmation from Cisco that the rpms are updated on their`
		`+ CDN, verify them by using curl. For example:`
		`+`
		`+ ....`
		`+ $ curl -I http://ciscobinary.openh264.org/openh264-2.1.1-1.fc32.x86_64.rpm`
		`+ ....`
		`+`
		`+ Now push these composes to sundries01.iad2.fedoraproject.org and`
		`+ mm-backend01.iad2.fedoraproject.org`
		`+`
		`+ On sundries01 we need to sync to a directory that is owned by _apache_,`
		`+ so first we sync to the home directory on sundries01. Same with`
		`+ mm-backend01 as the directory is owned by _root_.`
		`+`
		`+ Create a temp working directory on sundries01`
		`+`
		`+ ....`
		`+ $ ssh sundries01.iad2.fedoraproject.org`
		`+ $ mkdir openh264-20200825`
		`+ ....`
		`+`
		`+ Create a temp working directory on mm-backend01`
		`+`
		`+ ....`
		`+ $ ssh mm-backend01.iad2.fedoraproject.org`
		`+ $ mkdir openh264-20200825`
		`+ ....`
		`+`
		`+ Then from your local machine, sync the compose`
		`+`
		`+ ....`
		`+ $ cd openh264-20200825`
		`+ $ rsync -avhHP 32-openh264 sundries01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825`
		`+ $ rsync -avhHP 32-openh264 mm-backend01.iad2.fedoraproject.org:/home/fedora/mohanboddu/openh264-20200825`
		`+ ....`
		`+`
		`+ On sundries01`
		`+`
		`+ ....`
		`+ $ cd openh264-20200825`
		`+ $ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/web/codecs.fedoraproject.org/openh264/32/`
		`+ ....`
		`+`
		`+ On mm-backend01`
		`+`
		`+ ....`
		`+ $ cd openh264-20200825`
		`+ $ sudo rsync -avhHP 32-openh264/compose/Temporary/ /srv/codecs.fedoraproject.org/openh264/32/`
		`+ ....`
		`+`
		`+ ===== Extra info`
		`+`
		`+ Normally that should be it, but in some cases you may want to push`
		`+ things out faster than normal, and here's a few things you can do to do`
		`+ that:`
		`+`
		`+ On mm-backend01.iad2.fedoraproject.org you can run:`
		`+`
		`+ ....`
		`+ # sudo -u mirrormanager /usr/local/bin/umdl-required codecs /var/log/mirrormanager/umdl-required.log`
		`+ ....`
		`+`
		`+ This will have mirrormanager scan the codecs dir and update it if it's`
		`+ changed.`
		`+`
		`+ On batcave01.iad2.fedoraproject.org you can use ansible to force all the`
		`+ proxies to sync the codec content from sundries01:`
		`+`
		`+ ....`
		`+ # nsible -a '/usr/bin/rsync --delete -a --no-owner --no-group sundries01::codecs.fedoraproject.org/ /srv/web/codecs.fedoraproject.org/' proxies`
		`+ ....`
		`+`
		`+ Mirrorlist servers should update every 15min.`

modules/release_guide/pages/sop_mass_branching.adoc

file added

+520

		`@@ -0,0 +1,520 @@`
		`+ == Mass Branching`
		`+`
		`+ === Description`
		`+`
		+ At each alpha freeze we branch the pending release away from `devel/`
		`+ which allows rawhide to move on while the pending release goes into`
		`+ bugfix and polish mode.`
		`+`
		`+ === Action`
		`+`
		`+ ==== Update fedscm-admin`
		`+`
		`+ Add the new release to fedscm-admin and create an update and send it to`
		`+ limb.`
		`+`
		`+ Please take a look at the`
		`+ https://pagure.io/fedscm-admin/c/7862d58b5982803dbe4c47e0262c6ce78bc903db?branch=main[fedscm-admin`
		`+ commit].`
		`+`
		`+ ==== Repos to branch`
		`+`
		`+ All the following listed repos needs updating, including adding a new`
		`+ branch for branched release and updating rawhide branch with new release`
		`+ values.`
		`+`
		`+ [arabic]`
		`+ . https://pagure.io/pungi-fedora`
		`+ . https://pagure.io/fedora-kickstarts`
		`+ . https://pagure.io/fedora-comps`
		`+ . https://pagure.io/fedora-lorax-templates/`
		`+ . https://pagure.io/releng/fedora-module-defaults/`
		`+ . https://pagure.io/workstation-ostree-config/`
		`+ . https://src.fedoraproject.org/rpms/fedora-release`
		`+ . https://src.fedoraproject.org/rpms/fedora-repos`
		`+`
		`+ ==== PDC`
		`+`
		`+ The "product-release" needs to be created in PDC.`
		`+`
		+ In the `scripts/pdc/` directory, run:
		`+`
		`+ ....`
		`+ $ python create-product-release.py fedora $TOKEN Fedora $NEW_VERSION`
		`+ ....`
		`+`
		+ On `pdc-backend01.stg` (for testing) or `pdc-backend01` (for production)
		`+ clone (or update an existing one) the releng repo:`
		`+`
		`+ ....`
		`+ git clone https://pagure.io/releng.git`
		`+ ....`
		`+`
		+ In the `scripts/pdc/` directory, run (see the `--help` of the script for
		`+ information on how to run it):`
		`+`
		`+ ....`
		`+ $ python create-new-release-branches.py ... --createfile`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		+ the `--createfile` argument is necessary, that file is needed for the
		`+ next step.`
		`+ ====`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Due to memory leak issue in pdc, we need to set the config in`
		`+ /etc/pdc.d/fedora.json \{ "fedora": \{ "host":`
		`+ "http://pdc-web02.iad2.fedoraproject.org/rest_api/v1/", "develop":`
		`+ false, "ssl-verify": false, } }`
		`+ ====`
		`+`
		`+ ==== dist-git`
		`+`
		`+ Now that pdc has the new release and each package has been branched in`
		`+ pdc we need to update dist-git in two steps:`
		`+`
		`+ * Create the new branch in git`
		`+ * Update the gitolite.conf to allow user to push to this new branch`
		`+`
		`+ For both of these actions you will need the file generated by pdc above.`
		`+`
		`+ ===== Create the git branches`
		`+`
		+ On `pkgs01.stg` (for testing) or `pkgs02` (for production), run:
		`+`
		`+ ....`
		`+ $ sudo -u pagure python /usr/local/bin/mass-branching-git.py <new branch name> <input file>`
		`+ ....`
		`+`
		+ Where `<new branch name>` will be something like `f28` and the
		+ `<input file>` the path to the file generated by pdc above.
		`+`
		`+ ==== Ansible`
		`+`
		`+ Apps in ansible need to be updated to be aware of a new branch.`
		`+`
		`+ ===== Bodhi`
		`+`
		`+ Bodhi needs to be updated to add new release. This needs to be done in`
		`+ https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2[bodhi2`
		`+ role] in infra ansible repo. This change includes, updating`
		`+ koji-sync-listener.py, new-updates-sync, pungi configs for both rpm and`
		`+ modular updates, bodhi templates.`
		`+`
		`+`
		`+ ===== Enable Branched Compose`
		`+`
		`+ We need to enable the branched compose. This is done in`
		`+ https://pagure.io/fedora-infra/ansible/blob/main/f/roles/releng[releng`
		`+ role] of infra ansbile repo`
		`+`
		`+`
		`+ ===== Fedora Branched`
		`+`
		`+ Set FedoraBranched variable to True in infra ansible repo`
		`+`
		`+`
		`+ ===== Koji hub`
		`+`
		`+`
		`+ ===== Robosignatory`
		`+`
		`+ Robosignatory has two parts:`
		`+`
		`+ [arabic]`
		`+ . Disable branched signing, so that we can freeze branched until we get`
		`+ a compose`
		`+ . Adding new release`
		`+`
		`+ Both can be in`
		`+ https://pagure.io/fedora-infra/ansible/blob/main/f/roles/robosignatory[robosignatory`
		`+ role] in infra ansible repo`
		`+`
		`+`
		`+ ===== Push the changes`
		`+`
		`+ When done editing the files, commit, push and apply them via the`
		`+ corresponding ansible playbook:`
		`+`
		`+ ....`
		`+ sudo rbac-playbook groups/koji-hub.yml`
		`+ sudo rbac-playbook groups/releng-compose.yml`
		`+ sudo rbac-playbook groups/bodhi-backend.yml`
		`+ sudo rbac-playbook openshift-apps/greenwave.yml`
		`+ sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t pkgdb2`
		`+ sudo rbac-playbook groups/mbs.yml -t mbs`
		`+ ....`
		`+`
		`+ Ask someone in fedora infra to run the robosignatory playbook.`
		`+`
		`+ ==== Koji`
		`+`
		`+ The koji build system needs to have some tag/target work done to handle`
		`+ builds from the new branch and to update where builds from rawhide go.`
		`+`
		`+ Run`
		`+ https://pagure.io/releng/blob/main/f/scripts/branching/make-koji-release-tags[make-koji-release-tags]`
		`+ script in https://pagure.io/releng[pagure releng] repo`
		`+`
		`+ ==== Fedora Release`
		`+`
		+ The `fedora-release` package needs to be updated in Rawhide and
		`+ Branched.`
		`+`
		+ Changes to `fedora-release.spec` in the rawhide branch:
		`+`
		`+ [arabic]`
		+ . Increment `%define dist_version`:
		`+ +`
		`+ ....`
		`+ -%define dist_version 35`
		`+ +%define dist_version 36`
		`+ ....`
		+ . Increment `Version:` and reset `Release:`:
		`+ +`
		`+ ....`
		`+ -Version: 35`
		`+ -Release: 0.3%{?eln:.eln%{eln}}`
		`+ +Version: 36`
		`+ +Release: 0.1%{?eln:.eln%{eln}}`
		`+ ....`
		+ . Add a `%changelog` entry:
		`+ +`
		`+ ....`
		`+ %changelog`
		`+ +* Tue Feb 23 2021 Mohan Boddu <mboddu@bhujji.com> - 36-0.1`
		`+ +- Setup for rawhide being F36`
		`+ ....`
		`+`
		+ Changes to `fedora-release.spec` in the branched branch:
		`+`
		`+ [arabic]`
		+ . Adjust `release_name` and unset `is_rawhide`:
		`+ +`
		`+ ....`
		`+ -%define release_name Rawhide`
		`+ -%define is_rawhide 1`
		`+ +%define release_name Thirty Five`
		`+ +%define is_rawhide 0`
		`+ ....`
		+ . Verify the correct number for `dist_version` and `Version:`:
		`+ +`
		`+ ....`
		`+ %define dist_version 35`
		`+ Version: 35`
		`+ ....`
		+ . Bump `Release:`:
		`+ +`
		`+ ....`
		`+ -Release: 0.3%{?eln:.eln%{eln}}`
		`+ +Release: 0.4%{?eln:.eln%{eln}}`
		`+ ....`
		+ . Add a `%changelog` entry:
		`+ +`
		`+ ....`
		`+ %changelog`
		`+ +* Tue Feb 23 2021 Mohan Boddu <mboddu@bhujji.com> - 35-0.4`
		`+ +- Branching F35 from rawhide`
		`+ ....`
		`+`
		`+ ==== Fedora Repos`
		`+`
		+ The `fedora-repos` package needs to be updated in Rawhide, Branched, and
		`+ also in all stable release branches (in order to receive new GPG keys`
		`+ and updated symlinks).`
		`+`
		+ Changes to the rawhide branch (mostly in `fedora-repos.spec`):
		`+`
		`+ [arabic]`
		`+ . Generate and add a _Rawhide+1_ GPG key file, then add it to the spec`
		`+ file:`
		`+ +`
		`+ ....`
		`+ Source57: RPM-GPG-KEY-fedora-37-primary`
		`+ ....`
		+ . Update the `archmap` file and define architectures for _Rawhide+1_:
		`+ +`
		`+ ....`
		`+ +fedora-37-primary: x86_64 armhfp aarch64 ppc64le s390x`
		`+ ....`
		+ . Increment `%global rawhide_release`:
		`+ +`
		`+ ....`
		`+ -%global rawhide_release 35`
		`+ +%global rawhide_release 36`
		`+ ....`
		+ . Bump `Version:` and reset `Release:`:
		`+ +`
		`+ ....`
		`+ -Version: 35`
		`+ -Release: 0.2%{?eln:.eln%{eln}}`
		`+ +Version: 36`
		`+ +Release: 0.1%{?eln:.eln%{eln}}`
		`+ ....`
		+ . Add a `%changelog` entry:
		`+ +`
		`+ ....`
		`+ %changelog`
		`+ +* Tue Feb 23 2021 Tomas Hrcka <thrcka@redhat.com> - 36-0.1`
		`+ +- Setup for rawhide being F36`
		`+ ....`
		`+`
		+ Changes to the branched branch (mostly in `fedora-repos.spec`):
		`+`
		`+ [arabic]`
		`+ . Copy the _Rawhide+1_ GPG key file from the _rawhide_ branch, then add`
		`+ it to the spec file:`
		`+ +`
		`+ ....`
		`+ Source57: RPM-GPG-KEY-fedora-37-primary`
		`+ ....`
		+ . Copy the `archmap` file from the _rawhide_ branch.
		+ . Update `%global rawhide_release`:
		`+ +`
		`+ ....`
		`+ -%global rawhide_release 35`
		`+ +%global rawhide_release 36`
		`+ ....`
		+ . Enable `updates_testing_enabled`:
		`+ +`
		`+ ....`
		`+ -%global updates_testing_enabled 0`
		`+ +%global updates_testing_enabled 1`
		`+ ....`
		+ . Bump `Release:`:
		`+ +`
		`+ ....`
		`+ -Release: 0.2%{?eln:.eln%{eln}}`
		`+ +Release: 0.3%{?eln:.eln%{eln}}`
		`+ ....`
		+ . Add a `%changelog` entry:
		`+ +`
		`+ ....`
		`+ %changelog`
		`+ +* Tue Feb 23 2021 Tomas Hrcka <thrcka@redhat.com> - 35-0.3`
		`+ +- Update Rawhide definition, enable updates-testing for Branched`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		+ Build `fedora-release` and `fedora-repos` packages for Branched release
		`+ before enabling the Rawhide gating.`
		`+ ====`
		`+`
		+ Changes to the stable branches (mostly in `fedora-repos.spec`):
		`+`
		`+ [arabic]`
		`+ . Copy the _Rawhide+1_ GPG key file from the _rawhide_ branch, then add`
		`+ it to the spec file:`
		`+ +`
		`+ ....`
		`+ Source57: RPM-GPG-KEY-fedora-37-primary`
		`+ ....`
		+ . Copy the `archmap` file from the _rawhide_ branch.
		+ . Update `%global rawhide_release`:
		`+ +`
		`+ ....`
		`+ -%global rawhide_release 35`
		`+ +%global rawhide_release 36`
		`+ ....`
		+ . Bump `Release:`:
		`+ +`
		`+ ....`
		`+ -Release: 0.2%{?eln:.eln%{eln}}`
		`+ +Release: 0.3%{?eln:.eln%{eln}}`
		`+ ....`
		+ . Add a `%changelog` entry:
		`+ +`
		`+ ....`
		`+ %changelog`
		`+ +* Tue Feb 23 2021 Tomas Hrcka <thrcka@redhat.com> - 34-0.3`
		`+ +- Update Rawhide definition`
		`+ ....`
		`+`
		`+ ==== Bodhi`
		`+`
		`+ ===== Linking Empty Repos`
		`+`
		`+ We need to link empty repos so that new-updates-sync wont complain about`
		`+ missing repos. The following commands should be run on`
		`+ bodhi-backend01.phx2.fedoraproject.org`
		`+`
		`+ ....`
		`+ $ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-updates`
		`+ $ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-updates-testing`
		`+ $ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-modular-updates`
		`+ $ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f32-modular-updates-testing`
		`+ ....`
		`+`
		`+ ===== Creating Empty Repos`
		`+`
		`+ To create empty repos on the master mirror, run`
		`+ https://pagure.io/releng/blob/main/f/scripts/branching/create_empty_repos.sh[create_emtpy_repos.sh]`
		`+ from https://pagure.io/releng[pagure releng] repo. This should be run on`
		`+ bodhi-backend01.phx2.fedoraproject.org`
		`+`
		`+ ....`
		`+ $ sudo -u ftpsync sh scripts/branching/create_empty_repos.sh 31`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Please verify the repo permissions that are created under`
		`+ /pub/fedora/linux/development/<fedora_release_number> and`
		`+ /pub/fedora-secondary/development/<fedora_release_number>. They should`
		`+ be owned by _ftpsync:ftpsync_`
		`+ ====`
		`+`
		`+ ===== Creating rawhide release`
		`+`
		`+ To create a rawhide release in bodhi, you need to run`
		`+`
		`+ ....`
		`+ $ bodhi releases create --name "F36" --long-name "Fedora 36" --id-prefix FEDORA --version 36 --branch f36 --dist-tag f36 --stable-tag f36 --testing-tag f36-updates-testing --candidate-tag f36-updates-candidate --pending-stable-tag f36-updates-pending --pending-testing-tag f36-updates-testing-pending --pending-signing-tag f36-signing-pending --state pending --override-tag f36-override --create-automatic-updates --not-composed-by-bodhi`
		`+ ....`
		`+`
		`+ To create a container release for rawhide in bodhi, you need to run`
		`+`
		`+ ....`
		`+ $ bodhi releases create --name "F36C" --long-name "Fedora 36 Containers" --id-prefix FEDORA-CONTAINER --version 36 --branch f36 --dist-tag f36-container --stable-tag f36-container-updates --testing-tag f36-container-updates-testing --candidate-tag f36-container-updates-candidate --pending-stable-tag f36-container-updates-pending --pending-testing-tag f36-container-updates-testing-pending --state pending --override-tag f36-container-override`
		`+ ....`
		`+`
		`+ To create a flatpak release for branched in bodhi, you need to run`
		`+`
		`+ ....`
		`+ $ bodhi releases create --name "F35F" --long-name "Fedora 35 Flatpaks" --id-prefix FEDORA-FLATPAK --version 35 --branch f35 --dist-tag f35-flatpak --stable-tag f35-flatpak-updates --testing-tag f35-flatpak-updates-testing --candidate-tag f35-flatpak-updates-candidate --pending-stable-tag f35-flatpak-updates-pending --pending-testing-tag f35-flatpak-updates-testing-pending --state pending --override-tag f35-flatpak-override`
		`+ ....`
		`+`
		+ You need to run the `bodhi openshift` playbook, so that UI will know
		`+ about the new release. Then, you need to restart`
		`+ fm-consumer@config.service and bodhi-celery.service services on`
		`+ bodhi-backend01.phx2.fedoraproject.org`
		`+`
		`+ ....`
		`+ $ sudo rbac-playbook openshift-apps/bodhi.yml`
		`+ $ sudo systemctl restart fm-consumer@config.service bodhi-celery.service`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Build fedora-release, fedora-repos package for *rawhide after enabling`
		`+ the rawhide gating*`
		`+ ====`
		`+`
		`+ ===== Update rawhide koji repo`
		`+`
		`+ We need to point the _rawhide_ buildroot repo to the newly created`
		`+ rawhide buildroot. This way kojira doesn't make a newrepo for _rawhide_`
		`+ target as often as fxx-build (new rawhide buildroot).`
		`+`
		`+ Run the following command from any of the compose boxes`
		`+`
		`+ ....`
		`+ $ cd /mnt/koji/repos/rawhide; rm -f latest; ln -s ../f34-build/latest`
		`+ ./latest`
		`+ ....`
		`+`
		`+ ===== Update block_retired.py script`
		`+`
		`+ https://pagure.io/releng/blob/main/f/scripts/block_retired.py[block_retired.py]`
		`+ script in releng repo should be updated with rawhide release and also`
		`+ branched release should be added to the script.`
		`+`
		`+ Please look at this`
		`+ https://pagure.io/releng/c/9eb97f491f7a767ab8b90498adfa3b34ee235247?branch=main[block_retired.py`
		`+ commit] as an example.`
		`+`
		`+ ===== Updating MirrorManager`
		`+`
		`+ We need to update the mirrormanager so that it will point rawhide to the`
		`+ new rawhide release.`
		`+`
		`+ Please follow the instructions in the`
		`+ https://pagure.io/fedora-infrastructure/issue/9239#comment-671446[fedora`
		`+ infra ticket] to update the database of mirrormanager.`
		`+`
		`+ ===== Enable autosigning on branched release`
		`+`
		`+ Once the branched compose is composed, we need to re-enable`
		`+ robosignatory on branched release`
		`+`
		`+ ===== ELN related work`
		`+`
		`+ Add the new rawhide key to eln pungi config. For example, look at this`
		`+ https://pagure.io/pungi-fedora/c/e993441164ee83374df7f463777f2bf1d456fd6d?branch=eln[pungi`
		`+ eln config commit]`
		`+`
		`+ Change the trigger notification for DistroBuildSync to the new Rawhide`
		`+ version. For example, look at this`
		`+ https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync-config/-/commit/1497d9aea42cf00af646b4a0f9f9ed1a7f0a477f[commit].`
		`+`
		`+ ===== Branch new rawhide in Koschei`
		`+`
		`+ Branch new fedora rawhide in`
		`+ https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koschei/#_branching_a_new_fedora_release[koschei].`
		`+`
		`+ ==== Fedora Container Base Image`
		`+`
		`+ In order to enable builds for Container Base Images via the`
		`+ https://docs.pagure.org/releng/layered_image_build_service.html[Fedora`
		`+ Layered Image Build System] we will need to import a new image for`
		+ Rawhide as well as for the new `fedora:rawhide` and `fedora:${RAWHIDE}`
		`+ tags.`
		`+`
		`+ Check for the latest successful Rawhide Base Image composed image`
		`+ https://koji.fedoraproject.org/koji/packageinfo?packageID=21546[here].`
		`+`
		+ On `compose-x86-01.phx2` run:
		`+`
		`+ ....`
		`+ # Update this to be the correct URL for your image`
		`+ $ BASEIMAGE_URL="https://kojipkgs.fedoraproject.org//packages/Fedora-Docker-Base/Rawhide/20170310.n.0/images/Fedora-Docker-Base-Rawhide-20170310.n.0.x86_64.tar.xz"`
		`+`
		`+ # Update this to whatever version number Rawhide now points to`
		`+ $ RAWHIDE="27"`
		`+`
		`+ # Load the latest, find it's image name`
		`+ $ sudo docker load < <(curl -s "${BASEIMAGE_URL}")`
		`+ $ sudo docker images \| grep base-rawhide`
		`+ fedora-docker-base-rawhide-20170310.n.0.x86_64 latest ffd832a990ca 5 hours ago 201.8 MB`
		`+`
		`+ # Tag everything`
		`+ $ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 candidate-registry.fedoraproject.org/fedora:rawhide`
		`+ $ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 candidate-registry.fedoraproject.org/fedora:${RAWHIDE}`
		`+ $ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 registry.fedoraproject.org/fedora:rawhide`
		`+ $ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 registry.fedoraproject.org/fedora:${RAWHIDE}`
		`+`
		`+ # Push the images`
		`+ $ sudo docker push candidate-registry.fedoraproject.org/fedora:rawhide`
		`+ $ sudo docker push candidate-registry.fedoraproject.org/fedora:${RAWHIDE}`
		`+ $ sudo docker push registry.fedoraproject.org/fedora:rawhide`
		`+ $ sudo docker push registry.fedoraproject.org/fedora:${RAWHIDE}`
		`+`
		`+ # Clean up after ourselves`
		`+ $ sudo docker rmi fedora-docker-base-rawhide-20170310.n.0.x86_64`
		`+ Untagged: fedora-docker-base-rawhide-20170310.n.0.x86_64:latest`
		`+ $ for i in $(sudo docker images -q -f 'dangling=true'); do sudo docker rmi $i; done`
		`+ ....`
		`+`
		`+ ===== Update sync script`
		`+`
		`+ In releng repository update`
		`+ https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh#_38[script].`
		`+`
		`+ And set current_rawhide variable.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ FIXME: Need some love here`
		`+ ====`

modules/release_guide/pages/sop_mass_rebuild_modules.adoc

file added

+122

		`@@ -0,0 +1,122 @@`
		`+ == Mass Rebuild of Modules`
		`+`
		`+ === Description`
		`+`
		`+ Periodically we do mass rebuilds of modules in rawhide during the`
		`+ development cycle. This SOP will outline the steps necessary to do this.`
		`+`
		`+ === Assumptions`
		`+`
		`+ This assumes that the mass rebuild has already been approved and`
		`+ scheduled via release engineering and FESCo. Coordinate with`
		`+ infrastructure as well for any needed updates.`
		`+`
		`+ === Considerations`
		`+`
		`+ * The most important thing to keep in mind while doing a mass rebuild is`
		`+ to communicate clearly what actions are being performed and the status`
		`+ of the rebuild.`
		`+ * Check in on scripts frequently to avoid a long stalled command from`
		`+ adding significant delays in completing the rebuild.`
		`+`
		`+ === Actions`
		`+`
		`+ ==== Preparatory Steps`
		`+`
		`+ The following steps should be completed after the`
		`+ https://docs.pagure.org/releng/sop_mass_rebuild_packages.html[mass`
		`+ rebuild of packages] is done.`
		`+`
		`+ . Update Scripts`
		`+`
		`+ The mass rebuild depends on two main scripts from the`
		`+ https://pagure.io/releng[releng git repository]. Each one requires some`
		`+ changes in variables for each new mass rebuild cycle.`
		`+`
		`+ ____`
		`+ * {blank}`
		`+ +`
		`+ _mass-rebuild-modules.py_::`
		`+ ** rebuildid`
		`+ * {blank}`
		`+ +`
		`+ _massrebuildsinfo.py_::`
		`+ ** module_mass_rebuild_epoch`
		`+ ** module_mass_rebuild_platform`
		`+ ____`
		`+`
		`+ Change the following items:`
		`+`
		+ * the `rebuildid` to match the release for which you are mass rebuilding
		`+ modules as per in massrebuildsinfo.py`
		+ * `module_mass_rebuild_epoch` mostly will be the epoch of mass rebuild
		`+ of packages`
		+ * `module_mass_rebuild_platform` should be the rawhide module platform
		`+`
		`+ ==== Starting the Mass Rebuild of Modules`
		`+`
		+ The `mass-rebuild-modules.py` script takes care of:
		`+`
		`+ * Discovering available available modules from PDC`
		`+ * Find the module info from mbs and check if a module build is submitted`
		`+ after the epoch date`
		`+ * Checking out modules from dist-git`
		`+ * Switching to appropriate stream`
		`+ * Find modulemd file`
		`+ * Use libmodulemd to determine if this module stream applies to this`
		`+ platform version`
		`+ * If needs rebuilding, committing the change`
		`+ * Push the commit`
		`+ * Submitting the build request through mbs`
		`+`
		`+ . Connect to the mass-rebuild Machine`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh compose-branched01.iad2.fedoraproject.org`
		`+ ....`
		`+ ____`
		`+ . Start a terminal multiplexer`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ tmux`
		`+ ....`
		`+ ____`
		`+ . Clone or checkout the latest copy of the`
		`+ https://pagure.io/releng[releng git repository].`
		`+ . Run the [.title-ref]#mass-rebuild-modules.py# script from`
		`+ _releng/scripts_`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd path/to/releng_repo/scripts`
		`+ $ ./mass-rebuild-modules.py <path_to_token_file> build --wait 2>&1 \| tee ~/massbuildmodules.out`
		`+ ....`
		`+ ____`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The token file should be located in infra's private ansible repo, or ask`
		`+ infra to get it to you using this`
		`+ https://pagure.io/fedora-infrastructure/issue/8048#comment-587789[process].`
		`+ ====`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The [.title-ref]#build# option is really important to pay attention,`
		`+ since the mass branching of modules will also use the same script, just`
		`+ changing the option to [.title-ref]#branch# and`
		`+ [.title-ref]#module_mass_branching_platform# in`
		`+ [.title-ref]#massrebuildsinfo.py#`
		`+ ====`
		`+`
		`+ ==== Post Mass Rebuild Tasks`
		`+`
		`+ Once the module mass rebuild is done, send an email to the`
		`+ devel-announce@ list`
		`+`
		`+ . Send the final notification to the`
		`+ _devel-announce@lists.fedoraproject.org_ list`

modules/release_guide/pages/sop_mass_rebuild_packages.adoc

file added

+262

		`@@ -0,0 +1,262 @@`
		`+ == Mass Rebuild`
		`+`
		`+ === Description`
		`+`
		`+ Periodically we do mass rebuilds of rawhide during the development`
		`+ cycle. This SOP will outline the steps necessary to do this.`
		`+`
		`+ === Assumptions`
		`+`
		`+ This assumes that the mass rebuild has already been approved and`
		`+ scheduled via release engineering and FESCo. Coordinate with`
		`+ infrastructure as well for any needed koji updates.`
		`+`
		`+ This also assumes that the mass rebuild does not need to be done in`
		`+ dependency order, and that the mass rebuild does not involve a ABI`
		`+ change.`
		`+`
		`+ === Considerations`
		`+`
		`+ * The most important thing to keep in mind while doing a mass rebuild is`
		`+ to communicate clearly what actions are being performed and the status`
		`+ of the rebuild.`
		`+ * Check in on scripts frequently to avoid a long stalled command from`
		`+ adding significant delays in completing the rebuild.`
		`+ * Check with secondary arches, whether they up-to-date enough with`
		`+ primary, create rebuild tag and target when they are. It will then take`
		`+ care of rebuilds of the arch specific packages in appropriate kojis.`
		`+`
		`+ === Actions`
		`+`
		`+ ==== Preparatory Steps`
		`+`
		`+ The following steps may be completed in the weeks leading up to the`
		`+ scheduled mass rebuild.`
		`+`
		`+ . Create the Mass Rebuild Pagure Issue`
		`+ +`
		`+ ____`
		`+ Create an issue on the https://pagure.io/releng/issues[Release`
		`+ Engineering issues page] that points at the schedule for the current`
		`+ release.`
		`+`
		`+ See https://pagure.io/releng/issue/6898[the Fedora 27 mass rebuild issue`
		`+ example].`
		`+ ____`
		`+ . Set up the Mass Rebuild Wiki Page`
		`+ +`
		`+ ____`
		`+ The mass rebuild wiki page should answer the following questions for`
		`+ maintainers:`
		`+`
		`+ * Why the mass rebuild is happening`
		`+ * How to opt out of the mass rebuild`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ See https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild[the Fedora 26`
		`+ Wiki example].`
		`+ ====`
		`+ ____`
		`+ . Send out the Mass Rebuild Notice`
		`+ +`
		`+ ____`
		`+ Send out the same information posted on the wiki to the`
		`+ [.title-ref]#devel-announce@lists.fedoraproject.org# mailing list.`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ See`
		`+ https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[the`
		`+ Fedora 26 e-mail example].`
		`+ ====`
		`+ ____`
		`+ . Create a Tag to Contain the Mass Rebuild`
		`+ +`
		`+ ____`
		`+ Mass rebuilds require their own tag to contain all related builds. The`
		`+ example assumes we are doing a rebuild for Fedora 26.`
		`+`
		`+ ....`
		`+ $ koji add-tag f26-rebuild --parent f26`
		`+ ....`
		`+ ____`
		`+ . Request Package Auto-Signing for New Mass-Rebuild Tag`
		`+ +`
		`+ ____`
		`+ File a ticket with https://pagure.io/fedora-infrastructure/issues[Fedora`
		`+ Infrastructure] requesting the new mass-rebuild tag be enabled for`
		`+ package auto-signing.`
		`+ ____`
		`+ . Create the Koji Target for the Mass Rebuild`
		`+ +`
		`+ ____`
		`+ Using the same [.title-ref]#f26-rebuild# tag created in the previous`
		`+ example:`
		`+`
		`+ ....`
		`+ $ koji add-target f26-rebuild f26-build`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ koji add-target _target-name_ _buildroot-tag_ _destination-tag_`
		`+ describes the syntax format above. If the _destination-tag_ is not`
		`+ specified then it will be the same as the _target-name_.`
		`+ ====`
		`+ ____`
		`+ . Update Scripts`
		`+ +`
		`+ ____`
		`+ The mass rebuild depends on four main scripts from the`
		`+ https://pagure.io/releng[releng git repository]. Each one requires some`
		`+ changes in variables for each new mass rebuild cycle.`
		`+`
		`+ * {blank}`
		`+ +`
		`+ mass-rebuild.py::`
		`+ ** buildtag`
		`+ ** targets`
		`+ ** epoch`
		`+ ** comment`
		`+ ** target`
		`+ * {blank}`
		`+ +`
		`+ find-failures.py::`
		`+ ** buildtag`
		`+ ** desttag`
		`+ ** epoch`
		`+ * mass-tag.py`
		`+ * {blank}`
		`+ +`
		`+ need-rebuild.py::`
		`+ ** buildtag`
		`+ ** target`
		`+ ** updates`
		`+ ** epoch`
		`+ ____`
		`+`
		`+ Change the following items:`
		`+`
		`+ * the build tag, holding tag, and target tag should be updated to`
		`+ reflect the Fedora release you're building for`
		+ * the `epoch` should be updated to the point at which all features that
		`+ the mass rebuild is for have landed in the build system (and a newRepo`
		`+ task completed with those features)`
		`+ * the comment which is inserted into spec changelogs`
		`+`
		`+ ==== Starting the Mass Rebuild`
		`+`
		+ The `mass-rebuild.py` script takes care of:
		`+`
		`+ * Discovering available packages in koji`
		`+ * Trimming out packages which have already been rebuilt`
		`+ * Checking out packages from git`
		`+ * Bumping the spec file`
		`+ * Committing the change`
		`+ * git tagging the change`
		`+ * Submitting the build request to Koji`
		`+`
		`+ . Connect to the mass-rebuild Machine`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh branched-composer.phx2.fedoraproject.org`
		`+ ....`
		`+ ____`
		`+ . Start a terminal multiplexer`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ tmux`
		`+ ....`
		`+ ____`
		`+ . Clone or checkout the latest copy of the`
		`+ https://pagure.io/releng[releng git repository].`
		`+ . Run the mass-rebuild.py script from _releng/scripts_`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd path/to/releng_repo/scripts`
		`+ $ ./mass-rebuild.py 2>&1 \| tee ~/massbuild.out`
		`+ ....`
		`+ ____`
		`+`
		`+ ==== Monitoring Mass Rebuilds`
		`+`
		`+ The community has a very high interest in the status of rebuilds and`
		`+ many maintainers will want to know if their build failed right away. The`
		+ `find-failures.py` and `need-rebuild.py` scripts are designed to update
		`+ publicly available URLs for stakeholders to monitor.`
		`+`
		`+ . Connect to a Compose Machine`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh compose-x86-02.phx2.fedoraproject.org`
		`+ ....`
		`+ ____`
		`+ . Start a terminal multiplexer`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ tmux`
		`+ ....`
		`+ ____`
		`+ . Clone or checkout the latest copy of the`
		`+ https://pagure.io/releng[releng git repository]`
		`+ . {blank}`
		`+ +`
		`+ Set Up the Rebuild Failures Notification Web Site::`
		+ The `find_failures.py` script discovers attempted builds that have
		`+ failed. It lists those failed builds and sorts them by package owner.`
		`+ +`
		`+ ....`
		`+ $ while true; do ./find_failures.py > f26-failures.html && cp f26-failures.html /mnt/koji/mass-rebuild/f26-failures.html; sleep 600; done`
		`+ ....`
		`+ . Start a second pane in the terminal emulator`
		`+ . {blank}`
		`+ +`
		`+ Set up the Site for Packages that Need Rebuilt::`
		+ The `need-rebuild.py` script discovers packages that have not yet been
		`+ rebuilt and generates an html file listing them sorted by package`
		`+ owner. This gives external stakeholders a rough idea of how much work`
		`+ is remaining in the mass rebuild.`
		`+ +`
		`+ ....`
		`+ $ while true; do ./need-rebuild.py > f26-need-rebuild.html && cp f26-need-rebuild.html /mnt/koji/mass-rebuild/f26-need-rebuild.html; sleep 600; done`
		`+ ....`
		`+`
		`+ ==== Post Mass Rebuild Tasks`
		`+`
		`+ Once the mass rebuild script completes, and all the pending builds have`
		+ finished, the builds will need to be tagged. The `mass-tag.py` script
		`+ will accomplish this task. The script will:`
		`+`
		`+ * Discover completed builds`
		`+ * Trim out builds that are older than the latest build for a given`
		`+ package`
		`+ * Tag remaining builds into their final destination (without generating`
		`+ email)`
		`+`
		`+ . Clone or checkout the latest copy of the`
		`+ https://pagure.io/releng[releng git repository]`
		+ . Run the `mass-tag.py` script (requires koji kerberos authentication)
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd path/to/releng_repo/scripts`
		`+ $ ./mass-tag.py --source f36-rebuild --target f36`
		`+ ....`
		`+ ____`
		`+ . Send the final notification to the`
		`+ _devel-announce@lists.fedoraproject.org_ list`
		`+ +`
		`+ ____`
		`+ The contents should look something like this`
		`+ https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QAMEEWUG7ND5E7LQYXQSQLRUDQPSBINA/[example`
		`+ email].`
		`+ ____`

modules/release_guide/pages/sop_package_blocking.adoc

file added

+17

		`@@ -0,0 +1,17 @@`
		`+ == Package Blocking`
		`+`
		`+ === Description`
		`+`
		`+ If a`
		`+ https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life[package`
		`+ is removed (retired) from Fedora], for example because it was renamed,`
		`+ it needs to be blocked in Koji. This prevents creating new package`
		`+ builds and distribution of built RPMs. Packages are blocked in the`
		+ listing of `tags`, due to inheritance it is enough to block packages at
		`+ the oldest tag will make it unavailable also in upstream tags.`
		`+`
		`+ === Action`
		`+`
		`+ The blocking of retired packages is done by the`
		`+ https://pagure.io/releng/blob/master/f/scripts/block_retired.py[block_retired.py]`
		`+ script as part of the daily Rawhide and Branched composes.`

modules/release_guide/pages/sop_package_unblocking.adoc

file added

+121

		`@@ -0,0 +1,121 @@`
		`+ == Package Unblocking`
		`+`
		`+ === Description`
		`+`
		`+ Packages are sometimes unblocked from Fedora, usually when a package had`
		`+ been orphaned and now has a new owner. When this happens, release`
		`+ engineering needs to "unblock" the package from koji tags.`
		`+`
		`+ === Action`
		`+`
		`+ ==== Find Unblock requests`
		`+`
		`+ Unblock requests are usually reported in the`
		`+ https://pagure.io/releng/issues[rel-eng issue tracker].`
		`+`
		`+ ==== Perform the unblocking`
		`+`
		`+ First assign the ticket to yourself to show, that you are handling the`
		`+ request.`
		`+`
		`+ ===== Discover proper place to unblock`
		`+`
		`+ The ticket should tell you which Fedora releases to unblock the package`
		`+ in. Typically it'll say "Fedora 13" or "F14". This means we need to`
		`+ unblock it at that Fedora level and all future tags. However depending`
		`+ on where the package was blocked we may have to do our unblock action at`
		`+ a different Fedora level.`
		`+`
		+ To discover where a package is blocked, use the `list-pkgs` method of
		`+ koji.`
		`+`
		`+ ....`
		`+ $ koji list-pkgs --help`
		`+ Usage: koji list-pkgs [options]`
		`+ (Specify the --help global option for a list of other help options)`
		`+`
		`+ Options:`
		`+ -h, --help show this help message and exit`
		`+ --owner=OWNER Specify owner`
		`+ --tag=TAG Specify tag`
		`+ --package=PACKAGE Specify package`
		`+ --quiet Do not print header information`
		`+ --noinherit Don't follow inheritance`
		`+ --show-blocked Show blocked packages`
		`+ --show-dups Show superseded owners`
		`+ --event=EVENT# query at event`
		`+ --ts=TIMESTAMP query at timestamp`
		`+ --repo=REPO# query at event for a repo`
		`+ ....`
		`+`
		`+ For example if we wanted to see where python-psco was blocked we would`
		`+ do:`
		`+`
		`+ ....`
		`+ $ koji list-pkgs --package python-psyco --show-blocked`
		`+ Package Tag Extra Arches Owner`
		`+ ----------------------- ----------------------- ---------------- ---------------`
		`+ python-psyco dist-f14 konradm [BLOCKED]`
		`+ python-psyco olpc2-ship2 shahms`
		`+ python-psyco olpc2-trial3 shahms`
		`+ ...`
		`+ ....`
		`+`
		`+ Here we can see that it was blocked at dist-f14. If we got a request`
		`+ that was to unblock it before f14, we can simply use the dist-f14 target`
		`+ to unblock. However if they want it unblocked after f14, we would use`
		`+ the earliest dist-f?? tag the user wants, such as dist-f15 if the user`
		`+ asked for it to be unblocked in Fedora 15+`
		`+`
		`+ ===== Performing the unblock`
		`+`
		+ To unblock a package for a tag, use the `unblock-pkg` method of Koji.
		`+`
		`+ ....`
		`+ $ koji unblock-pkg --help`
		`+ Usage: koji unblock-pkg [options] tag package [package2 ...]`
		`+ (Specify the --help global option for a list of other help options)`
		`+`
		`+ Options:`
		`+ -h, --help show this help message and exit`
		`+ ....`
		`+`
		`+ For example, if we were asked to unblock python-psyco in F14 we would`
		`+ issue:`
		`+`
		`+ ....`
		`+ $ koji unblock-pkg dist-f14 python-psyco`
		`+ ....`
		`+`
		`+ Now the ticket can be closed.`
		`+`
		`+ === Verification`
		`+`
		`+ To verify that the package was successfully unblocked use the`
		+ `list-pkgs` koji command:
		`+`
		`+ ....`
		`+ $ koji list-pkgs --package python-psyco --show-blocked`
		`+ ....`
		`+`
		`+ We should see the package listed as not blocked at dist-f14 or above:`
		`+`
		`+ ....`
		`+ Package Tag Extra Arches Owner`
		`+ ----------------------- ----------------------- ---------------- ---------------`
		`+ python-psyco olpc2-trial3 jkeating`
		`+ python-psyco olpc2-ship2 jkeating`
		`+ python-psyco olpc2-update1 jkeating`
		`+ python-psyco trashcan jkeating`
		`+ python-psyco f8-final jkeating`
		`+ ...`
		`+ ....`
		`+`
		`+ We should not see it listed as blocked in dist-f14 or any later Fedora`
		`+ tags.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ * Watch the next day's rawhide/branched/whatever report for a slew of`
		`+ broken deps related to the package. We may have to re-block the package`
		`+ in order to fix the deps.`

modules/release_guide/pages/sop_pdc.adoc

file added

+7

		`@@ -0,0 +1,7 @@`
		`+ == Product Definition Center`
		`+`
		`+ This is a stub SOP for PDC. The doc really lives in the infra SOPs,`
		`+ here: https://infrastructure.fedoraproject.org/infra/docs/pdc.rst`
		`+`
		`+ Please update and maintain that SOP. Leave this document here as a`
		`+ breadcrumb to make it easier to find that doc.`

modules/release_guide/pages/sop_process_dist_git_requests.adoc

file added

+33

		`@@ -0,0 +1,33 @@`
		`+ == Process fedora-scm-requests tickets`
		`+`
		`+ === Description`
		`+`
		`+ When a packager wants a new package added to Fedora or a new dist-git`
		`+ branch blessed, they need to go through the new package process and,`
		`+ once their package review is approved, they use the`
		`+ [.title-ref]#fedrepo-req# cli tool to file a ticket in the`
		`+ https://pagure.io/releng/fedora-scm-requests[fedora-scm-requests queue].`
		`+`
		`+ Periodically, (daily?) release engineering will need to review and`
		`+ process this queue using the [.title-ref]#fedrepo-req-admin# tool.`
		`+`
		`+ === Setup`
		`+`
		`+ A release engineering will need to have several values set locally as`
		`+ well as sufficient permissions in a number of server-side systems.`
		`+`
		`+ . A pagure.io token. See the fedrepo-req README for instructions on`
		`+ where to get this.`
		`+ . src.fedoraproject.org token generated by [.title-ref]#pagure-admin#.`
		`+ Ask @pingou how to get one. If doing this yourself, go to pkgs01 and run`
		`+ [.title-ref]#PAGURE_CONFIG=/etc/pagure/pagure.cfg pagure-admin`
		`+ admin-token create -h# for more info.`
		`+ . pdc token. See the PDC SOP for getting one of these.`
		`+`
		`+ === Action`
		`+`
		`+ . Run [.title-ref]#fedrepo-req-admin list# to list all open requests.`
		`+ . Run [.title-ref]#fedrepo-req-admin process N# to process a particular`
		`+ ticket.`
		`+ . Run [.title-ref]#fedrepo-req-admin processall# to iterate over all the`
		`+ tickets.`

modules/release_guide/pages/sop_promoting_container_content.adoc

file added

+50

		`@@ -0,0 +1,50 @@`
		`+ == Promoting Container Content`
		`+`
		`+ === Description`
		`+`
		`+ Even though the promotion of content is aimed to be fully automated,`
		`+ sometimes there is the need or desire to promote content from the`
		`+ candidate registry to the stable registry. Below is how to accomplish`
		`+ this using https://github.com/projectatomic/skopeo[skopeo].`
		`+`
		`+ === Action`
		`+`
		`+ This action should be performed on`
		+ `compose-x86-01.phx2.fedoraproject.org` by an user who is a member of
		+ the `sysadmin-releng` https://admin.fedoraproject.org/accounts/[FAS]
		`+ Group.`
		`+`
		`+ The container image should be provided by the requester, they will have`
		`+ the information from their container build in the`
		`+ https://docs.pagure.org/releng/layered_image_build_service.html[Fedora`
		`+ Layered Image Build System]. It will have a name resembling`
		+ `candidate-registry.fedoraproject.org/f26/foo:0.1-1.f26container`.
		`+`
		`+ Sync content between the candidate registry and production registry,`
		+ effectively "promoting it". Substitute the `$IMAGE_NAME` in the
		`+ following example with whatever the actual image name is that is`
		`+ provided. (This would be everything after`
		+ `candidate-registry.fedoraproject.org`, so using the above example then
		+ `$IMAGE_NAME` would be `f26/foo:0.1-1.f26container`.)
		`+`
		`+ ....`
		`+ $ sudo skopeo copy \`
		`+ --src-cert-dir /etc/docker/certs.d/candidate-registry.fedoraproject.org/ \`
		`+ --dest-cert-dir /etc/docker/certs.d/registry.fedoraproject.org/ \`
		`+ docker://candidate-registry.fedoraproject.org/$IMAGE_NAME \`
		`+ docker://registry.fedoraproject.org/$IMAGE_NAME`
		`+ ....`
		`+`
		`+ === Verification`
		`+`
		`+ In order to verify, we need to inspect the stable registry, again with`
		`+ https://github.com/projectatomic/skopeo[skopeo] to ensure the image`
		`+ metadata exists.`
		`+`
		`+ ....`
		`+ $ skopeo inspect docker://registry.fedoraproject.org/$IMAGE_NAME`
		`+ ....`
		`+`
		+ In this JSON output you will see a list element titled `RepoTags` and in
		+ there should be the `$VERSION-$RELEASE` listed there, following our
		+ example above this entry would be `0.1-1.f26container`.

modules/release_guide/pages/sop_pushing_updates.adoc

file added

+225

		`@@ -0,0 +1,225 @@`
		`+ == Pushing Updates`
		`+`
		`+ === Description`
		`+`
		`+ Fedora updates are typically pushed once a day. This SOP covers the`
		`+ steps involved.`
		`+`
		`+ ==== Coordinate`
		`+`
		`+ Releng has a rotation of who pushes updates when. Please coordinate and`
		`+ only push updates when you are expected to or have notified other releng`
		`+ folks you are doing so. See:`
		`+ https://apps.fedoraproject.org/calendar/release-engineering/ for the`
		+ list or on irc you can run `.pushduty` in any channel with zodbot to see
		`+ who is on duty this week.`
		`+`
		`+ ==== Login to machine to sign updates`
		`+`
		`+ Login to a machine that is configured for sigul client support and has`
		`+ the bodhi client installed. Currently, this machine is:`
		+ `bodhi-backend01.phx2.fedoraproject.org`
		`+`
		`+ ==== Decide what releases you're going to push.`
		`+`
		`+ * If there is a Freeze ongoing, you SHOULD NOT push all stable requests`
		`+ for a branched release, only specific blocker or freeze exception`
		`+ requests that QA will request in a releng ticket.`
		`+ * If there is no Freeze ongoing you can push all Fedora and EPEL`
		`+ releases at the same time if you wish.`
		`+ * From time to time there may be urgent requests in some branches, you`
		`+ can only push those if requested. Note however that bodhi2 will`
		`+ automatically push branches with security updates before others.`
		`+`
		`+ ==== Get a list of packages to push`
		`+`
		`+ ....`
		`+ $ sudo -u apache bodhi-push --releases 'f26,f25,f24,epel-7,EL-6' --username <yourusername>`
		`+ <enter your password+2factorauth, then your fas password>`
		`+ ....`
		`+`
		`+ Sometimes you see a warning "Warning: foobar-1.fcxx has unsigned builds`
		`+ and has been skipped" which means those updates are currently getting`
		`+ signed and it can verified by listing the tagged builds in`
		`+ fxx-signing-pending tag.`
		`+`
		`+ ::::`
		`+ $ koji list-tagged fxx-signing-pending`
		`+`
		`+ You can say 'n' to the push at this point if you wish to sign packages`
		`+ (see below). Or you can keep this request open in a window while you`
		`+ sign the packages, then come back and say y.`
		`+`
		`+ List the releases above you wish to push from: 25 24 5 6 7, etc`
		`+`
		+ You can also specify `--request=testing` to limit pushes. Valid types
		+ are `testing` or `stable`.
		`+`
		`+ The list of updates should be in the cache directory named`
		+ `Stable-$Branch` or `Testing-$Branch` for each of the Branches you
		`+ wished to push.`
		`+`
		`+ During freezes you will need to do two steps: (If say, fedora 26`
		`+ branched was frozen):`
		`+`
		`+ ....`
		`+ $ sudo -u apache bodhi-push --releases f26 --request=testing \`
		`+ --username <username>`
		`+ ....`
		`+`
		`+ Then`
		`+`
		`+ ....`
		`+ $ sudo -u apache bodhi-push --releases 'f25,f24,epel-7,EL-6' --username <username>`
		`+ ....`
		`+`
		`+ During the Release Candidate compose phase we tag builds to be included`
		`+ into a -compose tag (e.g. f26-compose). When we have a candidate that`
		`+ has been signed off as gold we need to ensure that all builds tagged`
		`+ into the -compose tag have been pushed stable. Once we have pushed all`
		`+ -compose builds stable we then have to clone the base tag (e.g. f26) to`
		`+ a tag for the milestone for Alpha and Beta (e.g. f26-Alpha). After final`
		`+ release we need to lock the base tag and adjust the release status in`
		`+ bodhi so that updates now hit the -updates tag (e.g. f26-updates). Once`
		`+ we have cloned the tag or locked the tag and adjusted bodhi we are free`
		`+ to push stable updates again.`
		`+`
		`+ ==== Pushing Stable updates during freeze`
		`+`
		`+ During feezes we need to push to stable builds included in the compose.`
		`+ QA will file a ticket with the nvrs to push.`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ If you are pushing a bodhi update that contains multiple builds, you`
		`+ need only pass bodhi-push a single build nvr and all the others in that`
		`+ update will be detected and pushed along with it. However, if you are`
		`+ pushing multiple disjoint bodhi updates then each build will need to be`
		`+ listed individually.`
		`+ ====`
		`+`
		`+ ....`
		`+ $ sudo -u apache bodhi-push --builds '<nvr1>,<nvr2>,...' --username <username>`
		`+ ....`
		`+`
		`+ ===== There are no updates to push.`
		`+`
		+ If you are getting the message `There are no updates to push.` or the
		`+ list of packages you are seeing to push out for the Stable updates`
		+ request is not correct compared to what you specified in the `--builds`
		`+ section of the command above then one of two things likely happened.`
		`+`
		`+ . The update hasn't yet reached appropriate karma`
		`+ +`
		`+ This should be handled case-by-case, if the QA Team has requested this`
		`+ be pushed as stable to fix a blocker but there's not yet enough karma`
		`+ for an autostable prompt to occur then you should verify with QA that`
		`+ these are ready to go out even without karma. If they are, then log into`
		`+ the Bodhi WebUI and modify the karma threshold of the update to 1 and`
		`+ add karma (if necessary). This is not something we should do as normal`
		`+ practice and is considered an edge case. When update requests come to`
		`+ RelEng, it should have appropriate karma. Sometimes it doesn't and as`
		`+ long as QA approves, we need not block on it.`
		`+ . The update was never requested for stable`
		`+ +`
		`+ It's possible the update wasn't requested for stable, you can resolve`
		`+ this by running the following on one of the bodhi-backend systems:`
		`+ +`
		`+ ....`
		`+ bodhi updates request <BODHI-REQUEST-ID> stable`
		`+ ....`
		`+`
		`+ ==== Perform the bodhi push`
		`+`
		`+ Say 'y' to push for the above command.`
		`+`
		`+ === Verification`
		`+`
		+ . Monitor Bodhi's composes with `bodhi-monitor-composes`
		`+ +`
		`+ ....`
		`+ $ sudo -u apache watch -d -n 60 bodhi-monitor-composes`
		`+ ....`
		`+ . Monitor the systemd journal`
		`+ +`
		`+ ....`
		`+ $ sudo journalctl -o short -u fedmsg-hub -l -f`
		`+ ....`
		`+ . Check the processes`
		`+ +`
		`+ ....`
		`+ $ ps axf\|grep bodhi`
		`+ ....`
		`+ . Watch for fedmsgs through the process. It will indicate what releases`
		+ it's working on, etc. You may want to watch in `#fedora-fedmsg`.
		`+ +`
		`+ ....`
		`+ bodhi.masher.start -- kevin requested a mash of 48 updates`
		`+ bodhi.mashtask.start -- bodhi masher started a push`
		`+ bodhi.mashtask.mashing -- bodhi masher started mashing f23-updates`
		`+ bodhi.mashtask.mashing -- bodhi masher started mashing f22-updates-testing`
		`+ ...`
		`+ bodhi.update.complete.stable -- moceap's wondershaper-1.2.1-5.fc23 bodhi update completed push to stable https://admin.fedoraproject.org/updates/FEDORA-2015-13052`
		`+ ...`
		`+ bodhi.errata.publish -- Fedora 23 Update: wondershaper-1.2.1-5.fc23 https://admin.fedoraproject.org/updates/FEDORA-2015-13052`
		`+ bodhi.mashtask.complete -- bodhi masher successfully mashed f23-updates`
		`+ bodhi.mashtask.sync.wait -- bodhi masher is waiting for f22-updates-testing to hit the master mirror`
		`+ ....`
		`+ . Seach for problems with a particular push:`
		`+ +`
		`+ ....`
		`+ sudo journalctl --since=yesterday -o short -u fedmsg-hub \| grep dist-6E-epel (or f22-updates, etc)`
		`+ ....`
		`+ . Note: Bodhi will look at the things you have told it to push and see`
		`+ if any have security updates, those branches will be started first. It`
		`+ will then fire off threads (up to 3 at a time) and do the rest.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ Pushes often fall over due to tagging issues or unsigned packages. Be`
		`+ prepared to work through the failures and restart pushes from time to`
		`+ time`
		`+`
		`+ ....`
		`+ $ sudo -u apache bodhi-push --resume`
		`+ ....`
		`+`
		`+ Bodhi will ask you which push(es) you want to resume.`
		`+`
		`+ === Common issues / problems with pushes`
		`+`
		`+ * When the push fails due to new unsigned packages that were added after`
		`+ you started the process. re-run step 4a or 4b with just the package`
		`+ names that need to be signed, then resume.`
		`+ * When the push fails due to an old package that has no signature, run:`
		+ `koji write-signed-rpm <gpgkeyid> <n-v-r>` and resume.
		`+ * When the push fails due to a package not being tagged with`
		`+ updates-testing when being moved stable:`
		+ `koji tag-pkg dist-<tag>-updates-testing <n-v-r>`
		`+ * When signing fails, you may need to ask that the sigul bridge or`
		`+ server be restarted.`
		`+ * If the updates push fails with a:`
		+ `OSError: [Errno 16] Device or resource busy: '/var/lib/mock/-x86_64/root/var/tmp/rpm-ostree.'`
		`+ You need to umount any tmpfs mounts still open on the backend and resume`
		`+ the push.`
		`+ * If the updates push fails with:`
		+ `"OSError: [Errno 39] Directory not empty: '/mnt/koji/mash/updates//../.repocache/repodata/'`
		`+ you need to restart fedmsg-hub on the backend and resume.`
		`+ * If the updates push fails with:`
		+ `IOError: Cannot open /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/epel7-160228.1356/../epel7.repocache/repodata/repomd.xml doesn't exists or not a regular file`
		`+ This issue will be resolved with NFSv4, but in the mean time it can be`
		`+ worked around by removing the [.title-ref]#.repocache# directory and`
		`+ resuming the push.`
		+ `$ sudo rm -fr /mnt/koji/mash/updates/epel7.repocache`
		`+ * If the Atomic OSTree compose fails with some sort of`
		`+ [.title-ref]#Device or Resource busy# error, then run`
		`+ [.title-ref]#mount# to see if there are any stray [.title-ref]#tmpfs#`
		`+ mounts still active:`
		+ `tmpfs on /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq type tmpfs (rw,relatime,seclabel,mode=755)`
		`+ You can then`
		+ `$ sudo umount /var/lib/mock/fedora-22-updates-testing-x86_64/root/var/tmp/rpm-ostree.bylgUq`
		`+ and resume the push.`
		`+`
		`+ Other issues should be addressed by releng or bodhi developers in`
		+ `#fedora-releng`.

modules/release_guide/pages/sop_rawhide_bodhi.adoc

file added

+95

		`@@ -0,0 +1,95 @@`
		`+ == Enabling Rawhide in Bodhi`
		`+`
		`+ === Description`
		`+`
		`+ This SOP covers the steps needed to enable Rawhide in Bodhi.`
		`+`
		`+ ==== Create the release in Bodhi`
		`+`
		`+ In oder to start creating updates in Bodhi for rawhide, the release`
		`+ needs to be created in Bodhi. Rawhide in Bodhi is represented by the`
		`+ Fedora version (ie Fedora 31), but it is set in the prerelease state.`
		`+`
		`+ ===== Add the koji tags`
		`+`
		`+ ....`
		`+ $ koji add-tag --parent f31 f31-updates-candidate`
		`+ $ koji add-tag --parent f31 f31-updates-testing`
		`+ $ koji add-tag --parent f31-updates-testing f31-updates-testing-pending`
		`+ $ koji edit-tag --perm autosign f31-updates-testing-pending`
		`+ $ koji add-tag --parent f31 f31-updates-pending`
		`+ $ koji add-tag --parent f31 f31-override`
		`+ ....`
		`+`
		`+ ===== Change the koji targets`
		`+`
		`+ ....`
		`+ $ koji edit-target f31 --dest-tag f31-updates-candidate`
		`+ $ koji edit-target f31-candidate --dest-tag f31-updates-candidate`
		`+ $ koji edit-target rawhide --dest-tag f31-updates-candidate`
		`+ ....`
		`+`
		`+ ===== Create the release in bodhi`
		`+`
		`+ ....`
		`+ $ bodhi releases create --name "F31" --long-name "Fedora 31" --id-prefix FEDORA --version 31 --branch f31 \`
		`+ --dist-tag f31 --stable-tag f31 --testing-tag f31-updates-testing --candidate-tag f31-updates-candidate \`
		`+ --pending-stable-tag f31-updates-pending --pending-testing-tag f31-updates-testing-pending \`
		`+ --state pending --override-tag f31-override --create-automatic-updates --not-composed-by-bodhi`
		`+ ....`
		`+`
		`+ The important flags are [.title-ref]#--not-composed-by-bodhi# which`
		`+ tells bodhi not to include the rawhide updates in the nightly pushes and`
		`+ [.title-ref]#--create-automatic-updates# which tells bodhi to`
		`+ automatically create an update listen to koji tag (build tagged with the`
		`+ pending-testing-tag) messages.`
		`+`
		`+ ===== Bodhi configuration`
		`+`
		`+ Bodhi is configured to required zero mandatory days in testing for the`
		`+ rawhide release. This is done in ansible`
		`+ roles/bodhi2/base/templates/production.ini.j2 with the following.`
		`+`
		`+ ....`
		`+ f{{ FedoraRawhideNumber }}.pre_beta.mandatory_days_in_testing = 0`
		`+ ....`
		`+`
		`+ ===== Robosignatory configuration`
		`+`
		`+ Robosignatory needs to be configured to signed the rawhide builds before`
		`+ these builds are tested by the CI pipeline.`
		`+`
		`+ ....`
		`+ {`
		`+ "from": "f31-updates-candidate",`
		`+ "to": "f31-updates-testing-pending",`
		`+ "key": "fedora-31",`
		`+ "keyid": "3c3359c4"`
		`+ },`
		`+ ....`
		`+`
		`+ ==== Branching Rawhide`
		`+`
		`+ When it is time to branch rawhide, a new release should be created (eg.`
		`+ F32) following the steps above. The old rawhide release (eg. F31) should`
		`+ stay configured as rawhide until we active Bodhi for it (2 weeks later).`
		`+ To activate Bodhi on the old rawhide (eg. F31) the existing release in`
		`+ bodhi should be modified has follow.`
		`+`
		`+ ::::`
		`+ $ bodhi releases edit --name "F31" --stable-tag f31-updates`
		`+ --no-create-automatic-updates --composed-by-bodhi`
		`+`
		`+ ===== Robosignatory configuration`
		`+`
		`+ At Bodhi activation time the Robosignatory configuration needs to be`
		`+ update to match the normal configuration of bodhi releases.`
		`+`
		`+ ....`
		`+ {`
		`+ "from": "f31-signing-pending",`
		`+ "to": "f31-updates-testing-pending",`
		`+ "key": "fedora-31",`
		`+ "keyid": "3c3359c4"`
		`+ },`
		`+ ....`

modules/release_guide/pages/sop_release_container_base_image.adoc

file added

+76

		`@@ -0,0 +1,76 @@`
		`+ == Release the Fedora Container Base Image`
		`+`
		`+ === Description`
		`+`
		`+ This SOP covers the steps involved in changing and releasing the Fedora`
		`+ Container Base image.`
		`+`
		`+ Fedora releases 2 container base images, [.title-ref]#fedora# and`
		`+ [.title-ref]#fedora-minimal#. These images are available on 3 registries`
		`+ [.title-ref]#registry.fedoraproject.org#, [.title-ref]#quay.io# and`
		`+ [.title-ref]#DockerHub# (fedora-minimal is not available in DockerHub).`
		`+`
		`+ ==== Modify a base image (Kickstart)`
		`+`
		`+ Base images are built in koji using the [.title-ref]#image-factory#`
		`+ application to build the container image root filesystem (rootfs).`
		`+ Kickstart files are used to configure how the image is built and what is`
		`+ available in the image The solution consist of 3 Kickstarts.`
		`+`
		`+ https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-common.ks[fedora-container-common]`
		`+`
		`+ https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base.ks[fedora-container-base]`
		`+`
		`+ https://pagure.io/fedora-kickstarts/blob/main/f/fedora-container-base-minimal.ks[fedora-container-base-minimal]`
		`+`
		`+ Changes made on the rawhide branch will results in the rawhide image,`
		`+ other branches (f30, f31) should be used to modify other releases.`
		`+`
		`+ ==== Compose Configuration (Pungi)`
		`+`
		`+ The configuration used to compose the container images is available in`
		`+ the pungi-fedora repository.`
		`+`
		`+ For rawhide the configuration is in`
		`+`
		`+ https://pagure.io/pungi-fedora/blob/main/f/fedora.conf`
		`+`
		`+ While for other releases the configuration is in a dedicated file`
		`+`
		`+ https://pagure.io/pungi-fedora/blob/f31/f/fedora-container.conf`
		`+`
		`+ ==== Release on registry.fedoraproject.org and quay.io`
		`+`
		`+ If you want to release the base image on registry.fp.o and quay.io you`
		`+ can use the following script.`
		`+`
		`+ https://pagure.io/releng/blob/main/f/scripts/sync-latest-container-base-image.sh[sync-latest-container-base-image.sh]`
		`+`
		`+ You will need to run that script from on of the releng composer machines`
		`+ in the infrastructure in order to have the credentials.`
		`+`
		`+ If you do not have access to that machines, you can request the release`
		`+ by opening a ticket on the https://pagure.io/releng/issues[releng`
		`+ tracker].`
		`+`
		`+ The script can then be executed as follow`
		`+`
		`+ ....`
		`+ $./sync-latest-container-base-image.sh 31`
		`+ $./sync-latest-container-base-image.sh 32`
		`+ ....`
		`+`
		`+ This will take care of pushing the [.title-ref]#fedora# and`
		`+ [.title-ref]#fedora-minimal# images to both registries.`
		`+`
		`+ ==== Release on DockerHub`
		`+`
		`+ Releasing on DockerHub is a little different since Fedora is an`
		`+ "offical" image there. In order to release new images there we have to`
		`+ update a Dockerfile and rootfs tarball on the following repo.`
		`+`
		`+ https://github.com/fedora-cloud/docker-brew-fedora[docker-brew-fedora].`
		`+`
		`+ For the details on how to run the script please see`
		`+`
		`+ https://github.com/fedora-cloud/docker-brew-fedora/blob/main/README.md[README].`

modules/release_guide/pages/sop_release_package_signing.adoc

file added

+68

		`@@ -0,0 +1,68 @@`
		`+ == Release Package Signing`
		`+`
		`+ === Description`
		`+`
		`+ For each of Fedora's public releases (Alpha, Beta, and Final) it is`
		`+ Release Engineering's responsibility to sign all packages with Fedora's`
		`+ GPG key. This provides confidence to Fedora's users about the`
		`+ authenticity of packages provided by Fedora.`
		`+`
		+ The `/sop_create_release_signing_key` document explains the process for
		`+ creating the GPG key used.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ This script takes a very long time to run, as much as 4 or 5 days, so it`
		`+ needs to be started well in advance of when you need the packages all`
		`+ signed.`
		`+`
		`+ Signing all the packages will cause a lot of churn on the mirrors, so`
		`+ expect longer than usual compose and rsync times, as well as potential`
		`+ issues with proxies as file contents change but the name remains the`
		`+ same.`
		`+`
		`+ === Action`
		`+`
		+ . Log into a system with `sigul` and start a `screen` or `tmux` session.
		`+ The signing process takes a long time--screen allows the process to`
		`+ continue if you session gets disconnected.`
		`+ +`
		`+ ....`
		`+ $ screen -S sign`
		`+ ....`
		`+ +`
		`+ or`
		`+ +`
		`+ ....`
		`+ $ tmux new -s sign`
		`+ ....`
		+ . Check out the Release Engineering `git` repo
		`+ +`
		`+ ::::`
		`+ $ git clone https://pagure.io/releng.git`
		+ . Change directories to the `scripts` directory to execute
		+ `sigulsign_unsigned.py`.
		`+ +`
		`+ For example, to sign everything for Fedora 13 Alpha we would issue:`
		`+ +`
		`+ ::::`
		`+ $ ./sigulsign_unsigned.py -vv --tag dist-f13 fedora-13`
		`+ +`
		`+ This signs the packages with verbose output so you can track progress`
		`+ incrementally.`
		`+`
		`+ === Verification`
		`+`
		+ Once the signing is done, use `rpmdev-checksig` to verify that a package
		`+ has been signed. You can use the output of a recent rawhide compose to`
		`+ test. In this example we use a released Fedora 12 package:`
		`+`
		`+ ....`
		`+ $ rpmdev-checksig /pub/fedora/linux/releases/12/Everything/i386/os/Packages/pungi-2.0.20-1.fc12.noarch.rpm`
		`+ /pub/fedora/linux/releases/12/Everything/i386/os/Packages/pungi-2.0.20-1.fc12.noarch.rpm: MISSING KEY - 57bbccba`
		`+ ....`
		`+`
		+ This output shows that the apckage was signed with key `57bbccba`, and
		`+ that this key does not exist in your local rpm database. If the key did`
		`+ exist in the local rpm database it's likely there would be no output so`
		`+ it's best to run this on a system that does not have gpg keys imported.`

modules/release_guide/pages/sop_remote_dist_git_branches.adoc

file added

+43

		`@@ -0,0 +1,43 @@`
		`+ == Remove dist-git branches`
		`+`
		`+ === Description`
		`+`
		`+ Release Engineering is often asked by maintainers to remove branches in`
		`+ dist-git by maintainers.`
		`+`
		`+ === Action`
		`+`
		`+ . Log into batcave01`
		`+ +`
		`+ ....`
		`+ ssh <fas-username>@batcave01.iad2.fedoraproject.org`
		`+ ....`
		`+ . Get root shell`
		`+ . Log into pkgs01.iad2.fedoraproject.org :`
		`+ +`
		`+ ....`
		`+ ssh pkgs01.iad2.fedoraproject.org`
		`+ ....`
		`+ . Change to the package's directory`
		`+ +`
		`+ ....`
		`+ cd /srv/git/rpms/<package>.git/`
		`+ ....`
		`+ . Remove the branch`
		`+ +`
		`+ ....`
		`+ git branch -D <branchname> </pre>`
		`+ ....`
		`+`
		`+ === Verification`
		`+`
		`+ To verify just list the branches.`
		`+`
		`+ ....`
		`+ git branch`
		`+ ....`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ Make sure that the branch in question isn't one of our pre-created`
		+ branches `f??/rawhide`, `olpc?/rawhide`, `el?/rawhide`

modules/release_guide/pages/sop_requesting_task_automation_users.adoc

file added

+27

		`@@ -0,0 +1,27 @@`
		`+ == Requesting Automation Users`
		`+`
		`+ [[sop_requesting_task_automation_user]]`
		`+ === Description`
		`+`
		+ When performing automated Release Engineering tasks using `RelEng
		+ Automation <_releng-automation>` you will sometimes find that you need
		+ to perform an action in the Infrastructure with `sudo` that does not yet
		`+ have an automation user associated with it.`
		`+`
		`+ === Actions`
		`+`
		`+ ==== Requesting a new loopabull user`
		`+`
		`+ File a ticket with https://pagure.io/fedora-infrastructure/[Fedora`
		`+ Infrastructure] making sure to satisfy the following requirements:`
		`+`
		`+ * Provide a justification for these permissions being needed (What are`
		`+ you trying to do and why?)`
		`+ * Commands needing to be run with sudo`
		`+ * Destination server on which the commands need to be run`
		+ * The `loopabull_` username requested to be created for this OR which
		+ `loopabull_` username needs it's pre-existing permissions enhanced
		`+`
		`+ For reference:`
		`+ https://pagure.io/fedora-infrastructure/issue/5943[Example`
		`+ Infrastructure Ticket]`

modules/release_guide/pages/sop_retire_orphaned_packages.adoc

file added

+83

		`@@ -0,0 +1,83 @@`
		`+ == Retire Orphaned Packages`
		`+`
		`+ === Description`
		`+`
		`+ Every release prior to the`
		`+ https://fedoraproject.org/wiki/Schedule[Feature Freeze/Branching]`
		`+ Release Engineering retires`
		`+ https://fedoraproject.org/wiki/Orphaned_package_that_need_new_maintainers[orphaned`
		`+ packages]. This keeps out unowned software and prevents future problems`
		`+ down the road.`
		`+`
		`+ === Action`
		`+`
		`+ The orphan process takes place in stages:`
		`+`
		`+ . Detecting a list of orphans and the dependencies that will be broken`
		`+ if the orphans are removed.`
		`+ . Sending the list of potential orphans to devel@lists.fedoraproject.org`
		`+ for community review and removal from the orphan list.`
		`+ . Retriring packages nobody wants to adopt.`
		`+`
		`+ ==== Detecting Orphans`
		`+`
		+ A script called `find_unblocked_orphans.py` assists in the detection
		+ process. It should be run on a machine that has `koji` and
		+ `python-fedora` installed. It runs without options and takes a while to
		`+ complete.`
		`+`
		+ `find_unblocked_orphans.py` is available in the
		`+ https://pagure.io/releng[Release Engineering git repository]`
		`+`
		`+ ==== Announcing Packages to be retired`
		`+`
		+ `find_unblocked_orphans.py` outputs text to stdout on the command line
		`+ in a form suitable for the body of an email message.`
		`+`
		`+ ....`
		`+ $ ./find-unblocked-orphans.py > email-message`
		`+ ....`
		`+`
		`+ Email the output to the development list`
		+ (`devel@lists.fedodraproject.org`) at least a month before the feature
		`+ freeze, send mails with updated lists as necessary. This gives`
		`+ maintainers an opportunity to pick up orphans that are important to them`
		`+ or are required by other packages.`
		`+`
		`+ ==== Retiring Orphans`
		`+`
		`+ Once maintainers have been given an opportunity to pick up orphaned`
		`+ packages, the remaining`
		`+ https://fedoraproject.org/wiki/How_to_remove_a_package_at_end_of_life[packages`
		`+ are retired]`
		`+`
		`+ ===== Bugs`
		`+`
		`+ This procedure probably leaves open bugs for the retired packages`
		`+ behind. It is not within the scope of release engineering to take care`
		`+ of these. If bugs are closed, only bugs targeted at Rawhide should be`
		`+ affected, since other branches might still be maintained.`
		`+`
		`+ === Verification`
		`+`
		`+ To verify that the packages were blocked correctly we can use the`
		+ `latest-pkg` `koji` action.
		`+`
		`+ ....`
		`+ $ koji latest-pkg dist-f21 wdm`
		`+ ....`
		`+`
		+ This should return nothing, as the `wdm` package is blocked.
		`+`
		`+ === Consider Before Running`
		`+`
		`+ Generally we retire anything that doesn't leave broken dependencies. If`
		`+ there are orphans whose removal would result in broken dependencies a`
		+ second warning should be sent to `devel@lists.fedoraproject.org` and to
		+ `<package>-owner@fedoraproject.org` for each dependent package.
		`+`
		`+ Allow another couple of days for maintainers to take notice and fix`
		`+ these package so the package repository can be maintained without broken`
		`+ dependencies or needing to the package. It is not good to have broken`
		`+ package dependencies in our package repositories so every effort should`
		`+ be made to find owners or to fix the broken dependencies.`

modules/release_guide/pages/sop_signing_builds.adoc

file added

+29

		`@@ -0,0 +1,29 @@`
		`+ == Sign the packages`
		`+`
		`+ * This doc explains how to sign builds in the release(s).`
		`+ * Manual signing should rarely ever be needed anymore. Just make sure`
		`+ that robosignatory is setup for all tags that are created.`
		`+ * If a build seems to be stuck in the autosigning queue (one of the`
		`+ -pending or -signing-pending tags), just koji untag and koji tag the`
		`+ package. This will retrigger autosigning.`
		`+ * If bodhi is reporting a build as unsigned but the build is not in the`
		`+ -signing-pending tag, that means bodhi missed the tagging. Just run the`
		`+ following command to make the build get retagged again, giving Bodhi`
		`+ another change at seeing the signing`
		`+ +`
		`+ ....`
		`+ $ koji move $dist-updates-testing-pending $dist-signing-pending $build`
		`+ ....`
		`+ * If need be, sign builds using scripts/sigulsign_unsigned.py from`
		`+ releng git repo`
		`+ +`
		`+ _NOTE! This will NOT help if Bodhi marks a build as unsigned!_`
		`+ +`
		`+ ....`
		`+ $ ./sigulsign_unsigned.py -vv --write-all \`
		`+ --sigul-batch-size=25 fedora-22 \`
		`+ $(cat /var/cache/sigul/Stable-F22 /var/cache/sigul/Testing-F22)`
		`+ ....`
		`+`
		`+ (Make sure you sign each release with the right key... ie, 'fedora-19'`
		`+ key with F19 packages, or 'epel-6' with EL-6 packages)`

modules/release_guide/pages/sop_sigul_client_setup.adoc

file added

+57

		`@@ -0,0 +1,57 @@`
		`+ == Sigul Client Setup`
		`+`
		`+ This document describes how to configure a sigul client. For more`
		`+ information on sigul, please see link:User-Mitr[User:Mitr]`
		`+`
		`+ === Prerequisites`
		`+`
		+ . Install `sigul` and its dependencies. It is available in both Fedora
		`+ and EPEL:`
		`+ +`
		`+ On Fedora:`
		`+ +`
		`+ ....`
		`+ dnf install sigul`
		`+ ....`
		`+ +`
		`+ On RHEL/CentOS (Using EPEL):`
		`+ +`
		`+ ....`
		`+ yum install sigul`
		`+ ....`
		`+ . Ensure that your koji certificate and the link:Fedora-Cert[Fedora CA`
		`+ certificates] are present on the system you're running the sigul client`
		`+ from at the following locations:`
		+ * `~/.fedora.cert`
		+ * `~/.fedora-server-ca.cert`
		+ * `~/.fedora-upload-ca.cert`
		`+ . Admin privileges on koji are required to write signatures.`
		`+`
		`+ === Configuration`
		`+`
		+ . Run `sigul_setup_client`
		`+ . Choose a password for your NSS database. By default this will be`
		+ stored on-disk in `~/.sigul/client.conf`.
		`+ . Choose an export password. You will only need to remember it until`
		+ finishing `sigul_setup_client`.
		`+ . Enter the DB password you chose earlier, then the export password. You`
		+ should see the message `pk12util: PKCS12 IMPORT SUCCESSFUL`
		+ . Enter the DB password again. You should see the message `Done`.
		`+ . Assuming that you are running the sigul client within phx2, edit`
		+ `~/.sigul/client.conf` to include the following lines:
		`+`
		`+ ....`
		`+ [client]`
		`+ bridge-hostname: sign-bridge.phx2.fedoraproject.org`
		`+ server-hostname: sign-vault.phx2.fedoraproject.org`
		`+ ....`
		`+`
		`+ === Updating your Fedora certificate`
		`+`
		`+ When your Fedora certificate expires, after updating it run the`
		`+ following commands:`
		`+`
		`+ ....`
		`+ $ certutil -d ~/.sigul -D -n sigul-client-cert`
		`+ $ sigul_setup_client`
		`+ ....`

modules/release_guide/pages/sop_stage_final_release_for_mirrors.adoc

file added

+38

		`@@ -0,0 +1,38 @@`
		`+ == Stage Final Release for Mirrors`
		`+`
		`+ === Description`
		`+`
		`+ When the release has been fully tested and approved at the "Go/No-Go"`
		`+ meeting it is ready for release to the Fedora mirrors.`
		`+`
		`+ === Action`
		`+`
		`+ . Gather the needed info for running the staging script: Release`
		+ Version: the numerical version number of the release `24` ComposeID: The
		`+ ID of the Compose Label: Compsoe label for the location in stage`
		+ `24_RC-1.2` for example Key: the name of teh release key `fedora-24` or
		+ `fedora-24-secondary` as examples Prerelease: 0 or 1 sets if the release
		`+ goes in test/ or not Arch: <optional> For secondary arches, changes some`
		`+ internal locations`
		`+ +`
		`+ ....`
		`+ $ scripts/stage-release.sh 24 Fedora-24-20160614.0 24_RC-1.2 fedora-24 0`
		`+ ....`
		`+ . Sync the release to the Red Hat internal archive following internally`
		`+ documented`
		`+`
		`+ === Check and set EOL on previous releases to reflect reality`
		`+`
		`+ . check PDC for active releases and respective EOL date`
		`+ . if needed run the adjust-eol-all.py script from releng repository to`
		`+ correct any mistakes`
		`+`
		`+ === Verification`
		`+`
		`+ Verification is somewhat difficult as one cannot look at the content via`
		`+ the web server due to permissions. Typically we ask somebody from the`
		`+ Infrastructure team to give the tree a second set of eyes.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ Hope the release is good!`

modules/release_guide/pages/sop_template.adoc

file added

+22

		`@@ -0,0 +1,22 @@`
		`+ == Standard Operating Procedure Template`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ This is a template file, this can be copied to a new filename in order`
		`+ to start a new document.`
		`+`
		+ New SOP Documents should be added to the `.. toctree::` list in the
		+ `sop.rst` file.
		`+`
		`+ These documents are formatted using sphinx-doc reStructure Text, for`
		`+ markup reference please see: http://sphinx-doc.org/rest.html`
		`+ ====`
		`+`
		`+ === Description`
		`+`
		`+ === Action`
		`+`
		`+ === Verification`
		`+`
		`+ === Consider Before Running`

modules/release_guide/pages/sop_unretire.adoc

file added

+165

		`@@ -0,0 +1,165 @@`
		`+ == Unretiring a package branch`
		`+`
		`+ === Description`
		`+`
		`+ Sometimes, packagers request that we _unretire_ a package branch that`
		`+ has previously been retired.`
		`+`
		`+ This typically happens on the [.title-ref]#rawhide# branch, but could`
		`+ conceivably happen on any stable or arbitrary branch.`
		`+`
		`+ === Action`
		`+`
		`+ ==== Validate Package Ready for Unretirement`
		`+`
		`+ . Verify the package was not retired for any reason, such as legal or`
		`+ license issues, that would prevent it from being re-instated.`
		`+ . Ensure a bugzilla was filed to review the package for unretirement.`
		`+ . Verify with the the requestor exactly which tags they would like`
		`+ unblocked as part of the unretirement request.`
		`+`
		`+ ==== Revert the Retirement Commit`
		`+`
		`+ . Connect to one of the compose systems.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ ssh compose-x86-02.phx2.fedoraproject.org`
		`+ ....`
		`+ ____`
		`+ . Clone the git-dist package using the the proper release engineering`
		`+ credentials.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ GIT_SSH=/usr/local/bin/relengpush fedpkg --user releng clone PACKAGENAME`
		`+ ....`
		`+ ____`
		`+ . Enter the directory of the cloned package and configure the git user`
		`+ information.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ cd PACKAGENAME`
		`+ $ git config --local user.name "Fedora Release Engineering"`
		`+ $ git config --local user.email "releng@fedoraproject.org"`
		`+ ....`
		`+ ____`
		`+ . Git revert the [.title-ref]#dead.package# file commit in dist-git on`
		`+ the particular branch using its commit hash_id. Ensure the commit`
		`+ message contains a URL to the request in pagure.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ git revert -s COMMIT_HASH_ID`
		`+ $ GIT_SSH=/usr/loca/bin/relengpush fedpkg --user releng push`
		`+ ....`
		`+ ____`
		`+`
		`+ ==== Unblock the Package in Koji`
		`+`
		`+ . Check the current state of the branches in koji for the package.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ koji list-pkgs --show-blocked --package=PACKAGENAME`
		`+ ....`
		`+ ____`
		`+ . Unblock each requested tag using koji.`
		`+ +`
		`+ ....`
		`+ $ koji unblock-pkg TAGNAME PACKAGENAME`
		`+ ....`
		`+`
		`+ ==== Verify Package is Not Orphaned`
		`+`
		`+ . Check package ownership`
		`+ +`
		`+ Navigate to [.title-ref]#https://src.fedoraproject.org/# and check`
		`+ package owner.`
		`+ . If the package is orphaned, then give the package to the requestor`
		`+ using the [.title-ref]#give-package# script from the`
		`+ https://pagure.io/releng[Release Engineering Repo].`
		`+ +`
		`+ ....`
		`+ $ ./scripts/distgit/give-package --package=PACKAGENAME --custodian=REQUESTOR`
		`+ ....`
		`+ +`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ This script requires the user to be a member of the group`
		`+ [.title-ref]#cvsadmin# in FAS.`
		`+ ====`
		`+`
		`+ ==== Update Product Definition Center (PDC)`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ If there are more than one tag to be unblocked then the PDC update step`
		`+ should be completed for each tag and package.`
		`+ ====`
		`+`
		`+ . Log into the https://pdc.fedoraproject.org/[Fedora PDC instance] using`
		`+ a FAS account.`
		`+ . Check PDC for the entry for the [.title-ref]#PACKAGENAME# in each`
		`+ [.title-ref]#TAG# that was unblocked in a previous step.`
		`+ +`
		`+ ____`
		`+ ....`
		`+ https://pdc.fedoraproject.org/rest_api/v1/component-branch-slas/?branch=TAG&global_component=PACKAGENAME`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ If no information is returned by this query then it is not in PDC and is`
		`+ likely not yet a branch. The requestor should use the`
		`+ [.title-ref]#fedpkg request-branch# utility to ask for a branch.`
		`+ ====`
		`+ ____`
		`+ . If the package existed within PDC then obtain a token from the PDC`
		`+ site while logged in by navigating to the`
		`+ [.title-ref]#https://pdc.fedoraproject.org/rest_api/v1/auth/token/obtain/#`
		`+ URL with the Firefox web browser.`
		`+ . Press F12 once the page has loaded and select the tab labeled`
		`+ [.title-ref]#Network#. Refresh the web page and find the line whose`
		`+ string in the file column matches`
		`+ [.title-ref]#/rest_api/v1/auth/token/obtain/#.`
		`+ . Right click on specified line and select Copy>Copy as cURL. Paste this`
		`+ into a terminal session and add [.title-ref]#-H "Accept:`
		`+ application/json"#. It should look something similar to the below:`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ curl 'https://pdc.fedoraproject.org/rest_api/v1/auth/token/obtain/' \`
		`+ -H 'Host: pdc.fedoraproject.org' \`
		`+ -H .0) Gecko/20100101 Firefox/57.0' \`
		`+ -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8' \`
		`+ -H 'Accept-Language: en-US,en;q=0.5' \`
		`+ --compressed \`
		`+ -H 'Cookie: csrftoken=CSRF_TOKEN_HASH; SERVERID=pdc-web01; mellon-saml-sesion-cookie=SAML_SESSION_HASH; sessionid=SESSION_ID_HASH' \`
		`+ -H 'Connection: keep-alive' \`
		`+ -H 'Upgrade-Insecure-Requests: 1' \`
		`+ -H 'Cache-Control: max-age=0' \`
		`+ -H "Accept: application/json"`
		`+ ....`
		`+ ____`
		`+ . Using the token obtained from the previous step run the`
		`+ [.title-ref]#adjust-eol.py# script from the`
		`+ https://pagure.io/releng[Release Engineering Repo].`
		`+ +`
		`+ ____`
		`+ ....`
		`+ $ PYTHONPATH=scripts/pdc/ scripts/pdc/adjust-eol.py fedora MYTOKEN PACKAGENAME rpm TAG default -y`
		`+ ....`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ The local machine will have configuration information in the`
		`+ [.title-ref]#/etc/pdc.d/# directory. This is why _fedora_ can be passed`
		`+ as an argument instead of the full API endpoint URL.`
		`+ ====`
		`+ ____`

modules/release_guide/pages/sop_update_critpath.adoc

file added

+55

		`@@ -0,0 +1,55 @@`
		`+ == Update Critpath`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ Critpath = "Critical Path"`
		`+`
		`+ This is a collection of packages deemed "critical" to Fedora`
		`+ ====`
		`+`
		`+ === Description`
		`+`
		`+ PDC has information about which packages are critpath and which are not.`
		`+ A script that reads the yum repodata (critpath group in comps, and the`
		`+ package dependencies) is used to generate this. Since package`
		`+ dependencies change, this list should be updated periodically.`
		`+`
		`+ === Action`
		`+`
		`+ . Release engineering scripts for updating critpath live in the`
		`+ https://pagure.io/releng[releng git repository].`
		`+ . Check the critpath.py script to see if the list of releases needs to`
		`+ be updated:`
		`+ +`
		`+ ....`
		`+ for r in ['12', '13', '14', '15', '16', '17']: # 13, 14, ...`
		`+ releasepath[r] = 'releases/%s/Everything/$basearch/os/' % r`
		`+ updatepath[r] = 'updates/%s/$basearch/' % r`
		`+`
		`+ # Branched Fedora goes here`
		`+ branched = '18'`
		`+ ....`
		`+ +`
		`+ The for loop has the version numbers for releases that have gone past`
		`+ final. branched has the release that's been branched from rawhide but`
		`+ not yet hit final. (These have different paths in the repository and may`
		`+ not have an updates directory, thus they're in separate sections).`
		`+ . Run the script with the release to generate info for (for a release`
		`+ that's hit final, this is the release number example: "17". For`
		`+ branched, it's "branched").`
		`+ +`
		`+ ....`
		`+ ./critpath.py --srpm -o critpath.txt branched`
		`+ ....`
		`+ . Run the update script to add that to PDC:`
		`+ +`
		`+ ....`
		`+ ./update-critpath --user toshio f18 critpath.txt`
		`+ ....`
		`+ +`
		`+ The username is your fas username. You must be in cvsadmin to be able to`
		`+ change this. The branch is the name of the dist-git branch. critpath.txt`
		`+ is the file that the output of critpath.py went into. The script needs a`
		`+ PDC token to talk to the server, configured in /etc/pdc.d/. See the PDC`
		`+ SOP for more info.`

modules/release_guide/pages/sop_update_releng_docs.adoc

file added

+54

		`@@ -0,0 +1,54 @@`
		`+ == Update RelEng Rendered Docs`
		`+`
		`+ === Description`
		`+`
		`+ When an improvement happens to the Release Engineering documentation`
		+ following the `contributing <contributing>` for the
		`+ http://sphinx-doc.org/[Sphinx]`
		`+ https://en.wikipedia.org/wiki/ReStructuredText[reStructured Text] source`
		+ found in `docs/source` within the https://pagure.io/releng[RelEng git
		`+ repository] someone has to manually perform a process in order to update`
		`+ the documentation that is hosted in the https://pagure.io/pagure[pagure]`
		`+ documentation space for https://docs.pagure.org/releng/[Fedora RelEng`
		`+ docs].`
		`+`
		`+ === Action`
		`+`
		`+ In order to render the documentation using`
		`+ http://sphinx-doc.org/[Sphinx], you need to first be sure to have the`
		`+ package installed:`
		`+`
		`+ ....`
		`+ $ dnf install python-sphinx`
		`+ ....`
		`+`
		`+ Then we'll need to clone the RelEng repository and the RelEng docs`
		`+ repository (the docs git repository is provided by pagure`
		`+ automatically). There is a script in the [.title-ref]#releng# repository`
		`+ that takes care of cleanly updating the documentation site for us.`
		`+`
		`+ ....`
		`+ $ ./scripts/update-docs.sh`
		`+ ....`
		`+`
		`+ The documentation is now live.`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ This will require someone with permissions to push to the rawhide branch`
		`+ for the releng repository. If you are curious whom all has this ability,`
		+ please refer to the `Main Page <index>` and contact someone from the
		`+ "Team Composition"`
		`+ ====`
		`+`
		`+ === Verification`
		`+`
		`+ Visit the https://docs.pagure.org/releng/[Fedora RelEng docs] website`
		`+ and verify that the changes are reflected live on the docs site.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ No considerations at this time. The docs git repository is simply a`
		`+ static html hosting space and we can just re-render the docs and push to`
		`+ it again if necessary.`

modules/release_guide/pages/sop_updating_comps.adoc

file added

+53

		`@@ -0,0 +1,53 @@`
		`+ == Updating Comps`
		`+`
		`+ === Description`
		`+`
		`+ When we start a new Fedora development cycle (when we branch rawhide) we`
		`+ have to create a new comps file for the new release. This SOP covers`
		`+ that action.`
		`+`
		`+ === Action`
		`+`
		`+ . clone the comps repo`
		`+ +`
		`+ ....`
		`+ $ git clone ssh://git@pagure.io/fedora-comps.git`
		`+ ....`
		`+ . Create the new comps file for next release:`
		`+ +`
		`+ ....`
		`+ $ cp comps-f24.xml.in comps-f25.xml.in`
		`+ ....`
		`+ . Edit Makefile to update comps-rawhide target`
		`+ +`
		`+ ....`
		`+ - -comps-rawhide: comps-f24.xml`
		`+ - - @mv comps-f24.xml comps-rawhide.xml`
		`+ +comps-rawhide: comps-f25.xml`
		`+ + @mv comps-f25.xml comps-rawhide.xml`
		`+ ....`
		`+ . Add the new comps file to source control:`
		`+ +`
		`+ ....`
		`+ $ git add comps-f25.xml.in`
		`+ ....`
		`+ . Edit the list of translated comps files in po/POTFILES.in to reflect`
		`+ currently supported releases.`
		`+ +`
		`+ ....`
		`+ -comps-f22.xml`
		`+ +comps-f25.xml`
		`+ ....`
		`+ . Send it up:`
		`+ +`
		`+ ::::`
		`+ $ git push`
		`+`
		`+ === Verification`
		`+`
		`+ One can review the logs for rawhide compose after this change to make`
		`+ sure the right comps file was used.`
		`+`
		`+ === Consider Before Running`
		`+`
		`+ Nothing yet.`

modules/release_guide/pages/troubleshooting.adoc

file added

+260

		`@@ -0,0 +1,260 @@`
		`+ == Fedora Release Engineering Troubleshooting Guide`
		`+`
		`+ Fedora Release Engineering consists of many different systems, many`
		`+ different code bases and multiple tools. Needless to say, things can get`
		`+ pretty complex in a hurry. This aspect of Fedora Release Engineering is`
		`+ not very welcoming to newcomers who would like to get involved. This`
		`+ guide stands as a place to educate those new to the processes, systems,`
		`+ code bases, and tools. It also is to serve as a reference to those who`
		`+ aren't new but maybe are fortunate enough to not have needed to diagnose`
		`+ things in recent memory.`
		`+`
		`+ We certainly won't be able to document every single possible compontent`
		`+ in the systems that could go wrong but hopefully over time this document`
		`+ will stand as a proper knowledge base for reference and educational`
		`+ purposes on the topics listed below.`
		`+`
		`+ === Compose`
		`+`
		`+ If something with a compose has gone wrong, there's a number of places`
		`+ to find information. Each of these are discussed below.`
		`+`
		`+ ==== releng-cron list`
		`+`
		`+ The compose output logs are emailed to the releng-cron mailing list. It`
		`+ is good practice to check the`
		`+ https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/[releng-cron`
		`+ mailing list archives] and find the latest output and give it a look.`
		`+`
		`+ ==== compose machines`
		`+`
		`+ If the`
		`+ https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/[releng-cron`
		`+ mailing list archives] doesn't prove to be useful, you can move on to`
		`+ checking the contents of the composes themselves on the primary compose`
		`+ machines in the Fedora Infrastructure. At the time of this writing,`
		`+ there are multiple machines based on the specific compose you are`
		`+ looking for:`
		`+`
		`+ * Two-Week Atomic Nightly Compose`
		+ ** `compose-x86-01.phx2.fedoraproject.org`
		`+ * Branched Compose`
		+ ** `branched-composer.phx2.fedoraproject.org`
		`+ * Rawhide Compose`
		+ ** `rawhide-composer.phx2.fedoraproject.org`
		`+`
		`+ Depending on which specific compose you are in search of will depend on`
		`+ what full path you will end up inspecting:`
		`+`
		`+ * For Two Week Atomic you will find the compose output in`
		+ `/mnt/fedora_koji/compose/`
		`+ * For Release Candidate / Test Candidate composes you will find compose`
		+ output in `/mnt/koji/compose/`
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ It's possible the mock logs are no longer available. The mock chroots`
		`+ are rewritten on subsequent compose runs.`
		`+ ====`
		`+`
		`+ You can also check for mock logs if they are still persisting from the`
		`+ compose you are interested in. Find the specific mock chroot directory`
		+ name (that will reside in `/var/lib/mock/`) you are looking for by
		`+ checking the appropriate compose mock configuration (the following is`
		`+ only a sample provided at the time of this writing):`
		`+`
		`+ ....`
		`+ $ ls /etc/mock/compose`
		`+ /etc/mock/fedora-22-compose-aarch64.cfg /etc/mock/fedora-branched-compose-aarch64.cfg`
		`+ /etc/mock/fedora-22-compose-armhfp.cfg /etc/mock/fedora-branched-compose-armhfp.cfg`
		`+ /etc/mock/fedora-22-compose-i386.cfg /etc/mock/fedora-branched-compose-i386.cfg`
		`+ /etc/mock/fedora-22-compose-x86_64.cfg /etc/mock/fedora-branched-compose-x86_64.cfg`
		`+ /etc/mock/fedora-23-compose-aarch64.cfg /etc/mock/fedora-rawhide-compose-aarch64.cfg`
		`+ /etc/mock/fedora-23-compose-armhfp.cfg /etc/mock/fedora-rawhide-compose-armhfp.cfg`
		`+ /etc/mock/fedora-23-compose-i386.cfg /etc/mock/fedora-rawhide-compose-i386.cfg`
		`+ /etc/mock/fedora-23-compose-x86_64.cfg /etc/mock/fedora-rawhide-compose-x86_64.cfg`
		`+ ....`
		`+`
		`+ ==== running the compose yourself`
		`+`
		`+ If you happen to strike out there and are still in need of debugging, it`
		`+ might be time to just go ahead and run the compose yourself. The exact`
		`+ command needed can be found in the cron jobs located on their respective`
		`+ compose machines, this information can be found in the`
		+ `compose-machines` section. Also note that each respective compose
		`+ command should be ran from it's respective compose machine as outlined`
		+ in the `compose-machines` section.
		`+`
		`+ An example is below, setting the compose directory as your`
		+ `username-debug.1`, the `.1` at the end is common notation for an
		`+ incremental run of a compose. If you need to do another, increment it to`
		+ `.2` and continue. This is helpful to be able to compare composes.
		`+`
		`+ [NOTE]`
		`+ .Note`
		`+ ====`
		`+ It is recommended that the following command be run in`
		`+ https://www.gnu.org/software/screen/[screen] or`
		`+ https://tmux.github.io/[tmux]`
		`+ ====`
		`+`
		`+ ....`
		+ $ TMPDIR=`mktemp -d /tmp/twoweek.XXXXXX` && cd $TMPDIR \
		`+ && git clone -n https://pagure.io/releng.git && cd releng && \`
		`+ git checkout -b stable twoweek-stable && \`
		`+ LANG=en_US.UTF-8 ./scripts/make-updates 23 updates $USER-debug.1`
		`+ ....`
		`+`
		+ The above command was pulled from the `twoweek-atomic` cron job with
		`+ only the final parameter being altered. This is used as the name of the`
		`+ output directory.`
		`+`
		`+ The compose can take some time to run, so don't be alarmed if you don't`
		`+ see output in a while. This should provide you all the infromation`
		`+ needed to debug and/or diagnose further. When in doubt, as in`
		+ `#fedora-releng` on `irc.libera.chat`.
		`+`
		`+ === Docker Layered Image Build Service`
		`+`
		`+ The`
		`+ https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service[Docker`
		`+ Layered Image Build Service] is built using a combination of`
		`+ technologies such as https://www.openshift.org/[OpenShift],`
		`+ https://github.com/projectatomic/osbs-client[osbs-client], and the`
		`+ https://github.com/release-engineering/koji-containerbuild[koji-containerbuild]`
		`+ plugin that when combined are often refered to as an OpenShift Build`
		`+ Service instance (OSBS). Something to note is that`
		`+ https://www.openshift.org/[OpenShift] is a`
		`+ http://kubernetes.io/[kubernetes] distribution with many features built`
		`+ on top of http://kubernetes.io/[kubernetes] such as the`
		`+ https://docs.openshift.org/latest/dev_guide/builds.html[build primitive]`
		`+ that is used as the basis for the build service. This information will`
		`+ hopefully shed light on some of the terminology and commands used below.`
		`+`
		`+ There are a few "common" scenarios in which build may fail or hang that`
		`+ will need some sort of inspection of the build system.`
		`+`
		`+ ==== Build Appears to stall after being scheduled`
		`+`
		`+ In the event that a build scheduled through koji appears to be stalled`
		+ and is not in a `free` state (i.e. - has been scheduled). An
		+ administrator will need to ssh into `osbs-master01` or
		+ `osbs-master01.stg` (depending stage vs production) and inspect the
		`+ build itself.`
		`+`
		`+ ....`
		`+ $ oc status`
		`+ In project default on server https://10.5.126.216:8443`
		`+`
		`+ svc/kubernetes - 172.30.0.1 ports 443, 53, 53`
		`+`
		`+ bc/cockpit-f24 custom build of git://....`
		`+ build #8 succeeded 7 weeks ago`
		`+ build #9 failed 33 minutes ago`
		`+`
		`+ $ oc describe build/cockpit-f24-9`
		`+ # lots of output about status of the specific build`
		`+`
		`+ $ oc logs build/cockpit-f24-9`
		`+ # extremely verbose logs, these should in normal circumstances be found in`
		`+ # the koji build log post build`
		`+ ....`
		`+`
		`+ The information found in the commands above will generally identify the`
		`+ issue.`
		`+`
		`+ ==== Build fails but there's no log output in the Koji Task`
		`+`
		`+ Sometimes there is a communications issue between Koji and OSBS which`
		`+ cause for a failure to be listed in Koji but without all the logs. These`
		+ can be diagnosed by checking the `kojid` logs on the Koji builder listed
		`+ in the task output.`
		`+`
		`+ For example:`
		`+`
		`+ ....`
		`+ $ fedpkg container-build`
		`+ Created task: 90123598`
		`+ Task info: http://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90123598`
		`+ Watching tasks (this may be safely interrupted)...`
		`+ 90123598 buildContainer (noarch): free`
		`+ 90123598 buildContainer (noarch): free -> open (buildvm-04.stg.phx2.fedoraproject.org)`
		`+ 90123599 createContainer (x86_64): free`
		`+ 90123599 createContainer (x86_64): free -> open (buildvm-02.stg.phx2.fedoraproject.org)`
		`+ 90123599 createContainer (x86_64): open (buildvm-02.stg.phx2.fedoraproject.org) -> closed`
		`+ 0 free 1 open 1 done 0 failed`
		`+ 90123598 buildContainer (noarch): open (buildvm-04.stg.phx2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: 'Image build failed. OSBS build id: cockpit-f24-9'>`
		`+ 0 free 0 open 1 done 1 failed`
		`+`
		`+ 90123598 buildContainer (noarch) failed`
		`+ ....`
		`+`
		`+ In this example the buildContiner task was scheduled and ran on`
		+ `buildvm-04.stg` with the actual createContainer task being on
		+ `buildvm-02.stg`, and `buildvm-02.stg` is where we're going to want to
		`+ begin looking for failures to communicate with OSBS as this is the point`
		`+ of contact with the external system.`
		`+`
		+ Logs can be found in `/var/log/kojid.log` or if necessary, check the
		`+ koji hub in question. Generally, you will want to start with the first`
		`+ point of contact with OSBS and "work your way back" so in the above`
		+ example you would first check `buildvm-02.stg`, then move on to
		+ `buildvm-04.stg` if nothing useful was found in the logs of the previous
		`+ machine, and again move on to the koji hub if neither of the builder`
		`+ machines involved provided useful log information.`
		`+`
		`+ ==== Build fails because it can't get to a network resource`
		`+`
		`+ Sometimes there is a situation where the firewall rules get messed up on`
		`+ one of the OpenShift Nodes in the environment. This can cause output`
		`+ similar to the following:`
		`+`
		`+ ....`
		`+ $ fedpkg container-build --scratch`
		`+ Created task: 90066343`
		`+ Task info: http://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90066343`
		`+ Watching tasks (this may be safely interrupted)...`
		`+ 90066343 buildContainer (noarch): free`
		`+ 90066343 buildContainer (noarch): free -> open (buildvm-03.stg.phx2.fedoraproject.org)`
		`+ 90066344 createContainer (x86_64): open (buildvm-04.stg.phx2.fedoraproject.org)`
		`+ 90066344 createContainer (x86_64): open (buildvm-04.stg.phx2.fedoraproject.org) -> FAILED: Fault: <Fault 2001: "Image build failed. Error in plugin distgit_fetch_artefacts: OSError(2, 'No such file or directory'). OSBS build id: scratch-20161102132628">`
		`+ 0 free 1 open 0 done 1 failed`
		`+ 90066343 buildContainer (noarch): open (buildvm-03.stg.phx2.fedoraproject.org) -> closed`
		`+ 0 free 0 open 1 done 1 failed`
		`+ ....`
		`+`
		`+ If we go to the OSBS Master and run the following commands, we will see`
		`+ the root symptom:`
		`+`
		`+ ....`
		`+ # oc logs build/scratch-20161102132628`
		`+ Error from server: Get https://osbs-node02.stg.phx2.fedoraproject.org:10250/containerLogs/default/scratch-20161102132628-build/custom-build: dial tcp 10.5.126.213:10250: getsockopt: no route to host`
		`+`
		`+ # ping 10.5.126.213`
		`+ PING 10.5.126.213 (10.5.126.213) 56(84) bytes of data.`
		`+ 64 bytes from 10.5.126.213: icmp_seq=1 ttl=64 time=0.299 ms`
		`+ 64 bytes from 10.5.126.213: icmp_seq=2 ttl=64 time=0.299 ms`
		`+ 64 bytes from 10.5.126.213: icmp_seq=3 ttl=64 time=0.253 ms`
		`+ 64 bytes from 10.5.126.213: icmp_seq=4 ttl=64 time=0.233 ms`
		`+ ^C`
		`+ --- 10.5.126.213 ping statistics ---`
		`+ 4 packets transmitted, 4 received, 0% packet loss, time 3073ms`
		`+ rtt min/avg/max/mdev = 0.233/0.271/0.299/0.028 ms`
		`+`
		`+ # http get 10.5.126.213:10250`
		`+`
		`+ http: error: ConnectionError: HTTPConnectionPool(host='10.5.126.213', port=10250): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fdab064b320>: Failed to establish a new connection: [Errno 113] No route to host',)) while doing GET request to URL: http://10.5.126.213:10250/`
		`+ ....`
		`+`
		`+ In the above output, we can see that we do actually have network`
		`+ connectivity to the Node but we can not connect to the OpenShift service`
		+ that should be listening on port `10250`.
		`+`
		`+ To fix this, you need to ssh into the OpenShift Node that you can't`
		+ connect to via port `10250` and run the following commands. This should
		`+ resolve the issue.`
		`+`
		`+ ....`
		`+ iptables -F && iptables -t nat -F && systemctl restart docker && systemctl restart origin-node`
		`+ ....`

modules/release_guide/pages/workflow_automation.adoc

file added

+114

		`@@ -0,0 +1,114 @@`
		`+ == Fedora RelEng Workflow Automation`
		`+`
		`+ The Fedora RelEng Workflow Automation is a means to allow RelEng to`
		`+ define a pattern by which Release Engineering work is automated in an`
		`+ uniform fashion. The automation technology of choice is`
		`+ https://ansible.com/[ansible] and the "workflow engine" is powered by`
		`+ https://github.com/maxamillion/loopabull[loopabull], which is an event`
		`+ loop that allows us to pass the information contained within a`
		`+ http://www.fedmsg.com/en/latest/[fedmsg] and insert it into`
		`+ https://ansible.com/[ansible]`
		`+ https://docs.ansible.com/ansible/playbooks.html[playbooks]. This will`
		`+ effectively create an event driven workflow that can take action`
		`+ conditionally based on the contents of arbitrary`
		`+ http://www.fedmsg.com/en/latest/[fedmsg] data.`
		`+`
		`+ Background on the topic can be found in the`
		`+ https://fedoraproject.org/wiki/Changes/ReleaseEngineeringAutomationWorkflowEngine[Release`
		`+ Engineering Automation Workflow Engine] Change proposal, as well as in`
		`+ the https://pagure.io/releng-automation[releng-automation] pagure`
		`+ repository.`
		`+`
		`+ === RelEng Workflow Automation Architecture`
		`+`
		`+ By using http://www.fedmsg.com/en/latest/[fedmsg] as the source of`
		`+ information feeding the event loop, we will configure`
		`+ https://github.com/maxamillion/loopabull[loopabull] to listen for`
		`+ specific`
		`+ https://fedora-fedmsg.readthedocs.io/en/latest/topics.html[fedmsg`
		`+ topics] which will correspond with https://ansible.com/[ansible]`
		`+ https://docs.ansible.com/ansible/playbooks.html[playbooks]. When one of`
		`+ the appropriate`
		`+ https://fedora-fedmsg.readthedocs.io/en/latest/topics.html[fedmsg`
		`+ topics] is encountered across the message bus, it's message payload is`
		`+ then injected into the corresponding playbook as an extra set of`
		`+ variables. A member of the Fedora Release Engineering Team can at that`
		`+ point use this as a means to perform whatever arbitrary action or series`
		`+ of actions they can otherwise perform with https://ansible.com/[ansible]`
		`+ (including what we can enable via custom`
		`+ https://docs.ansible.com/ansible/modules.html[modules]) based on the`
		`+ input of the message payload.`
		`+`
		`+ The general overview of the architecture is below as well as a`
		`+ description of how it works:`
		`+`
		`+ ....`
		`+ +------------+`
		`+ \| fedmsg \|`
		`+ \| \|`
		`+ +---+--------+`
		`+ \| ^`
		`+ \| \|`
		`+ \| \|`
		`+ \| \|`
		`+ \| \|`
		`+ \| \|`
		`+ V \|`
		`+ +------------------+-----------------+`
		`+ \| \|`
		`+ \| Release Engineering \|`
		`+ \| Workflow Automation Engine \|`
		`+ \| \|`
		`+ \| - RabbitMQ \|`
		`+ \| - fedmsg-rabbitmq-serializer \|`
		`+ \| - loopabull \|`
		`+ \| \|`
		`+ +----------------+-------------------+`
		`+ \|`
		`+ \|`
		`+ \|`
		`+ \|`
		`+ V`
		`+ +-----------------------+`
		`+ \| \|`
		`+ \| composer/bodhi/etc \|`
		`+ \| \|`
		`+ +-----------------------+`
		`+ ....`
		`+`
		`+ The flow of data will begin with an event somewhere in the`
		`+ https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]`
		`+ that sends a http://www.fedmsg.com/en/latest/[fedmsg] across the message`
		`+ bus, then the messages will be taken in and serialized in to a`
		`+ https://www.rabbitmq.com/[rabbitmq] worker queue using`
		`+ https://pagure.io/fedmsg-rabbitmq-serializer[fedmsg-rabbitmq-serializer].`
		`+ Then https://github.com/maxamillion/loopabull[loopabull] will be`
		`+ listening to the rabbitmq worker queue for tasks to come in. Once a`
		`+ message is recieved, it is processed and once it is either no-op'd or a`
		`+ corresponding ansible playbook is run to completion, the message will be`
		+ `ack`'d and cleared from the worker queue. This will allow for us to
		`+ scale loopabull instances independently from the message queue as well`
		`+ as ensure that work is not lost because of a downed or busy loopabull`
		`+ instance. Also, as a point of note, the loopabull service instances will`
		`+ be scaled using https://freedesktop.org/wiki/Software/systemd/[systemd]`
		`+ https://fedoramagazine.org/systemd-template-unit-files/[unit templates].`
		`+`
		`+ Once a playbook has been triggered, it will run tasks on remote systems`
		`+ on behalf of a loopabull automation user. These users can be privileged`
		`+ if need be, however the scope of their privilege is based on the purpose`
		`+ they serve. These user accounts are provisioned by the`
		`+ https://fedoraproject.org/wiki/Infrastructure[Fedora Infrastructure]`
		`+ Team based on the requirements of the`
		+ `RelEng Task Automation User Request Standard Operating
		+ Procedure (SOP) <sop_requesting_task_automation_user>` document and
		`+ tasks are subject to code and security audit.`
		`+`
		`+ === Fedora Lib RelEng`
		`+`
		`+ https://pagure.io/flr[Fedora Lib RelEng] (flr), is a library and set of`
		`+ command line tools to expose the library that aims to provide re-usable`
		`+ code for common tasks that need to be done in Release Engineering.`
		`+ Combining this set of command line tools when necessary with the Release`
		`+ Engineering Automation pipeline allows for easy separation of`
		`+ permissions and responsibilities via sudo permissions on remote hosts.`
		`+ This is explained in more detail on the project's pagure page.`

infra-docs-fpo

#179 Initial import of releng SOPs

Merged a year ago by humaton. Opened a year ago by humaton.

humaton/infra-docs-fpo release_guide into master

Metadata

Changes Summary 63

infra-docs-fpo

Source Code

#179 Initial import of releng SOPs Merged a year ago by humaton. Opened a year ago by humaton. humaton/infra-docs-fpo release_guide into master

Metadata

Changes Summary 63

#179 Initial import of releng SOPs

Merged a year ago by humaton. Opened a year ago by humaton.

humaton/infra-docs-fpo release_guide into master