From 7129a801647b1a4771bbd97605eb7c43f20ceeba Mon Sep 17 00:00:00 2001 From: Dusty Mabe Date: Aug 08 2017 20:00:58 +0000 Subject: Add docs on how fedora builds OSTrees. --- diff --git a/docs/how-fedora-creates-ostrees.md b/docs/how-fedora-creates-ostrees.md new file mode 100644 index 0000000..95b44b3 --- /dev/null +++ b/docs/how-fedora-creates-ostrees.md @@ -0,0 +1,251 @@ +# How Do We Create OSTree Repos and Artifacts in Fedora + +# Introduction + +**NOTE:** For background on OSTree check out the + [docs](https://ostree.readthedocs.io/en/latest/). + +When you want to create a new OSTree using `rpm-ostree` you usually define +a few yum repos, and then a json file that says what rpms you +want to be composed in the tree. You then run an `rpm-ostree compose +tree` command to create the commit in the ostree repo. Once the +ostree commit has been created you can then create installer images +(ISOs) and cloud/VM images (qcow, etc) from that ostree. + +How does Fedora do this? It's a bit complicated, but I'll try to cover +the bases. + +First I'll give you a little high level view of the way things are +built before major release, all using Pungi. I'll then try to dig into +*The LEGOs* that are used to build the artifacts we care about. +Finally, I'll try to explain how things are different after a major +release is out and how we end up getting updated rpms and eventually +two week releases out the door. + + +# Before Fedora Major Release: Building Everything Using Pungi + +Prior to any major release of Fedora the release engineering team *builds the +whole world* every night. This is done using a tool called +[Pungi](https://pagure.io/pungi). A [script](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/nightly.sh) +is called to kick off the nightly run via a [cron job](https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/releng/files/branched). +The Pungi config that is used is the [fedora.conf](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf) +file from the [pungi-fedora](https://pagure.io/pungi-fedora) git repo. + +For Fedora 26 you can see where the ostree for Atomic Host is defined +in the Pungi config, [here](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_710-728), +in the **ostree** section. During a compose you can +see that the ostree will get placed into `"ostree_repo": "/mnt/koji/compose/atomic/26/",`. +After the compose has finished it will get synced to `/mnt/koji/atomic/26/` which corresponds +to the public URL https://kojipkgs.fedoraproject.org/atomic/26/. + +During the compose, once the ostree commit has been created, the +installer ISO and the cloud images can get created. The definition for +the [installer ISO](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_741-761) +and the [cloud images](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_355-383) +are all within the same Pungi config. + +# The LEGOs + +There are a few building blocks we need to talk about before we really jump into +how this whole thing is put together. First, we have to talk about how the OSTree +gets created and then the artifacts that are derived from it. + +## The RPM OSTree + +To create the OSTree Pungi ends up calling out to a [Koji](https://pagure.io/koji) +runroot task. This runroot task essentially runs a command and harvests the results. The +command that eventually gets run is an [rpm-ostree command](https://pagure.io/pungi/blob/381d08a81079581595e9e48174267b7e7bbd263b/f/pungi/ostree/tree.py#_31-32) +that looks something like this: + +```nohighlight +rpm-ostree compose tree --repo=/mnt/koji/compose/atomic/26/ \ +--write-commitid-to=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/logs/x86_64/Atomic/ostree-2/commitid.log \ +/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/ostree-2/config_repo/fedora-atomic-docker-host.json +``` + +From the [ostree part of the Pungi config](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_710-728) +you can see where some of the arguments for this command came from: + +``` +ostree = [ + ("^Atomic$", { + "x86_64": { + "treefile": "fedora-atomic-docker-host.json", + "config_url": "https://pagure.io/fedora-atomic.git", + "config_branch": "f26", + "source_repo_from": "Everything", + "ostree_repo": "/mnt/koji/compose/atomic/26/", + 'failable': ['*'], + } + }), +``` + +We store our inputs inputs to `rpm-ostree compose tree` in +the [fedora-atomic](https://pagure.io/fedora-atomic.git) git repo. The +[fedora-26.repo](https://pagure.io/fedora-atomic/blob/d79a03ecf213843d2cbff2145c88865d3e898183/f/fedora-26.repo) +file defines the dnf repositories and the +[fedora-atomic-host.json]([https://pagure.io/fedora-atomic/blob/d79a03ecf213843d2cbff2145c88865d3e898183/f/fedora-atomic-host.json) +file defines which of those repositories to use, which rpms to pull +from them, and a few other things. + + +## The Installer ISO + +For the installer ISO Pungi again calls out to a Koji runroot task. The +command that eventually gets run is a [lorax](https://github.com/rhinstaller/lorax) +command that looks something like this: + +```nohighlight +lorax --product=Fedora --version=26 --release=20170705.n.0 \ +--source=http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170705.n.0/compose/Everything/x86_64/os \ +--variant=Atomic --nomacboot --volid=Fedora-Atomic-ostree-x86_64-26 \ +--installpkgs=fedora-productimg-atomic \ +--add-template=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/lorax_templates/ostree-based-installer/lorax-configure-repo.tmpl \ +--add-template=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/lorax_templates/ostree-based-installer/lorax-embed-repo.tmpl \ +--add-template-var=ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/atomic/26/ \ +--add-template-var=ostree_update_repo=https://kojipkgs.fedoraproject.org/atomic/26/ \ +--add-template-var=ostree_osname=fedora-atomic \ +--add-template-var=ostree_install_ref=fedora/26/x86_64/atomic-host \ +--add-template-var=ostree_update_ref=fedora/26/x86_64/atomic-host \ +--logfile=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/logs/x86_64/Atomic/ostree_installer-1/lorax.log \ +--rootfs-size=3 /mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/ostree_installer +``` + +From the [installer ISO](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_741-761) +part of the Pungi config you can see some configuration that was set +and eventually translated into the lorax command above. + +``` +ostree_installer = [ + ("^Atomic$", { + "x86_64": { + "source_repo_from": "Everything", + "release": None, + "rootfs_size": "3", + "installpkgs": ["fedora-productimg-atomic"], + "add_template": ["ostree-based-installer/lorax-configure-repo.tmpl", + "ostree-based-installer/lorax-embed-repo.tmpl"], + "add_template_var": [ + "ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/atomic/26/", + "ostree_update_repo=https://kojipkgs.fedoraproject.org/atomic/26/", + "ostree_osname=fedora-atomic", + "ostree_install_ref=fedora/26/x86_64/atomic-host", + "ostree_update_ref=fedora/26/x86_64/atomic-host", + ], + 'template_repo': 'https://pagure.io/fedora-lorax-templates.git', + 'template_branch': 'f26', + 'failable': ['*'], + } + }), +``` + +There are a few lorax templates we are passing in as well as some variables +to those templates. The templates are stored in the [fedora-lorax-templates](https://pagure.io/fedora-lorax-templates.git) +git repo. + + +## The Cloud Images + +For the cloud images Koji has higher level support for building +them than it does for the installer ISO or for creating the ostree. +It doesn't need to run things in a runroot, which is generic, but +rather it can create an ImageBuild task to create an image. + +This is still all defined in the Pungi config. The `image-build` sections +for Atomic Host are [here](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_355-383) +in the Pungi config. One of these sections looks like: + +``` + 'image-build': { + 'format': [('qcow2','qcow2'), ('raw-xz','raw.xz')], + 'name': 'Fedora-Atomic', + 'kickstart': 'fedora-atomic.ks', + 'distro': 'Fedora-22', + 'disk_size': 6, + 'arches': ['x86_64'], + 'install_tree_from': 'Cloud', + 'subvariant': 'Atomic', + 'failable': ['*'], + } +``` + +You'll notice that we say what kickstart file to use, but we don't define +where to pull the kickstart file from. The git repo for the kickstarts is +defined along with a few other varialbes [earlier in the file](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_266-271). + +# After Fedora Major Release: Bodhi + Pungi + Release + +Once Fedora is officially released for a particular version there is a +*release day* yum/dnf repository that is made from the rpms that were +stable on the day of release. This repository is frozen and will not +change. + +There are two more repositories that become significant now. These are +the **updates** and the **updates-testing** repositories. The **updates** +repo contains packages that have passed testing and are available to Fedora users +whenever they run `dnf update`. The **updates-testing** repo is for +packages that have been built and have been submitted as an update for +people to test to make sure nothing is broken before graduating to **updates**. +The **updates-testing** repo is not enabled by default. A user would +have to willingly enable it for the purpose of doing testing. + +All of that is to say that there are other repos that exist after +release day that are for updated rpms. + +## Bodhi + +Within Fedora there is a tool known as [Bodhi](https://bodhi.fedoraproject.org/) +that is responsible for tracking what state particular packages are in +and moving them between the **updates** and **updates-testing** repos. +As part of this, Bodhi is currently responsible for creating the +repositories and also the OSTree commits from the new content that +it just created a repo for. Bodhi was the most logical candidate for +this at the time it was implemented because we wanted to create a new +commit as soon as we could grab new content (right after the +repositories are created). + +After creating the OSTree content it gets synced to `fedora/26/x86_64/updates/atomic-host` +ref within the OSTree repository at https://kojipkgs.fedoraproject.org/atomic/26/. +This is a global location/ref that users could pull from for existing +installed systems. + +## Pungi + +When Bodhi runs and creates a new **updates** repo it also creates a +new OSTree. When we do an official two week release we release other +artifacts (ISOs, Qcows, etc) too. These get built once a day and uses the +[fedora-atomic.conf](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora-atomic.conf) +Pungi config (which gets called from the +[twoweek-nightly.sh](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/twoweek-nightly.sh) +script). Basically the config creates repos from the `f26-atomic` Koji tag +which inherits from the `f26` tag (release day tag). It creates some +repos from rpms in the `Cloud` variant and then builds installer +images and qcows based on those rpms. The installer images and the +qcows actually pull the OSTree from the commit that was generated +by the last Bodhi run, though. So the yum/dnf repository created +during the Pungi run only affects the installer, not the installed +OSTree in the installed systems. + +## Performing A Two Week Atomic Host Release + +So with the OSTree created by Bodhi and the other artifacts created by +Pungi we can now test do an official two week Atomic Host release. +This is currently run by a member of releng and the process is +documented by [this](https://docs.pagure.org/releng/sop_two_week_atomic.html) +Standard Operating Procedure guide. It eventually tells the operator +to run [this releng script](https://pagure.io/releng/blob/e1fa88d3937412ca0f3c5d166f1b82c5106b1256/f/scripts/push-two-week-atomic.py) +to do the release. The script does several things but most noteworthy +is updating the `fedora/26/x86_64/atomic-host` ref within the OSTree +repo, syncing out the ISOs/qcows to the mirrors, and updating the +[website](https://getfedora.org/atomic/download/). + +# Conclusion + +Our processes differ slightly for before and after Fedora release. +The entire pipeline involves Koji, Bodhi, and Pungi, and many tools +underneath the covers. There is a lot of working being done to try to improve these +processes including [calling Pungi from Bodhi](https://pagure.io/atomic-wg/issue/300) +in the short term and possibly [building Atomic Host from a Module](https://pagure.io/atomic-wg/issue/312) +in the longer term. Please reach out to us with any questions about +this whole process in #atomic on freenode.