|
|
7129a80 |
# How Do We Create OSTree Repos and Artifacts in Fedora
|
|
|
7129a80 |
|
|
|
7129a80 |
# Introduction
|
|
|
7129a80 |
|
|
|
7129a80 |
**NOTE:** For background on OSTree check out the
|
|
|
7129a80 |
[docs](https://ostree.readthedocs.io/en/latest/).
|
|
|
7129a80 |
|
|
|
7129a80 |
When you want to create a new OSTree using `rpm-ostree` you usually define
|
|
|
7129a80 |
a few yum repos, and then a json file that says what rpms you
|
|
|
7129a80 |
want to be composed in the tree. You then run an `rpm-ostree compose
|
|
|
7129a80 |
tree` command to create the commit in the ostree repo. Once the
|
|
|
7129a80 |
ostree commit has been created you can then create installer images
|
|
|
7129a80 |
(ISOs) and cloud/VM images (qcow, etc) from that ostree.
|
|
|
7129a80 |
|
|
|
7129a80 |
How does Fedora do this? It's a bit complicated, but I'll try to cover
|
|
|
7129a80 |
the bases.
|
|
|
7129a80 |
|
|
|
7129a80 |
First I'll give you a little high level view of the way things are
|
|
|
7129a80 |
built before major release, all using Pungi. I'll then try to dig into
|
|
|
7129a80 |
*The LEGOs* that are used to build the artifacts we care about.
|
|
|
7129a80 |
Finally, I'll try to explain how things are different after a major
|
|
|
7129a80 |
release is out and how we end up getting updated rpms and eventually
|
|
|
7129a80 |
two week releases out the door.
|
|
|
7129a80 |
|
|
|
7129a80 |
|
|
|
7129a80 |
# Before Fedora Major Release: Building Everything Using Pungi
|
|
|
7129a80 |
|
|
|
7129a80 |
Prior to any major release of Fedora the release engineering team *builds the
|
|
|
7129a80 |
whole world* every night. This is done using a tool called
|
|
|
7129a80 |
[Pungi](https://pagure.io/pungi). A [script](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/nightly.sh)
|
|
|
7129a80 |
is called to kick off the nightly run via a [cron job](https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/releng/files/branched).
|
|
|
7129a80 |
The Pungi config that is used is the [fedora.conf](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf)
|
|
|
7129a80 |
file from the [pungi-fedora](https://pagure.io/pungi-fedora) git repo.
|
|
|
7129a80 |
|
|
|
7129a80 |
For Fedora 26 you can see where the ostree for Atomic Host is defined
|
|
|
7129a80 |
in the Pungi config, [here](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_710-728),
|
|
|
7129a80 |
in the **ostree** section. During a compose you can
|
|
|
7129a80 |
see that the ostree will get placed into `"ostree_repo": "/mnt/koji/compose/atomic/26/",`.
|
|
|
7129a80 |
After the compose has finished it will get synced to `/mnt/koji/atomic/26/` which corresponds
|
|
|
7129a80 |
to the public URL https://kojipkgs.fedoraproject.org/atomic/26/.
|
|
|
7129a80 |
|
|
|
7129a80 |
During the compose, once the ostree commit has been created, the
|
|
|
7129a80 |
installer ISO and the cloud images can get created. The definition for
|
|
|
7129a80 |
the [installer ISO](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_741-761)
|
|
|
7129a80 |
and the [cloud images](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_355-383)
|
|
|
7129a80 |
are all within the same Pungi config.
|
|
|
7129a80 |
|
|
|
7129a80 |
# The LEGOs
|
|
|
7129a80 |
|
|
|
7129a80 |
There are a few building blocks we need to talk about before we really jump into
|
|
|
7129a80 |
how this whole thing is put together. First, we have to talk about how the OSTree
|
|
|
7129a80 |
gets created and then the artifacts that are derived from it.
|
|
|
7129a80 |
|
|
|
7129a80 |
## The RPM OSTree
|
|
|
7129a80 |
|
|
|
7129a80 |
To create the OSTree Pungi ends up calling out to a [Koji](https://pagure.io/koji)
|
|
|
7129a80 |
runroot task. This runroot task essentially runs a command and harvests the results. The
|
|
|
7129a80 |
command that eventually gets run is an [rpm-ostree command](https://pagure.io/pungi/blob/381d08a81079581595e9e48174267b7e7bbd263b/f/pungi/ostree/tree.py#_31-32)
|
|
|
7129a80 |
that looks something like this:
|
|
|
7129a80 |
|
|
|
7129a80 |
```nohighlight
|
|
|
7129a80 |
rpm-ostree compose tree --repo=/mnt/koji/compose/atomic/26/ \
|
|
|
7129a80 |
--write-commitid-to=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/logs/x86_64/Atomic/ostree-2/commitid.log \
|
|
|
7129a80 |
/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/ostree-2/config_repo/fedora-atomic-docker-host.json
|
|
|
7129a80 |
```
|
|
|
7129a80 |
|
|
|
7129a80 |
From the [ostree part of the Pungi config](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_710-728)
|
|
|
7129a80 |
you can see where some of the arguments for this command came from:
|
|
|
7129a80 |
|
|
|
7129a80 |
```
|
|
|
7129a80 |
ostree = [
|
|
|
7129a80 |
("^Atomic$", {
|
|
|
7129a80 |
"x86_64": {
|
|
|
7129a80 |
"treefile": "fedora-atomic-docker-host.json",
|
|
|
7129a80 |
"config_url": "https://pagure.io/fedora-atomic.git",
|
|
|
7129a80 |
"config_branch": "f26",
|
|
|
7129a80 |
"source_repo_from": "Everything",
|
|
|
7129a80 |
"ostree_repo": "/mnt/koji/compose/atomic/26/",
|
|
|
7129a80 |
'failable': ['*'],
|
|
|
7129a80 |
}
|
|
|
7129a80 |
}),
|
|
|
7129a80 |
```
|
|
|
7129a80 |
|
|
|
7129a80 |
We store our inputs inputs to `rpm-ostree compose tree` in
|
|
|
7129a80 |
the [fedora-atomic](https://pagure.io/fedora-atomic.git) git repo. The
|
|
|
7129a80 |
[fedora-26.repo](https://pagure.io/fedora-atomic/blob/d79a03ecf213843d2cbff2145c88865d3e898183/f/fedora-26.repo)
|
|
|
7129a80 |
file defines the dnf repositories and the
|
|
|
7129a80 |
[fedora-atomic-host.json]([https://pagure.io/fedora-atomic/blob/d79a03ecf213843d2cbff2145c88865d3e898183/f/fedora-atomic-host.json)
|
|
|
7129a80 |
file defines which of those repositories to use, which rpms to pull
|
|
|
7129a80 |
from them, and a few other things.
|
|
|
7129a80 |
|
|
|
7129a80 |
|
|
|
7129a80 |
## The Installer ISO
|
|
|
7129a80 |
|
|
|
7129a80 |
For the installer ISO Pungi again calls out to a Koji runroot task. The
|
|
|
7129a80 |
command that eventually gets run is a [lorax](https://github.com/rhinstaller/lorax)
|
|
|
7129a80 |
command that looks something like this:
|
|
|
7129a80 |
|
|
|
7129a80 |
```nohighlight
|
|
|
7129a80 |
lorax --product=Fedora --version=26 --release=20170705.n.0 \
|
|
|
7129a80 |
--source=http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170705.n.0/compose/Everything/x86_64/os \
|
|
|
7129a80 |
--variant=Atomic --nomacboot --volid=Fedora-Atomic-ostree-x86_64-26 \
|
|
|
7129a80 |
--installpkgs=fedora-productimg-atomic \
|
|
|
7129a80 |
--add-template=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/lorax_templates/ostree-based-installer/lorax-configure-repo.tmpl \
|
|
|
7129a80 |
--add-template=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/lorax_templates/ostree-based-installer/lorax-embed-repo.tmpl \
|
|
|
7129a80 |
--add-template-var=ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/atomic/26/ \
|
|
|
7129a80 |
--add-template-var=ostree_update_repo=https://kojipkgs.fedoraproject.org/atomic/26/ \
|
|
|
7129a80 |
--add-template-var=ostree_osname=fedora-atomic \
|
|
|
7129a80 |
--add-template-var=ostree_install_ref=fedora/26/x86_64/atomic-host \
|
|
|
7129a80 |
--add-template-var=ostree_update_ref=fedora/26/x86_64/atomic-host \
|
|
|
7129a80 |
--logfile=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/logs/x86_64/Atomic/ostree_installer-1/lorax.log \
|
|
|
7129a80 |
--rootfs-size=3 /mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/ostree_installer
|
|
|
7129a80 |
```
|
|
|
7129a80 |
|
|
|
7129a80 |
From the [installer ISO](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_741-761)
|
|
|
7129a80 |
part of the Pungi config you can see some configuration that was set
|
|
|
7129a80 |
and eventually translated into the lorax command above.
|
|
|
7129a80 |
|
|
|
7129a80 |
```
|
|
|
7129a80 |
ostree_installer = [
|
|
|
7129a80 |
("^Atomic$", {
|
|
|
7129a80 |
"x86_64": {
|
|
|
7129a80 |
"source_repo_from": "Everything",
|
|
|
7129a80 |
"release": None,
|
|
|
7129a80 |
"rootfs_size": "3",
|
|
|
7129a80 |
"installpkgs": ["fedora-productimg-atomic"],
|
|
|
7129a80 |
"add_template": ["ostree-based-installer/lorax-configure-repo.tmpl",
|
|
|
7129a80 |
"ostree-based-installer/lorax-embed-repo.tmpl"],
|
|
|
7129a80 |
"add_template_var": [
|
|
|
7129a80 |
"ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/atomic/26/",
|
|
|
7129a80 |
"ostree_update_repo=https://kojipkgs.fedoraproject.org/atomic/26/",
|
|
|
7129a80 |
"ostree_osname=fedora-atomic",
|
|
|
7129a80 |
"ostree_install_ref=fedora/26/x86_64/atomic-host",
|
|
|
7129a80 |
"ostree_update_ref=fedora/26/x86_64/atomic-host",
|
|
|
7129a80 |
],
|
|
|
7129a80 |
'template_repo': 'https://pagure.io/fedora-lorax-templates.git',
|
|
|
7129a80 |
'template_branch': 'f26',
|
|
|
7129a80 |
'failable': ['*'],
|
|
|
7129a80 |
}
|
|
|
7129a80 |
}),
|
|
|
7129a80 |
```
|
|
|
7129a80 |
|
|
|
7129a80 |
There are a few lorax templates we are passing in as well as some variables
|
|
|
7129a80 |
to those templates. The templates are stored in the [fedora-lorax-templates](https://pagure.io/fedora-lorax-templates.git)
|
|
|
7129a80 |
git repo.
|
|
|
7129a80 |
|
|
|
7129a80 |
|
|
|
7129a80 |
## The Cloud Images
|
|
|
7129a80 |
|
|
|
7129a80 |
For the cloud images Koji has higher level support for building
|
|
|
7129a80 |
them than it does for the installer ISO or for creating the ostree.
|
|
|
7129a80 |
It doesn't need to run things in a runroot, which is generic, but
|
|
|
7129a80 |
rather it can create an ImageBuild task to create an image.
|
|
|
7129a80 |
|
|
|
7129a80 |
This is still all defined in the Pungi config. The `image-build` sections
|
|
|
7129a80 |
for Atomic Host are [here](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_355-383)
|
|
|
7129a80 |
in the Pungi config. One of these sections looks like:
|
|
|
7129a80 |
|
|
|
7129a80 |
```
|
|
|
7129a80 |
'image-build': {
|
|
|
7129a80 |
'format': [('qcow2','qcow2'), ('raw-xz','raw.xz')],
|
|
|
7129a80 |
'name': 'Fedora-Atomic',
|
|
|
7129a80 |
'kickstart': 'fedora-atomic.ks',
|
|
|
7129a80 |
'distro': 'Fedora-22',
|
|
|
7129a80 |
'disk_size': 6,
|
|
|
7129a80 |
'arches': ['x86_64'],
|
|
|
7129a80 |
'install_tree_from': 'Cloud',
|
|
|
7129a80 |
'subvariant': 'Atomic',
|
|
|
7129a80 |
'failable': ['*'],
|
|
|
7129a80 |
}
|
|
|
7129a80 |
```
|
|
|
7129a80 |
|
|
|
7129a80 |
You'll notice that we say what kickstart file to use, but we don't define
|
|
|
7129a80 |
where to pull the kickstart file from. The git repo for the kickstarts is
|
|
|
7129a80 |
defined along with a few other varialbes [earlier in the file](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora.conf#_266-271).
|
|
|
7129a80 |
|
|
|
7129a80 |
# After Fedora Major Release: Bodhi + Pungi + Release
|
|
|
7129a80 |
|
|
|
7129a80 |
Once Fedora is officially released for a particular version there is a
|
|
|
7129a80 |
*release day* yum/dnf repository that is made from the rpms that were
|
|
|
7129a80 |
stable on the day of release. This repository is frozen and will not
|
|
|
7129a80 |
change.
|
|
|
7129a80 |
|
|
|
7129a80 |
There are two more repositories that become significant now. These are
|
|
|
7129a80 |
the **updates** and the **updates-testing** repositories. The **updates**
|
|
|
7129a80 |
repo contains packages that have passed testing and are available to Fedora users
|
|
|
7129a80 |
whenever they run `dnf update`. The **updates-testing** repo is for
|
|
|
7129a80 |
packages that have been built and have been submitted as an update for
|
|
|
7129a80 |
people to test to make sure nothing is broken before graduating to **updates**.
|
|
|
7129a80 |
The **updates-testing** repo is not enabled by default. A user would
|
|
|
7129a80 |
have to willingly enable it for the purpose of doing testing.
|
|
|
7129a80 |
|
|
|
7129a80 |
All of that is to say that there are other repos that exist after
|
|
|
7129a80 |
release day that are for updated rpms.
|
|
|
7129a80 |
|
|
|
7129a80 |
## Bodhi
|
|
|
7129a80 |
|
|
|
7129a80 |
Within Fedora there is a tool known as [Bodhi](https://bodhi.fedoraproject.org/)
|
|
|
7129a80 |
that is responsible for tracking what state particular packages are in
|
|
|
7129a80 |
and moving them between the **updates** and **updates-testing** repos.
|
|
|
7129a80 |
As part of this, Bodhi is currently responsible for creating the
|
|
|
7129a80 |
repositories and also the OSTree commits from the new content that
|
|
|
7129a80 |
it just created a repo for. Bodhi was the most logical candidate for
|
|
|
7129a80 |
this at the time it was implemented because we wanted to create a new
|
|
|
7129a80 |
commit as soon as we could grab new content (right after the
|
|
|
7129a80 |
repositories are created).
|
|
|
7129a80 |
|
|
|
7129a80 |
After creating the OSTree content it gets synced to `fedora/26/x86_64/updates/atomic-host`
|
|
|
7129a80 |
ref within the OSTree repository at https://kojipkgs.fedoraproject.org/atomic/26/.
|
|
|
7129a80 |
This is a global location/ref that users could pull from for existing
|
|
|
7129a80 |
installed systems.
|
|
|
7129a80 |
|
|
|
7129a80 |
## Pungi
|
|
|
7129a80 |
|
|
|
7129a80 |
When Bodhi runs and creates a new **updates** repo it also creates a
|
|
|
7129a80 |
new OSTree. When we do an official two week release we release other
|
|
|
7129a80 |
artifacts (ISOs, Qcows, etc) too. These get built once a day and uses the
|
|
|
7129a80 |
[fedora-atomic.conf](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/fedora-atomic.conf)
|
|
|
7129a80 |
Pungi config (which gets called from the
|
|
|
7129a80 |
[twoweek-nightly.sh](https://pagure.io/pungi-fedora/blob/392bc7589ecff19e91e03cef34265a270514745e/f/twoweek-nightly.sh)
|
|
|
7129a80 |
script). Basically the config creates repos from the `f26-atomic` Koji tag
|
|
|
7129a80 |
which inherits from the `f26` tag (release day tag). It creates some
|
|
|
7129a80 |
repos from rpms in the `Cloud` variant and then builds installer
|
|
|
7129a80 |
images and qcows based on those rpms. The installer images and the
|
|
|
7129a80 |
qcows actually pull the OSTree from the commit that was generated
|
|
|
7129a80 |
by the last Bodhi run, though. So the yum/dnf repository created
|
|
|
7129a80 |
during the Pungi run only affects the installer, not the installed
|
|
|
7129a80 |
OSTree in the installed systems.
|
|
|
7129a80 |
|
|
|
7129a80 |
## Performing A Two Week Atomic Host Release
|
|
|
7129a80 |
|
|
|
7129a80 |
So with the OSTree created by Bodhi and the other artifacts created by
|
|
|
7129a80 |
Pungi we can now test do an official two week Atomic Host release.
|
|
|
7129a80 |
This is currently run by a member of releng and the process is
|
|
|
7129a80 |
documented by [this](https://docs.pagure.org/releng/sop_two_week_atomic.html)
|
|
|
7129a80 |
Standard Operating Procedure guide. It eventually tells the operator
|
|
|
7129a80 |
to run [this releng script](https://pagure.io/releng/blob/e1fa88d3937412ca0f3c5d166f1b82c5106b1256/f/scripts/push-two-week-atomic.py)
|
|
|
7129a80 |
to do the release. The script does several things but most noteworthy
|
|
|
7129a80 |
is updating the `fedora/26/x86_64/atomic-host` ref within the OSTree
|
|
|
7129a80 |
repo, syncing out the ISOs/qcows to the mirrors, and updating the
|
|
|
7129a80 |
[website](https://getfedora.org/atomic/download/).
|
|
|
7129a80 |
|
|
|
7129a80 |
# Conclusion
|
|
|
7129a80 |
|
|
|
7129a80 |
Our processes differ slightly for before and after Fedora release.
|
|
|
7129a80 |
The entire pipeline involves Koji, Bodhi, and Pungi, and many tools
|
|
|
7129a80 |
underneath the covers. There is a lot of working being done to try to improve these
|
|
|
7129a80 |
processes including [calling Pungi from Bodhi](https://pagure.io/atomic-wg/issue/300)
|
|
|
7129a80 |
in the short term and possibly [building Atomic Host from a Module](https://pagure.io/atomic-wg/issue/312)
|
|
|
7129a80 |
in the longer term. Please reach out to us with any questions about
|
|
|
7129a80 |
this whole process in #atomic on freenode.
|