From 7668ac62cf5907edbbdc84b52a0103683cbadef6 Mon Sep 17 00:00:00 2001 From: Daan De Meyer Date: Jul 25 2024 13:22:13 +0000 Subject: Update systemd backport contribution documentation We stop using the staging repository in favor of using the upstream systemd github repository directly. This allows us to greatly reduce the diff of our downstream spec with the Fedora Rawhide spec. It also encourages us to participate more in upstream stable releases instead of duplicating the work of backporting patches from the main branch. If circumstances require it, we can always include patches directly in the rpm repository until we're ready to move to a new stable release. We also extend the testing section to use mkosi to run the systemd integration tests against the built rpms and we add another testing section on using the systemd-releng repository to test SELinux denials and how to update the selinux-policy backport in Hyperscale if needed to address any new SELinux denials. I will do a follow up PR to add a section on testing dracut and one on testing upgrades from stock CentOS Stream to CentOS Stream + Hyperscale. --- diff --git a/docs/internal/systemd.md b/docs/internal/systemd.md index acd4bfe..b8f6e41 100644 --- a/docs/internal/systemd.md +++ b/docs/internal/systemd.md @@ -6,208 +6,293 @@ We try to follow the [systemd build from Fedora rawhide](https://src.fedoraproje Links can be found [on the CI/CD page](ci.md). -# Contributing to the Staging Repo - -All of the patches we backport for the Hyperscale build of systemd are first merged to [centos-sig-hyperscale/systemd](https://pagure.io/centos-sig-hyperscale/systemd), -a Git repo where we stage patches. This allows people to use `meson` to build systemd with all of the Hyperscale patches, and run them against the systemd -upstream unit tests without having to build a RPM. - -We try to maintain as few patches as possible for each release, but we are open to including new features or bug fixes that are not yet available. -Generally we prefer there to either be an open or merged pull request (PR) in [systemd upstream](https://github.com/systemd/systemd) before we will -consider merging into the staging repo. - -Steps: - -* Get a Fedora account [here](https://accounts.fedoraproject.org) if you don't have one already. -* Fork [centos-sig-hyperscale/systemd](https://pagure.io/centos-sig-hyperscale/systemd). -* Check out the latest Hyperscale branch there. It should be the last one alphabetically in the form of "hs-v". - If you want to backport a patch to a stable systemd version that does not yet have a branch available in the staging repo, open an issue - [here](https://pagure.io/centos-sig-hyperscale/systemd/issues) to ask the maintainers to create a new branch for that version. -* If you think your PR from upstream will apply cleanly, you can download it and `git am ` it directly. Otherwise you can use - `git cherry-pick` and manual edits to ensure your change will build. - * At minimum, do a `meson test` to ensure unit tests pass. - * If you're backporting an upstream commit, update the BACKPORTS file with a link to the changes you're backporting -* Open a PR against [centos-sig-hyperscale/systemd](https://pagure.io/centos-sig-hyperscale/systemd) with your change. - * In the PR, include a link to your PR from systemd upstream. - * Tag anitazha/daandemeyer for reviews. -* Once merged, one of the systemd Hyperscale maintainers will update the RPM spec and publish a new version of systemd for Hyperscale - with the patches. - -# Preparing a new release in the Staging Repo - -- Run `create_or_checkout_release_branch.sh` from the [systemd-releng](https://pagure.io/centos-sig-hyperscale/systemd-releng) - repo. Pass the name of the systemd stable release you want to - base the new Hyperscale release on as an argument (e.g. `./create_or_checkout_release_branch.sh 250.3`). This will push two new - release branches to the pagure repository, one for the general Hyperscale release, one for the Meta specific release. -- Look at the BACKPORTS file for the backports done for the previous release, find out which are still relevant for the next release - and backport those changes to the new release branches. First, do the backports for the Hyperscale release branch, then merge the - Hyperscale release branch into the Meta specific release branch and then do any Meta specific backports. -- Add new entries to the BACKPORTS file for the new release branches, along with links to all the backports. -- Push the branches to the pagure remote. - -The new release branches are now ready to be used in the RPM spec. See the next -section on how to do a new release for the RPM spec. - # Contributing to the RPM spec / New Version Updates in the SIG Make sure you're [onboarded](https://sigs.centos.org/hyperscale/internal/onboarding/) onto the SIG before following these steps. -For the following examples, most/all of the links will go to the c9s-sig-hyperscale branch when there are mentions of -the RPM sources repo. Substitute as needed with other branches (e.g. c8s-sig-hyperscale). +For the following examples, most/all of the links will go to the c10s-sig-hyperscale branch when there are mentions of +the RPM sources repo. Substitute as needed with other branches (e.g. c9s-sig-hyperscale). -When we're building for 2 releases (e.g. c8s and c9s), you can do most/all of the work on one branch (e.g. c9s) and push -to the other branch (e.g. c8s) using the power of Git. Unless something happens, like a dependency on a newer release +When we're building for 2 releases (e.g. c10s and c9s), you can do most/all of the work on one branch (e.g. c10s) and merge the +changes to the other branch (e.g. c9s) using the power of Git. Unless something happens, like a dependency on a newer release not being available on an older release, rarely will changes on the newer branch fail on the older one. Steps: * Clone the CentOS RPM sources [repo](https://git.centos.org/rpms) - * Checkout the c9s-sig-hyperscale branch + * Checkout the c10s-sig-hyperscale branch * Add the Fedora RPM sources [repo](https://src.fedoraproject.org/rpms/systemd) as a git remote * `git remote add fedora https://src.fedoraproject.org/rpms/systemd && git fetch fedora` -* Sync against changes from [Fedora Rawhide](https://src.fedoraproject.org/rpms/systemd/commits/rawhide). - * The gist of this step is to figure out when we last synced from Fedora Rawhide, and get a diff between the latest - Rawhide changes and the last time we synced. Make sure you have the Fedora rawhide branch available for these steps. - * On the c9s-sig-hyperscale branch, scroll down to `%changelog` and find out what the the last non-CentOS Hyperscale - log line was. You can usually tell by the version-release string in the changelog that doesn't have the additional dot - number appended that we use for Hyperscale (e.g. 249.4-2 vs. 249.4-2.1). - * Switch to the Fedora rawhide branch and use `git blame` to find out what commit corresponds to the last non-CentOS - Hyperscale changelog line. - * Use `git diff ..fedora/rawhide` to get all the changes from the latest commit to the last time we synced to - CentOS Hyperscale. To apply these changes to the Hyperscale branch, run - `git apply -3 < $(git diff ..fedora/rawhide)`. This will apply the changes and generate merge conflicts when - the upstream changes conflict with our own changes. +* Merge changes from [Fedora Rawhide](https://src.fedoraproject.org/rpms/systemd/commits/rawhide). + * `git merge fedora/rawhide` + * Solve any merge conflicts + * Carefully go through each commit from Rawhide. Take special care for commits that introduce changes in default behavior + (e.g. https://src.fedoraproject.org/rpms/systemd/c/ba02e904964116b848080ca72243174f4ef3eced?branch=ba02e904964116b848080ca72243174f4ef3eced) + as these can have a drastic impact on our users. Whenever finding such a commit, bring it up + for discussion with the other members of the SIG to determine how we should handle the change. + * Another thing to look out for is newly added `Obsoletes` for the systemd package itself, for + example, `Obsoletes: systemd < 246.6-2`. These prevent the old version from being installed anymore + when the package with the `Obsoletes` is added to the rpm repository and should be removed until we + can be reasonably sure that a newer version of systemd than is being obsoleted is installed everywhere. * Next, you can check the remaining differences between the Fedora rawhide spec and the CentOS Hyperscale spec by running - `git diff fedora/rawhide..HEAD` while on the c9s-sig-hyperscale branch. Try to remove any differences between the two + `git diff fedora/rawhide..HEAD` while on the c10s-sig-hyperscale branch. Try to remove any differences between the two that aren't strictly necessary anymore. -* Update the source commits, version/release and changelog. - * At the top of the specfile, update the `hs_commit` macro to point to the latest commits for the Hyperscale and - Facebook branches for the current release on the staging [repo](https://pagure.io/centos-sig-hyperscale/systemd) - respectively. - * For the v250 release, these would be the latest commits on the hs-v250.3 and hs+fb-v250.3 branches respectively. - * At the top of the specfile, update the `Version` and `Release` values. This should match Fedora, except that we - also append a dot number value to indicate which iteration of the Hyperscale release we're on. - * At the bottom of the specfile under `changelog`, add an entry about these changes. -* Run the following commands to download the tarballs for the Hyperscale and Facebook branches: - * `spectool --define "_sourcedir $PWD" --define "facebook 1" -g systemd.spec` +* Include patches that haven't been merged upstream yet or which haven't been backported to a stable branch + * Note that we're very hesitant on including downstream patches and there needs to be a very good reason to include + downstream patches. Every downstream patch should have a comment to an upstream pull request where the functionality + is proposed for inclusion in systemd itself. + * If the patch has been upstreamed but isn't in a stable release yet, you can ask upstream for a new stable release or + prepare one yourself using the instructions found [here](https://systemd.io/RELEASE/). If there was a recent stable + release and the issue that we need fixed only happens rarely in very specific use cases, the corresponding commit can + be added as a patch to the rpm repository instead. + * Run `git format-patch ` in the systemd repository + * Move the patch to the rpm repository + * Add something like `PatchXXXX: ` to the rpm spec +* Update the version and release. + * If we're building rpms for the same systemd stable release (from https://github.com/systemd/systemd/releases) as + is shipped in Fedora Rawhide, no changes are required to the spec. + * If we're building a different release than the one shipped in Fedora Rawhide, update the default value for the + `Version` field to match the version of the stable release that we want to build (e.g. `256.2`). + * Update the `Release` value. This should match Fedora, except that we also append a dot number value to indicate + which iteration of the Hyperscale release we're on (e.g. if Fedora has `256.2-1`, Hyperscale should have `256.2-1.1`). +* Note that generally, aside from changing the version, release and including patches, all other changes should be implemented + in the Fedora spec and backported to the Hyperscale spec to keep the diff between the two as small as possible. + * Exceptions can be made when Fedora makes a change that we want to explicitly want to deviate from. + * The upstream systemd repository builds rpms for CentOS Stream using the Fedora Rawhide spec, so the more our spec matches + the Fedora Rawhide one, the better our test coverage. +* Download the source tarball from the systemd github repository: * `spectool --define "_sourcedir $PWD" -g systemd.spec` -* Update the `sources` file with the new tarball names and SHA512 hashes. - * For example, the c9s-sig-hyperscale [sources](https://git.centos.org/rpms/systemd/blob/c9s-sig-hyperscale/f/sources) - contains the SHA512 hashes along with the names of the tarballs. - * Use `sha512sum` to calculate the SHA512 hashes of the tarballs. -* Upload the new version tarballs to CentOS. +* Update the `sources` file with the new tarball name and SHA512 hash. + * Use `sha512sum` to calculate the SHA512 hash of the tarball. +* Upload the new version tarball to the CentOS lookaside cache. * Upload the tarball to the CentOS lookaside cache by calling the `lookaside_upload_sig` script - [here](https://git.centos.org/centos-git-common/blob/master/f/lookaside_upload_sig). -* Build locally with `mock` (See `Build Commands` section) - * The resulting rpms can be found in `/var/lib/mock/centos-stream-9-x86_64/result` + found [here](https://git.centos.org/centos-git-common/blob/master/f/lookaside_upload_sig). + * For example, the command could look as follows: + * `centos-git-common/lookaside_upload_sig -f systemd-256.2.tar.gz -n systemd` +* Build locally with `mock` + * `mock -r centos-stream-hyperscale-10-x86_64 -D "%dist .hs.el10" --sources . --spec systemd.spec` + * The resulting rpms can be found in `/var/lib/mock/centos-stream-10-x86_64/result` * Test that the new rpms work as expected (See `Testing` section) * All updates are in and working as expected? Push your changes to a fork of the rpm sources repository, create a pull request and ask the other Hyperscale maintainers to take a look! -# Testing - Stock CentOS Stream +# Testing -## Basic Tests +## mkosi -Depending on how many versions of CentOS we need to support (typically 2 max), you will likely want to set up some virtual machines -(VM) for testing. Head over to the CentOS Stream website to grab the 2 most recent versions and get your VMs set up. As soon -as base CentOS Stream is installed, snapshot your VM! After this point, testing will begin and any number of things can go wrong. +We recommend testing changes using [mkosi](https://github.com/systemd/mkosi). +We install mkosi from git to make sure we have the latest features and fixes: -Testing goes like: +```shell +git clone https://github.com/systemd/mkosi.git +sudo ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi +``` + +To install the dependencies required by mkosi, we can either use a tools tree or +install the required dependencies on our host system. To use a tools tree, write +the following to mkosi.local.conf within the same directory containing the mkosi +configuration that we're about to build (see below): + +```conf +[Host] +ToolsTree=default +ToolsTreeDistribution=centos +ToolsTreeRelease=9 +ToolsTreeRepositories=epel,epel-next,hyperscale-packages-main +``` -* `sudo dnf install centos-packager` # if you don't have it already for the `cbs` command -* `sudo dnf install epel-release centos-release-hyperscale` -* Use `cbs download-build --task-id [id]` to fetch the systemd rpms you built and tagged for testing. -* At this point, you will want to make another VM snapshot. -* `sudo dnf upgrade` to get the packges from EPEL and Hyperscale. -* `sudo dnf install systemd-networkd systemd-boot` if it wasn't installed already. -* `sudo dnf upgrade *.rpm` the RPMs you downloaded. -* `sudo systemctl start systemd-networkd` -* `systemctl status` to make sure the state is not degraded. -* Reboot the host. -* Start systemd-networkd again (if it isn't up) and check that the state is still not degraded. -* Scan the journal and dmesg logs for any errors related to systemd after the timeframe of the upgrade. -* If you made it this far without seeing errors or running into issues, you're good to tag the non-FB build for release! +To install all required dependencies on the host, run +`mkosi dependencies | xargs sudo dnf install`. Note that your host system will +need to have systemd 256 or newer packaged to be able to run the integration +tests successfully. -It's very normal to make it successfully through the hs+fb rollout (Meta only) only to find that the RPMs won't install, -services are broken, or the VM no longer boots with the hs version. Debugging time! -Installation issues tend to come from building against a new dependency that is not yet in the mirrors. -But the majority of the time, issues arise due to SELinux. CentOS Stream has SELinux enabled by default! +## Testing with the systemd integration tests -## Modifying SELinux Policies (and when your VM fails to boot) +We'll start by testing our newly built rpms using the systemd integration tests. + +Next, clone the upstream systemd repository, checkout the stable branch +corresponding to the version we're going to release and configure meson (or use +an existing checkout): + +```shell +git clone https://github.com/systemd/systemd.git +cd systemd +git checkout +meson setup build +``` -To check that the SELinux isn't spewing denials everywhere: +Then, in the systemd repository, build an image including the downloaded rpms: + +```shell +mkosi \ + --distribution=centos \ + --release= \ + --repositories=hyperscale-packages-main \ + --volatile-package-directory= \ + --environment=NO_BUILD=1 \ + --selinux-relabel=auto \ + --force ``` -journalctl -g AVC | audit2allow + +Finally, run the integration tests: + +```shell +SYSTEMD_INTEGRATION_TESTS=1 meson test -C build --no-rebuild --suite integration-tests --num-processes "$(($(nproc) / 3))" ``` -You normally want to run this before you install your new systemd RPMs, and then after upgrade and after boot as well. -If your VM no longer boots, roll back to the last working snapshot (hopefully right before you installed the version of -systemd you're testing) and try again with SELinux set to "permissive": +Run through these steps for all supported CentOS Stream releases. -* Open `/etc/selinux/config` in a text editor. -* Modify the `SELINUX=enforcing` line to `SELINUX=permissive`. -* Save, close, upgrade systemd, and reboot. +If you have access to a powerful machine, it's recommended to use it ro run the +integration tests to speed up the process. -It should boot this time. It also means when you run audit2allow there should be a bunch of new lines. -These show which policy denials. At this point you'll want to try modifying the SELinux policy. Inside your VM, -`git clone` the systemd RPM repo from Hyperscale and checkout the corresponding branch that you're testing. Then: +## Testing SELinux with the systemd-releng repository -* `sudo dnf install selinux-policy-devel` needed to build policies -* Copy the policy/denial lines from the audit2allow output that are relevant for systemd into `systemd_hs.te` - in the repo checkout. Save and close. -* `make -f Makefile.selinux all` to build the policy module. -* `sudo semodule -i systemd_hs.pp.bz2` to install the module you just built. -* Now when you run audit2allow you'll see messages that the denials are now passing/allowed by the new policy. -* I recommend rebooting with this new policy and seeing if there are any new/leftover denials you missed. It's possible - that some rules just don't audit or some new things appear now that they're not being masked by the old denials. - Whatever the case, rinse and repeat a few times until there are no more denials. -* SNAPSHOT YOUR VM. -* Set SELinux back to enforcing. -* Reboot. Hopefully things keep functioning. -* If they don't, restore your snapshot and keep tweaking the policy until it does. -* Now you can open a PR with your changes to `systemd_hs.te` and get this RPM version tagged! +To test use cases not covered by the systemd integration tests, we have a +separate set of mkosi configuration files in the [systemd-releng](https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng) +repository. -There are some other tools like `setroubleshoot` / `setroubleshootd` and `sealert` that you can try to help debug -SELinux. `audit2allow` is very reliable at getting to the point and tell you how to make SELinux denials stop. +Due to SELinux limitations, all mkosi commands in the selinux-releng repository +have to be run as the root user as we cannot do arbitrary SELinux relabeling +unless we're running with root privileges. -# Build Commands +First, clone the repository (or use an existing checkout): -For CBS you will need to be [onboarded to the SIG](onboarding.md). +```shell +git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng.git +cd systemd-releng +``` -From inside the fork of the [systemd Hyperscale RPM sources](https://git.centos.org/rpms/systemd/tree/c9s-sig-hyperscale) repo: +Next, build a Hyperscale image, again including the downloaded rpms: + +```shell +sudo mkosi --profile hyperscale -r --volatile-package-directory -f +``` + +We can now try to boot into the image: +```shell +sudo mkosi qemu ``` -# Local build with mock. SIG chroots are from mock-centos-sig-configs. -mock -r centos-stream-hyperscale-9-x86_64 -D "%dist .hs.el9" --sources . --spec systemd.spec -mock -r centos-stream-hyperscale-9-x86_64 -D "%facebook 1" -D "%dist .hs+fb.el9" --sources . --spec systemd.spec -# CBS scratch builds (use the SRPMs generated by mock above) -cbs build --scratch hyperscale9s-packages-main-el9s -cbs build --scratch hyperscale9s-packages-facebook-el9s +If the boot succeeds and we end up in a root shell, great! Otherwise, we'll need +to debug. If the boot failed this is likely due to SELinux denials. We can start +by booting with SELinux in permissive mode: + +```shell +sudo mkosi --kernel-command-line-extra=enforcing=0 qemu ``` -Once the specfile changes are pushed you can do a real build: +This should get you into a root shell. Regardless of whether we're in enforcing +or permissive mode, the first thing to check is whether there's any SELinux +denials: +```shell +journalctl -t audit -g AVC ``` -# CBS builds for CentOS Stream 8 -git checkout c8s-sig-hyperscale -cbs build hyperscale8s-packages-main-el8 "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" -cbs build hyperscale8s-packages-facebook-el8 "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" -# CBS builds for CentOS Stream 9 -git checkout c9s-sig-hyperscale -cbs build hyperscale9s-packages-main-el9s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" -cbs build hyperscale9s-packages-facebook-el9s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" +If there are denials, we can compare them with the denials on a Fedora Rawhide +system by running the following: + +```shell +sudo mkosi -d fedora -f qemu +journalctl -t audit -g AVC +``` + +If the denials don't pop up on Fedora Rawhide, that's a good indication that we +should backport a newer version of +[selinux-policy](https://git.centos.org/rpms/selinux-policy). To backport a +newer version, you can run the following: + +```shell +git clone https://git.centos.org/rpms/selinux-policy.git +cd selinux-policy +git remote add fedora https://src.fedoraproject.org/rpms/selinux-policy +git fetch rawhide +git checkout rawhide +fedpkg sources +git checkout cs-sig-hyperscale +git merge fedora/rawhide +``` + +To double check that the newer version fixes the denials, first use `mock` to +build the rpms: + +```shell +mock -r centos-stream-hyperscale--x86_64 --sources . --spec selinux-policy.spec +``` + +Then build and boot the image with the new selinux-policy rpms and check if the +denials are fixed: + +```shell +cd +sudo mkosi \ + --profile=hyperscale \ + --release= \ + --volatile-package-directory= \ + --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale--x86_64/result \ + --force \ + qemu +journalctl -t audit -g AVC +``` + +If the denials that were shown previously are gone, great! The new version of +selinux-policy includes fixes for the denials you were seeing. In this case you +can go ahead and build the new selinux-policy rpm in Hyperscale: + +```shell +cd +/lookaside_upload_sig -f macro-expander -n selinux-policy +/lookaside_upload_sig -f container-selinux.tgz -n selinux-policy +/lookaside_upload_sig -f selinux-policy-*.tar.gz -n selinux-policy +git checkout cs-sig-hyperscale +git push +cbs build hyperscales-packages-main-els git+https://git.centos.org/rpms/selinux-policy.git#(git rev-parse HEAD) +cbs tag-build hyperscales-packages-main-release selinux-policy-.hs.el +``` + +If the denials also occur on Fedora Rawhide, file an issue for the +[selinux-policy](https://github.com/fedora-selinux/selinux-policy/issues) +repository on Github. Include the lines from the `journalctl` output showing the +denials. Also include a reproducer to allow the maintainers to reproduce the +issue. A reproducer will generally look as follows (you can copy paste this +and modify it): + +```shell +git clone https://github.com/systemd/mkosi.git +ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi +git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng +sudo mkosi -d fedora -f qemu +``` + +You can extend the above with any extra commands required to reproduce the +SELinux denials in the virtual machine. Now it's a matter of waiting and working +with the policy maintainers to fix the denials, and backport the newer version +of selinux-policy once it is released in Fedora Rawhide (see above steps). + +# CBS Build Commands + +For CBS you will need to be [onboarded to the SIG](onboarding.md). + +From inside the fork of the [systemd Hyperscale RPM sources](https://git.centos.org/rpms/systemd/tree/c9s-sig-hyperscale) repo: + +Once the specfile changes are pushed you can do a real build: + +```shell +git checkout cs-sig-hyperscale + +# Scratch build +cbs build --scratch hyperscales-packages-main-els "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" + +# Official build +cbs build hyperscales-packages-main-els "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" # Tag it for testing -cbs tag-build hyperscale8s-packages-main-testing systemd-.hs.el8 -cbs tag-build hyperscale8s-packages-facebook-testing systemd-.hs+fb.el8 -cbs tag-build hyperscale9s-packages-main-testing systemd-.hs.el9 -cbs tag-build hyperscale9s-packages-facebook-testing systemd-.hs+fb.el9 +cbs tag-build hyperscales-packages-main-testing systemd-.hs.el -# Tag for release; This will let it go to the mirrors -cbs tag-build hyperscale8s-packages-main-release systemd-.hs.el8 -cbs tag-build hyperscale8s-packages-facebook-release systemd-.hs+fb.el8 -cbs tag-build hyperscale9s-packages-main-release systemd-.hs.el9 -cbs tag-build hyperscale9s-packages-facebook-release systemd-.hs+fb.el9 +# Tag for release (this will let it go to the mirrors) +cbs tag-build hyperscales-packages-main-release systemd-.hs.el ``` + +To do builds in the facebook tag, use the above commands but replace `main` by +`facebook`.