From 977e2df82a5c295cfb2f2748873ec43187b4231a Mon Sep 17 00:00:00 2001 From: Ivan Kruglov Date: Mar 06 2025 14:16:33 +0000 Subject: [PATCH 1/2] systemd: update testing doc --- diff --git a/docs/internal/systemd.md b/docs/internal/systemd.md index 9532a4d..f342a15 100644 --- a/docs/internal/systemd.md +++ b/docs/internal/systemd.md @@ -19,7 +19,7 @@ not being available on an older release, rarely will changes on the newer branch Steps: -* Clone the CentOS RPM sources [repo](https://git.centos.org/rpms/systemd) +* Clone the CentOS RPM sources [repo](https://gitlab.com/CentOS/Hyperscale/rpms/systemd) * Checkout the c10s-sig-hyperscale branch * Add the Fedora RPM sources [repo](https://src.fedoraproject.org/rpms/systemd) as a git remote * `git remote add fedora https://src.fedoraproject.org/rpms/systemd && git fetch fedora` @@ -79,285 +79,5 @@ Steps: # Testing -## mkosi - -We recommend testing changes using [mkosi](https://github.com/systemd/mkosi). -We install mkosi from git to make sure we have the latest features and fixes: - -```shell -git clone https://github.com/systemd/mkosi.git -sudo ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi -``` - -To install all required dependencies on the host system, run -`mkosi dependencies | xargs sudo dnf install`. Note that your host system will -need to have systemd 256 or newer packaged to be able to run the integration -tests successfully. - -## Testing with the systemd integration tests - -We'll start by testing our newly built rpms using the systemd integration tests. - -Next, clone the upstream systemd repository, checkout the stable branch -corresponding to the version we're going to release and configure meson (or use -an existing checkout): - -```shell -git clone https://github.com/systemd/systemd.git -cd systemd -git checkout -meson setup build -``` - -Then, in the systemd repository, build an image including the downloaded rpms: - -```shell -mkosi \ - --distribution=centos \ - --release= \ - --repositories=hyperscale-packages-main \ - --volatile-package-directory= \ - --environment=NO_BUILD=1 \ - --selinux-relabel=auto \ - --force -``` - -Finally, run the integration tests: - -```shell -SYSTEMD_INTEGRATION_TESTS=1 meson test -C build --no-rebuild --suite integration-tests --num-processes "$(($(nproc) / 3))" -``` - -Run through these steps for all supported CentOS Stream releases. - -If you have access to a powerful machine, it's recommended to use it ro run the -integration tests to speed up the process. - -## Testing SELinux with the systemd-releng repository - -To test use cases not covered by the systemd integration tests, we have a -separate set of mkosi configuration files in the [systemd-releng](https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng) -repository. - -Due to SELinux limitations, all mkosi commands in the selinux-releng repository -have to be run as the root user as we cannot do arbitrary SELinux relabeling -unless we're running with root privileges. - -First, clone the repository (or use an existing checkout): - -```shell -git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng.git -cd systemd-releng -``` - -Next, build a Hyperscale image, again including the downloaded rpms: - -```shell -sudo mkosi --profile hyperscale -r --volatile-package-directory -f -``` - -We can now try to boot into the image: - -```shell -sudo mkosi qemu -``` - -If the boot succeeds and we end up in a root shell, great! Otherwise, we'll need -to debug. If the boot failed this is either due to SELinux denials or because -dracut hasn't been updated to accomodate changes in newer systemd releases. We -can start by booting with SELinux in permissive mode: - -```shell -sudo mkosi --kernel-command-line-extra=enforcing=0 qemu -``` - -If this still doesn't get you into a root shell, the problem might lie with -dracut, see the next section for more details. Otherwise, regardless of whether -we're in enforcing or permissive mode, the first thing to check is whether -there's any SELinux denials: - -```shell -journalctl -t audit -g AVC -``` - -If there are denials, we can compare them with the denials on a Fedora Rawhide -system by running the following: - -```shell -sudo mkosi -d fedora -f qemu -journalctl -t audit -g AVC -``` - -If the denials don't pop up on Fedora Rawhide, that's a good indication that we -should backport a newer version of -[selinux-policy](https://git.centos.org/rpms/selinux-policy). To backport a -newer version, you can run the following: - -```shell -git clone https://git.centos.org/rpms/selinux-policy.git -cd selinux-policy -git remote add fedora https://src.fedoraproject.org/rpms/selinux-policy -git fetch rawhide -git checkout rawhide -fedpkg sources -git checkout cs-sig-hyperscale -git merge fedora/rawhide -``` - -To double check that the newer version fixes the denials, first use `mock` to -build the rpms: - -```shell -mock -r centos-stream-hyperscale--x86_64 --sources . --spec selinux-policy.spec -``` - -Then build and boot the image with the new selinux-policy rpms and check if the -denials are fixed: - -```shell -cd -sudo mkosi \ - --profile=hyperscale \ - --release= \ - --volatile-package-directory= \ - --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale--x86_64/result \ - --force \ - qemu -journalctl -t audit -g AVC -``` - -If the denials that were shown previously are gone, great! The new version of -selinux-policy includes fixes for the denials you were seeing. In this case you -can go ahead and build the new selinux-policy rpm in Hyperscale: - -```shell -cd -/lookaside_upload_sig -f macro-expander -n selinux-policy -/lookaside_upload_sig -f container-selinux.tgz -n selinux-policy -/lookaside_upload_sig -f selinux-policy-*.tar.gz -n selinux-policy -git checkout cs-sig-hyperscale -git push -cbs build hyperscales-packages-main-els git+https://git.centos.org/rpms/selinux-policy.git#(git rev-parse HEAD) -cbs tag-build hyperscales-packages-main-release selinux-policy-.hs.el -``` - -If the denials also occur on Fedora Rawhide, file an issue for the -[selinux-policy](https://github.com/fedora-selinux/selinux-policy/issues) -repository on Github. Include the lines from the `journalctl` output showing the -denials. Also include a reproducer to allow the maintainers to reproduce the -issue. A reproducer will generally look as follows (you can copy paste this -and modify it): - -```shell -git clone https://github.com/systemd/mkosi.git -ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi -git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng -sudo mkosi -d fedora -f qemu -``` - -You can extend the above with any extra commands required to reproduce the -SELinux denials in the virtual machine. Now it's a matter of waiting and working -with the policy maintainers to fix the denials, and backport the newer version -of selinux-policy once it is released in Fedora Rawhide (see above steps). - -## Testing the dracut backport with the systemd-releng repository - -Because new systemd releases often require changes in dracut as well, we also -maintain a backport of dracut from Fedora Rawhide. Every time a change in a new -release of systemd requires a corresponding change in dracut, a newer release of -dracut should be backported first before doing a new release of systemd in the -Hyperscale SIG. - -To backport a new version of dracut from Fedora Rawhide, you can run the -following steps: - -```shell -git clone https://git.centos.org/rpms/dracut.git -cd dracut -git remote add fedora https://src.fedoraproject.org/rpms/dracut -git fetch rawhide -git checkout rawhide -fedpkg sources -git checkout cs-sig-hyperscale -git merge fedora/rawhide -``` - -We can then build the new dracut rpms with mock: - -```shell -mock -r centos-stream-hyperscale--x86_64 --sources . --spec dracut.spec -``` - -And finally test whether the image built with mkosi boots with the new version -of dracut as follows: - -```shell -cd -sudo mkosi \ - --profile=hyperscale \ - --release= \ - --volatile-package-directory= \ - --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale--x86_64/result \ - --force \ - qemu -``` - -If everything works as expected, we can build and tag the new dracut version in -CBS: - -```shell -cd -/lookaside_upload_sig -f -n dracut -git checkout cs-sig-hyperscale -git push -cbs build hyperscales-packages-main-els git+https://git.centos.org/rpms/dracut.git#(git rev-parse HEAD) -cbs tag-build hyperscales-packages-main-release dracut-.hs.el -``` - -If there are still issues with booting, you'll need to debug the issue and -resolve it together with upstream. Be sure to also extend this section with any -useful debugging information to make the process easier for the next -contributor. - -## Testing upgrades from stock CentOS Stream with the systemd-releng repository - -To test upgrades from stock CentOS Stream to CentOS Stream + Hyperscale, you -can build and boot without the Hyperscale profile and then upgrade from within -the virtual machine: - -```shell -sudo mkosi -f qemu -dnf upgrade -``` - -The packages enabling the CentOS Hyperscale repositories are automatically -installed when building the image, so running `dnf upgrade` after booting is -sufficient to upgrade the system to CentOS Hyperscale. After upgrading, run -`journalctl -p warning` to see if anything went wrong during the upgrade. - -# CBS Build Commands - -For CBS you will need to be [onboarded to the SIG](onboarding.md). - -From inside the fork of the [systemd Hyperscale RPM sources](https://git.centos.org/rpms/systemd/tree/c9s-sig-hyperscale) repo: - -Once the specfile changes are pushed you can do a real build: - -```shell -git checkout cs-sig-hyperscale - -# Scratch build -cbs build --scratch hyperscales-packages-main-els "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" - -# Official build -cbs build hyperscales-packages-main-els "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" - -# Tag it for testing -cbs tag-build hyperscales-packages-main-testing systemd-.hs.el - -# Tag for release (this will let it go to the mirrors) -cbs tag-build hyperscales-packages-main-release systemd-.hs.el -``` - -To do builds in the facebook tag, use the above commands but replace `main` by -`facebook`. +Submit MR to GitLab repo (https://gitlab.com/CentOS/Hyperscale/rpms/systemd). +The CI system will automatiicaly build and run systemd integration tests. From 35de7274a318116376cda32e5454634483e6e184 Mon Sep 17 00:00:00 2001 From: Ivan Kruglov Date: Mar 06 2025 14:17:01 +0000 Subject: [PATCH 2/2] systemd: fold outdated-but-usefull sections --- diff --git a/docs/internal/systemd.md b/docs/internal/systemd.md index f342a15..ded248c 100644 --- a/docs/internal/systemd.md +++ b/docs/internal/systemd.md @@ -81,3 +81,241 @@ Steps: Submit MR to GitLab repo (https://gitlab.com/CentOS/Hyperscale/rpms/systemd). The CI system will automatiicaly build and run systemd integration tests. + +Note: Information in sections below maybe outdated. + +
+ Testing SELinux with the systemd-releng repository + + To test use cases not covered by the systemd integration tests, we have a + separate set of mkosi configuration files in the [systemd-releng](https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng) + repository. + + Due to SELinux limitations, all mkosi commands in the selinux-releng repository + have to be run as the root user as we cannot do arbitrary SELinux relabeling + unless we're running with root privileges. + + First, clone the repository (or use an existing checkout): + + ```shell + git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng.git + cd systemd-releng + ``` + + Next, build a Hyperscale image, again including the downloaded rpms: + + ```shell + sudo mkosi --profile hyperscale -r --volatile-package-directory -f + ``` + + We can now try to boot into the image: + + ```shell + sudo mkosi qemu + ``` + + If the boot succeeds and we end up in a root shell, great! Otherwise, we'll need + to debug. If the boot failed this is either due to SELinux denials or because + dracut hasn't been updated to accomodate changes in newer systemd releases. We + can start by booting with SELinux in permissive mode: + + ```shell + sudo mkosi --kernel-command-line-extra=enforcing=0 qemu + ``` + + If this still doesn't get you into a root shell, the problem might lie with + dracut, see the next section for more details. Otherwise, regardless of whether + we're in enforcing or permissive mode, the first thing to check is whether + there's any SELinux denials: + + ```shell + journalctl -t audit -g AVC + ``` + + If there are denials, we can compare them with the denials on a Fedora Rawhide + system by running the following: + + ```shell + sudo mkosi -d fedora -f qemu + journalctl -t audit -g AVC + ``` + + If the denials don't pop up on Fedora Rawhide, that's a good indication that we + should backport a newer version of + [selinux-policy](https://git.centos.org/rpms/selinux-policy). To backport a + newer version, you can run the following: + + ```shell + git clone https://git.centos.org/rpms/selinux-policy.git + cd selinux-policy + git remote add fedora https://src.fedoraproject.org/rpms/selinux-policy + git fetch rawhide + git checkout rawhide + fedpkg sources + git checkout cs-sig-hyperscale + git merge fedora/rawhide + ``` + + To double check that the newer version fixes the denials, first use `mock` to + build the rpms: + + ```shell + mock -r centos-stream-hyperscale--x86_64 --sources . --spec selinux-policy.spec + ``` + + Then build and boot the image with the new selinux-policy rpms and check if the + denials are fixed: + + ```shell + cd + sudo mkosi \ + --profile=hyperscale \ + --release= \ + --volatile-package-directory= \ + --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale--x86_64/result \ + --force \ + qemu + journalctl -t audit -g AVC + ``` + + If the denials that were shown previously are gone, great! The new version of + selinux-policy includes fixes for the denials you were seeing. In this case you + can go ahead and build the new selinux-policy rpm in Hyperscale: + + ```shell + cd + /lookaside_upload_sig -f macro-expander -n selinux-policy + /lookaside_upload_sig -f container-selinux.tgz -n selinux-policy + /lookaside_upload_sig -f selinux-policy-*.tar.gz -n selinux-policy + git checkout cs-sig-hyperscale + git push + cbs build hyperscales-packages-main-els git+https://git.centos.org/rpms/selinux-policy.git#(git rev-parse HEAD) + cbs tag-build hyperscales-packages-main-release selinux-policy-.hs.el + ``` + + If the denials also occur on Fedora Rawhide, file an issue for the + [selinux-policy](https://github.com/fedora-selinux/selinux-policy/issues) + repository on Github. Include the lines from the `journalctl` output showing the + denials. Also include a reproducer to allow the maintainers to reproduce the + issue. A reproducer will generally look as follows (you can copy paste this + and modify it): + + ```shell + git clone https://github.com/systemd/mkosi.git + ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi + git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng + sudo mkosi -d fedora -f qemu + ``` + + You can extend the above with any extra commands required to reproduce the + SELinux denials in the virtual machine. Now it's a matter of waiting and working + with the policy maintainers to fix the denials, and backport the newer version + of selinux-policy once it is released in Fedora Rawhide (see above steps). +
+ +
+ Testing the dracut backport with the systemd-releng repository + + Because new systemd releases often require changes in dracut as well, we also + maintain a backport of dracut from Fedora Rawhide. Every time a change in a new + release of systemd requires a corresponding change in dracut, a newer release of + dracut should be backported first before doing a new release of systemd in the + Hyperscale SIG. + + To backport a new version of dracut from Fedora Rawhide, you can run the + following steps: + + ```shell + git clone https://git.centos.org/rpms/dracut.git + cd dracut + git remote add fedora https://src.fedoraproject.org/rpms/dracut + git fetch rawhide + git checkout rawhide + fedpkg sources + git checkout cs-sig-hyperscale + git merge fedora/rawhide + ``` + + We can then build the new dracut rpms with mock: + + ```shell + mock -r centos-stream-hyperscale--x86_64 --sources . --spec dracut.spec + ``` + + And finally test whether the image built with mkosi boots with the new version + of dracut as follows: + + ```shell + cd + sudo mkosi \ + --profile=hyperscale \ + --release= \ + --volatile-package-directory= \ + --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale--x86_64/result \ + --force \ + qemu + ``` + + If everything works as expected, we can build and tag the new dracut version in + CBS: + + ```shell + cd + /lookaside_upload_sig -f -n dracut + git checkout cs-sig-hyperscale + git push + cbs build hyperscales-packages-main-els git+https://git.centos.org/rpms/dracut.git#(git rev-parse HEAD) + cbs tag-build hyperscales-packages-main-release dracut-.hs.el + ``` + + If there are still issues with booting, you'll need to debug the issue and + resolve it together with upstream. Be sure to also extend this section with any + useful debugging information to make the process easier for the next + contributor. +
+ +
+ Testing upgrades from stock CentOS Stream with the systemd-releng repository + + To test upgrades from stock CentOS Stream to CentOS Stream + Hyperscale, you + can build and boot without the Hyperscale profile and then upgrade from within + the virtual machine: + + ```shell + sudo mkosi -f qemu + dnf upgrade + ``` + + The packages enabling the CentOS Hyperscale repositories are automatically + installed when building the image, so running `dnf upgrade` after booting is + sufficient to upgrade the system to CentOS Hyperscale. After upgrading, run + `journalctl -p warning` to see if anything went wrong during the upgrade. +
+ +
+ CBS Build Commands + For CBS you will need to be [onboarded to the SIG](onboarding.md). + + From inside the fork of the [systemd Hyperscale RPM sources](https://git.centos.org/rpms/systemd/tree/c9s-sig-hyperscale) repo: + + Once the specfile changes are pushed you can do a real build: + + ```shell + git checkout cs-sig-hyperscale + + # Scratch build + cbs build --scratch hyperscales-packages-main-els "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" + + # Official build + cbs build hyperscales-packages-main-els "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" + + # Tag it for testing + cbs tag-build hyperscales-packages-main-testing systemd-.hs.el + + # Tag for release (this will let it go to the mirrors) + cbs tag-build hyperscales-packages-main-release systemd-.hs.el + ``` + + To do builds in the facebook tag, use the above commands but replace `main` by + `facebook`. +