From 2bfb42cd82c33c47dff13c705cf4895a8eb4d05b Mon Sep 17 00:00:00 2001 From: Daan De Meyer Date: Feb 21 2023 10:38:01 +0000 Subject: systemd doc updates --- diff --git a/docs/internal/systemd.md b/docs/internal/systemd.md index 477af74..55cd466 100644 --- a/docs/internal/systemd.md +++ b/docs/internal/systemd.md @@ -55,33 +55,34 @@ section on how to do a new release for the RPM spec. Make sure you're [onboarded](https://sigs.centos.org/hyperscale/internal/onboarding/) onto the SIG before following these steps. -For the following examples, most/all of the links will go to the c8s-sig-hyperscale branch when there are mentions of -the RPM sources repo. Substitute as needed with the latest branches (e.g. c9s-sig-hyperscale). +For the following examples, most/all of the links will go to the c9s-sig-hyperscale branch when there are mentions of +the RPM sources repo. Substitute as needed with other branches (e.g. c8s-sig-hyperscale). -When we're building for 2 releases (e.g. c8s and c9s), you can do most/all of the work on one branch (e.g. c8s) and push -to the other branch (e.g. c9s) using the power of Git. Unless something happens, like a dependency being dropped on the -newer CentOS version, rarely will changes on the older branch fail on the newer one. +When we're building for 2 releases (e.g. c8s and c9s), you can do most/all of the work on one branch (e.g. c9s) and push +to the other branch (e.g. c8s) using the power of Git. Unless something happens, like a dependency on a newer release +not being available on an older release, rarely will changes on the newer branch fail on the older one. Steps: * Clone the CentOS RPM sources [repo](https://git.centos.org/rpms) - * Checkout the c8s-sig-hyperscale branch + * Checkout the c9s-sig-hyperscale branch * Add the Fedora RPM sources [repo](https://src.fedoraproject.org/rpms/systemd) as a git remote * `git remote add fedora https://src.fedoraproject.org/rpms/systemd && git fetch fedora` * Sync against changes from [Fedora Rawhide](https://src.fedoraproject.org/rpms/systemd/commits/rawhide). * The gist of this step is to figure out when we last synced from Fedora Rawhide, and get a diff between the latest - Rawhide changes and the last time we synced. Have the Fedora Rawhide repo checked out and pulled to the latest - commit. - * For example, looking at [this example commit](https://git.centos.org/rpms/systemd/blob/4f096da11ccb2b9dc2a13ca184ad3ae23ca68fa1/f/SPECS/systemd.spec) - again, scroll down to `%changelog` and the last non-CentOS Hyperscale log line was from `Tue Sep 14 2021 Sahana Prasad `. - You can tell because the version-release string in the changelog doesn't have the additional dot number appended that - we use for Hyperscale (e.g. 249.4-2 vs. 249.4-2.1). - * If we look for the Fedora Rawhide commit corresponding to that changelog line, it will take us to - [46a408102a0cfb5692d4625a33025811cab0343f](https://src.fedoraproject.org/rpms/systemd/c/46a408102a0cfb5692d4625a33025811cab0343f?branch=rawhide). - * `git diff 46a408102a0cfb5692d4625a33025811cab0343f..fedora/rawhide` will get us all the changes from the latest - commit to the last time we synced to CentOS Hyperscale. To apply these changes to the Hyperscale branch, run - `git apply -3 < $(git diff 46a408102a0cfb5692d4625a33025811cab0343f..fedora/rawhide)`. This will apply the changes and - generate merge conflicts when the upstream changes conflict with our own changes. + Rawhide changes and the last time we synced. Make sure you have the Fedora rawhide branch available for these steps. + * On the c9s-sig-hyperscale branch, scroll down to `%changelog` and find out what the the last non-CentOS Hyperscale + log line was. You can usually tell by the version-release string in the changelog that doesn't have the additional dot + number appended that we use for Hyperscale (e.g. 249.4-2 vs. 249.4-2.1). + * Switch to the Fedora rawhide branch and use `git blame` to find out what commit corresponds to the last non-CentOS + Hyperscale changelog line. + * Use `git diff ..fedora/rawhide` to get all the changes from the latest commit to the last time we synced to + CentOS Hyperscale. To apply these changes to the Hyperscale branch, run + `git apply -3 < $(git diff ..fedora/rawhide)`. This will apply the changes and generate merge conflicts when + the upstream changes conflict with our own changes. +* Next, you can check the remaining differences between the Fedora rawhide spec and the CentOS Hyperscale spec by running + `git diff fedora/rawhide..HEAD` while on the c9s-sig-hyperscale branch. Try to remove any differences between the two + that aren't strictly necessary anymore. * Update the source commits, version/release and changelog. * At the top of the specfile, update the `hs_commit` macro to point to the latest commits for the Hyperscale and Facebook branches for the current release on the staging [repo](https://pagure.io/centos-sig-hyperscale/systemd) @@ -94,73 +95,18 @@ Steps: * `spectool --define "_sourcedir $PWD" --define "facebook 1" -g systemd.spec` * `spectool --define "_sourcedir $PWD" -g systemd.spec` * Update the `sources` file with the new tarball names and SHA512 hashes. - * For example, the c8s-sig-hyperscale [sources](https://git.centos.org/rpms/systemd/blob/c8s-sig-hyperscale/f/sources) + * For example, the c9s-sig-hyperscale [sources](https://git.centos.org/rpms/systemd/blob/c9s-sig-hyperscale/f/sources) contains the SHA512 hashes along with the names of the tarballs. -* Git commit all these changes. Don't push just yet. - * If you push at this point it's not the end of the world, but we haven't confirmed the build succeeds yet! - * At this point it's also a good idea to ask for a sanity check from the other Hyperscale maintainers. You can do - a `git show` and paste the output to the [CentOS pastebin](https://pastebin.centos.org/). -* Build locally with `mock` - * One common reason for the RPM build failing is patch files between the staging repo and Fedora not playing nicely - together. If this is the case, you can try using `git am ` to apply the Fedora patches to the staging repo and - iron out build failures there. Rearrange patches as needed in the staging repo or specfile to get things to fit. - * Other reasons for failure are dependencies. This you'll need to figure out from logs. -* Do a scratch build in CBS - * You can use the source RPM generated by running `mock` to do the scratch build. It'll be located at - `/var/lib/mock/centos-stream-8-x86_64/root/builddir/build/SRPMS`. + * Use `sha512sum` to calculate the SHA512 hashes of the tarballs. * Upload the new version tarballs to CentOS. - * Calculate the SHA512 hash of a tarball using sha512sum. - * Check if the tarball has not already been uploaded [here](https://git.centos.org/sources/systemd/). * Upload the tarball to the CentOS lookaside cache by calling the `upload-sources.sh` script [here](https://pagure.io/centos-sig-hyperscale/systemd-releng/blob/main/f/upload-sources.sh) from the directory of your `sources` file. -* All updates are in and we have a successful build? Push your changes and get started with testing! - -# Build Commands - -For CBS you will need to be [onboarded to the SIG](onboarding.md). - -From inside the fork of the [systemd Hyperscale RPM sources](https://git.centos.org/rpms/systemd/tree/c8s-sig-hyperscale) repo: - -``` -# Create src.rpm -rpmbuild --define "%_topdir $PWD" --define "%_sourcedir $PWD" --define "%dist .hs.el8" -bs systemd.spec -rpmbuild --define "%_topdir $PWD" --define "%_sourcedir $PWD" --define "%facebook 1" --define "%dist .hs+fb.el8" -bs systemd.spec - -# Local build with mock; You can also skip these steps and use cbs directly -mock -r centos-stream-8-x86_64 -D "%dist .hs.el8" -mock -r centos-stream-8-x86_64 -D "%facebook 1" -D "%dist .hs+fb.el8" - -# CBS scratch builds -cbs build --scratch hyperscale8s-packages-main-el8 -cbs build --scratch hyperscale8s-packages-facebook-el8 -``` - -Once the specfile changes are pushed you can do a real build: - -``` -# CBS builds for CentOS Stream 8 -git checkout c8s-sig-hyperscale -cbs build hyperscale8s-packages-main-el8 "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" -cbs build hyperscale8s-packages-facebook-el8 "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" - -# CBS builds for CentOS Stream 9 -git checkout c9s-sig-hyperscale -cbs build hyperscale9s-packages-main-el9s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" -cbs build hyperscale9s-packages-facebook-el9s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" - -# Tag it for testing -cbs tag-build hyperscale8s-packages-main-testing systemd-.hs.el8 -cbs tag-build hyperscale8s-packages-facebook-testing systemd-.hs+fb.el8 -cbs tag-build hyperscale9s-packages-main-testing systemd-.hs.el9 -cbs tag-build hyperscale9s-packages-facebook-testing systemd-.hs+fb.el9 - -# Tag for release; This will let it go to the mirrors -cbs tag-build hyperscale8s-packages-main-release systemd-.hs.el8 -cbs tag-build hyperscale8s-packages-facebook-release systemd-.hs+fb.el8 -cbs tag-build hyperscale9s-packages-main-release systemd-.hs.el9 -cbs tag-build hyperscale9s-packages-facebook-release systemd-.hs+fb.el9 -``` +* Build locally with `mock` (See `Build Commands` section) + * The resulting rpms can be found in `/var/lib/mock/centos-stream-9-x86_64/result` +* Test that the new rpms work as expected (See `Testing` section) +* All updates are in and working as expected? Push your changes to a fork of the rpm sources repository, create a + pull request and ask the other Hyperscale maintainers to take a look! # Testing - Stock CentOS Stream @@ -191,11 +137,26 @@ services are broken, or the VM no longer boots with the hs version. Debugging ti Installation issues tend to come from building against a new dependency that is not yet in the mirrors. But the majority of the time, issues arise due to SELinux. CentOS Stream has SELinux enabled by default! +## mkosi + +You can also do testing using the mkosi image builder: + +``` +git clone https://github.com/systemd/mkosi +git clone https://pagure.io/centos-sig-hyperscale/mkosi hyperscale +ln -s mkosi/bin/mkosi ~/.local/bin +cd hyperscale +mkdir local +cp /var/lib/mock/centos-stream-9-x86_64/result/*.rpm local +createrepo_c local +mkosi --extra-search-path -f qemu +``` + ## Modifying SELinux Policies (and when your VM fails to boot) To check that the SELinux isn't spewing denials everywhere: ``` -sudo audit2allow --all +journalctl -g AVC | audit2allow ``` You normally want to run this before you install your new systemd RPMs, and then after upgrade and after boot as well. @@ -206,16 +167,16 @@ systemd you're testing) and try again with SELinux set to "permissive": * Modify the `SELINUX=enforcing` line to `SELINUX=permissive`. * Save, close, upgrade systemd, and reboot. -It should boot this time. It also means when you run `sudo audit2allow --all` there should be a bunch of new lines. +It should boot this time. It also means when you run audit2allow there should be a bunch of new lines. These show which policy denials. At this point you'll want to try modifying the SELinux policy. Inside your VM, `git clone` the systemd RPM repo from Hyperscale and checkout the corresponding branch that you're testing. Then: * `sudo dnf install selinux-policy-devel` needed to build policies -* Copy the policy/denial lines from the `sudo audit2allow --all` output into `systemd_hs.te` in the repo checkout. - Save and close. +* Copy the policy/denial lines from the audit2allow output that are relevant for systemd into `systemd_hs.te` + in the repo checkout. Save and close. * `make -f Makefile.selinux all` to build the policy module. * `sudo semodule -i systemd_hs.pp.bz2` to install the module you just built. -* Now when you run `sudo audit2allow --all` you'll messages that the denials are now passing/allowed by the new policy. +* Now when you run audit2allow you'll see messages that the denials are now passing/allowed by the new policy. * I recommend rebooting with this new policy and seeing if there are any new/leftover denials you missed. It's possible that some rules just don't audit or some new things appear now that they're not being masked by the old denials. Whatever the case, rinse and repeat a few times until there are no more denials. @@ -226,5 +187,47 @@ These show which policy denials. At this point you'll want to try modifying the * Now you can open a PR with your changes to `systemd_hs.te` and get this RPM version tagged! There are some other tools like `setroubleshoot` / `setroubleshootd` and `sealert` that you can try to help debug -SELinux. Honestly SELinux is a beast and I don't recommend wasting too much time here. `audit2allow` is very reliable +SELinux. Honestly SELinux is a beast and I don't recommend wasting too much time here. audit2allow is very reliable at getting to the point and tell you how to make SELinux denials stop. + +# Build Commands + +For CBS you will need to be [onboarded to the SIG](onboarding.md). + +From inside the fork of the [systemd Hyperscale RPM sources](https://git.centos.org/rpms/systemd/tree/c9s-sig-hyperscale) repo: + +``` +# Local build with mock. +mock -r centos-stream-hyperscale-9-x86_64 -D "%dist .hs.el9" --sources . --spec systemd.spec +mock -r centos-stream-hyperscale-9-x86_64 -D "%facebook 1" -D "%dist .hs+fb.el9" --sources . --spec systemd.spec + +# CBS scratch builds (use the SRPMs generated by mock above) +cbs build --scratch hyperscale9s-packages-main-el9s +cbs build --scratch hyperscale9s-packages-facebook-el9s +``` + +Once the specfile changes are pushed you can do a real build: + +``` +# CBS builds for CentOS Stream 8 +git checkout c8s-sig-hyperscale +cbs build hyperscale8s-packages-main-el8 "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" +cbs build hyperscale8s-packages-facebook-el8 "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" + +# CBS builds for CentOS Stream 9 +git checkout c9s-sig-hyperscale +cbs build hyperscale9s-packages-main-el9s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" +cbs build hyperscale9s-packages-facebook-el9s "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" + +# Tag it for testing +cbs tag-build hyperscale8s-packages-main-testing systemd-.hs.el8 +cbs tag-build hyperscale8s-packages-facebook-testing systemd-.hs+fb.el8 +cbs tag-build hyperscale9s-packages-main-testing systemd-.hs.el9 +cbs tag-build hyperscale9s-packages-facebook-testing systemd-.hs+fb.el9 + +# Tag for release; This will let it go to the mirrors +cbs tag-build hyperscale8s-packages-main-release systemd-.hs.el8 +cbs tag-build hyperscale8s-packages-facebook-release systemd-.hs+fb.el8 +cbs tag-build hyperscale9s-packages-main-release systemd-.hs.el9 +cbs tag-build hyperscale9s-packages-facebook-release systemd-.hs+fb.el9 +```