From e55ff2611fd2210b1995da8da2841ce74d394ad9 Mon Sep 17 00:00:00 2001 From: Anita Zhang Date: Jan 20 2023 05:48:26 +0000 Subject: systemd: add stock VM+SELinux testing instructions --- diff --git a/docs/internal/systemd.md b/docs/internal/systemd.md index 1cb75e6..477af74 100644 --- a/docs/internal/systemd.md +++ b/docs/internal/systemd.md @@ -161,3 +161,70 @@ cbs tag-build hyperscale8s-packages-facebook-release systemd-.hs+fb cbs tag-build hyperscale9s-packages-main-release systemd-.hs.el9 cbs tag-build hyperscale9s-packages-facebook-release systemd-.hs+fb.el9 ``` + +# Testing - Stock CentOS Stream + +## Basic Tests + +Depending on how many versions of CentOS we need to support (typically 2 max), you will likely want to set up some virtual machines +(VM) for testing. Head over to the CentOS Stream website to grab the 2 most recent versions and get your VMs set up. As soon +as base CentOS Stream is installed, snapshot your VM! After this point, testing will begin and any number of things can go wrong. + +Testing goes like: + +* `sudo dnf install centos-packager` # if you don't have it already for the `cbs` command +* `sudo dnf install epel-release centos-release-hyperscale` +* Use `cbs download-build [id]` to fetch the systemd rpms you built and tagged for testing. +* At this point, you will want to make another VM snapshot. +* `sudo dnf upgrade` to get the packges from EPEL and Hyperscale. +* `sudo dnf install systemd-networkd systemd-boot` if it wasn't installed already. +* `sudo dnf upgrade *.rpm` the RPMs you downloaded. +* `sudo systemctl start systemd-networkd` +* `systemctl status` to make sure the state is not degraded. +* Reboot the host. +* Start systemd-networkd again (if it isn't up) and check that the state is still not degraded. +* Scan the journal and dmesg logs for any errors related to systemd after the timeframe of the upgrade. +* If you made it this far without seeing errors or running into issues, you're good to tag the non-FB build for release! + +It's very normal to make it successfully through the hs+fb rollout (Meta only) only to find that the RPMs won't install, +services are broken, or the VM no longer boots with the hs version. Debugging time! +Installation issues tend to come from building against a new dependency that is not yet in the mirrors. +But the majority of the time, issues arise due to SELinux. CentOS Stream has SELinux enabled by default! + +## Modifying SELinux Policies (and when your VM fails to boot) + +To check that the SELinux isn't spewing denials everywhere: +``` +sudo audit2allow --all +``` +You normally want to run this before you install your new systemd RPMs, and then after upgrade and after boot as well. + +If your VM no longer boots, roll back to the last working snapshot (hopefully right before you installed the version of +systemd you're testing) and try again with SELinux set to "permissive": + +* Open `/etc/selinux/config` in a text editor. +* Modify the `SELINUX=enforcing` line to `SELINUX=permissive`. +* Save, close, upgrade systemd, and reboot. + +It should boot this time. It also means when you run `sudo audit2allow --all` there should be a bunch of new lines. +These show which policy denials. At this point you'll want to try modifying the SELinux policy. Inside your VM, +`git clone` the systemd RPM repo from Hyperscale and checkout the corresponding branch that you're testing. Then: + +* `sudo dnf install selinux-policy-devel` needed to build policies +* Copy the policy/denial lines from the `sudo audit2allow --all` output into `systemd_hs.te` in the repo checkout. + Save and close. +* `make -f Makefile.selinux all` to build the policy module. +* `sudo semodule -i systemd_hs.pp.bz2` to install the module you just built. +* Now when you run `sudo audit2allow --all` you'll messages that the denials are now passing/allowed by the new policy. +* I recommend rebooting with this new policy and seeing if there are any new/leftover denials you missed. It's possible + that some rules just don't audit or some new things appear now that they're not being masked by the old denials. + Whatever the case, rinse and repeat a few times until there are no more denials. +* SNAPSHOT YOUR VM. +* Set SELinux back to enforcing. +* Reboot. Hopefully things keep functioning. +* If they don't, restore your snapshot and keep tweaking the policy until it does. +* Now you can open a PR with your changes to `systemd_hs.te` and get this RPM version tagged! + +There are some other tools like `setroubleshoot` / `setroubleshootd` and `sealert` that you can try to help debug +SELinux. Honestly SELinux is a beast and I don't recommend wasting too much time here. `audit2allow` is very reliable +at getting to the point and tell you how to make SELinux denials stop.