We should probably just go ahead and enable Swap on ZRAM for Cloud images...

We should probably just go ahead and enable Swap on ZRAM for Cloud images...

I'm in favor.

It seems like systemd-oomd can now handle running (or at least not erroring) if no swap exists:

[vagrant@vanilla-f34 ~]$ sudo systemctl status systemd-oomd
● systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer
     Loaded: loaded (/usr/lib/systemd/system/systemd-oomd.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-05-11 18:50:21 UTC; 29s ago
       Docs: man:systemd-oomd.service(8)
   Main PID: 474 (systemd-oomd)
     Status: "Processing requests..."
      Tasks: 1 (limit: 4674)
     Memory: 1.8M (min: 64.0M low: 64.0M)
        CPU: 41ms
     CGroup: /system.slice/systemd-oomd.service
             └─474 /usr/lib/systemd/systemd-oomd

May 11 18:50:21 fedora systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer...
May 11 18:50:21 fedora systemd-oomd[474]: Swap is currently not detected; memory pressure usage will be degraded
May 11 18:50:21 fedora systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer.

systemd-oomd and zram make sense for cloud images.

Kernels 5.8+ have a completely reworked anonymous memory balancing logic, so it's useful on SSDs even with latency sensitive workloads. That is to say, I think it's reasonable for Cloud folks to consider a swap partition by default without worrying about swap thrashing being an issue.

There might be a higher penalty for Cloud edition than desktops, with zram or zswap, due to their current non-adaptability. The lack of a complete implementation in the kernel suggests rare edge cases on the desktop with poor compression ratios and then we see high CPU overhead. This may not be so rare on Cloud use cases, I can't really estimate it.

Here's another way to look at it: The eviction efficacy of a swap partition is 100%. For zram-based swap swap efficacy might be 50-75%, because it depends on the compression ratio, and this range is typical for desktop workloads. It's a catch-22 with tradeoffs, if we know nothing about a configuration or workload.

Another issue is that zram and zswap don't yet support cgroups for resource control. There are some neat accounting and throttle capablilities with cgroupsv2 found in the memory.swap.* entries in sysfs.

I'd say zram-based swap is better than no swap because it will help avoid reclaim, which is expensive and can look like swap thrashing. But conventional swap to an SSD, is probably even better for Cloud edition workloads.

cc @anitazha

We discussed this in the cloud meeting today.

12:56:52*     dustymabe | #agreed for f35 fedora cloud we will enable swap-on-zram and sytemd-oomd 

Chiming in to say that without swap PSI increases will be more abrupt rather than gradual. But PSI-based killing will still work (and systemd-oomd will still work as well, it will just ignore swap checks/return zeroes). I can update the documentation on this to be more clear but I don't think a condition check upstream will be necessary (but you're free to add one downstream).

Chiming in to say that without swap PSI increases will be more abrupt rather than gradual. But PSI-based killing will still work (and systemd-oomd will still work as well, it will just ignore swap checks/return zeroes).

Thanks for the info!

I can update the documentation on this to be more clear

Yes please, if you don't mind. If you could link me to a PR that would be great. I'm trying to get as much information as possible about the "no swap" behavior of sytemd-oomd for the Fedora CoreOS working group discussion and more clarity in upstream docs would be useful.

but I don't think a condition check upstream will be necessary (but you're free to add one downstream).

👍 - we'll keep it in line with upstream for now.

Opened PR for man page update https://github.com/systemd/systemd/pull/19876

Opened PR to enable zram in cloud-base https://pagure.io/fedora-kickstarts/pull-request/819

zram generator is now enabled in Rawhide. :wine_glass:

@davdunc - nice.

Since systemd-oomd-defaults is already installed there's nothing left to do here IIUC.

$ rpm -q systemd-oomd-defaults
systemd-oomd-defaults-248-2.fc34.x86_64