#335 creating smaller cloud images
Closed: fixed 14 days ago by ngompa. Opened 2 years ago by chrismurphy.

Rawhide=Btrfs
34 = ext4

175M -rw-rw-r--. 1 chris chris 175M Jun 22 14:16  Fedora-Cloud-Base-34-20210622.0.x86_64.raw.xz
325M -rw-rw-r--. 1 chris chris 325M Jun 22 14:04  Fedora-Cloud-Base-Rawhide-20210622.n.0.x86_64.raw.xz
--- xz -d
631M -rw-r--r--. 1 root root 4.0G Jun 22 15:06 Fedora-Cloud-Base-34-20210622.0.x86_64.raw
3.8G -rw-r--r--. 1 root root 5.0G Jun 22 15:07 Fedora-Cloud-Base-Rawhide-20210622.n.0.x86_64.raw

OK so what's going on here? Both are clearly sparse, but the ext4 one is more sparse.

  • lorax does e2fsck -E discard to ext4 file systems following umount, there is no equivalent in xfs_repair or btrfs check, for an offline discard; and it's preferred to use fstrim (FITRIM) which is supported by most file systems when mounted anyway.
    https://bugzilla.redhat.com/show_bug.cgi?id=1971186

  • images are currently zero'd by this:
    https://pagure.io/fedora-kickstarts/blob/main/f/fedora-cloud-base.ks#_109
    But on btrfs with zstd:1 mount option, this pile of zeros is being compressed. Meaning it's no longer redundant data when xz (and qcow2's zlib) get to it. This could be leading to additional inefficiency, because xz can't really do anything for blocks already compressed by zstd.

  • images also have rpms downloaded to /home, while they are subsequently deleted, without a subsequent fstrim, those blocks still remain in the image and as they are compressed rpms, aren't very compressible by xz either. Hence zeroing above.

  • if the image is being built in qemu, it's likely attached as a virtio device, and by default qemu sets discards=off or else fstrim does nothing. That it is doing something for e2fsck -E discard now suggests Fedora infra is using discards=unmap.

To do better we need:

  • Check the scripts that create/start the qemu VMs, to make sure discards=unamp so that fstrim passes through to the host, and punches holes in the backing file (the image).

  • Answer the question, do we want sparse images or preallocated images? if there's some concern about images with holes, we can end the build process with fallocate -lto replace all the holes with unwritten blocks. i.e. not sparse but also we aren't writing a bunch of zeros that end up getting compressed in the VM by btrfs zstd.

(qcow2 ignored because it's a converted copy of the raw, using zlib compression; so it's got the same relative issue as the raw.xz)


lorax/src/pylorax/installer.py:433:                    # For image installs, run fstrim to discard unused blocks. This way
lorax/src/pylorax/creator.py:558:    rc = execWithRedirect("/usr/sbin/fsck.ext4", ["-y", "-f", "-E", "discard", rootfs_img])

However, it doesn't appear in https://kojipkgs.fedoraproject.org//packages/Fedora-Cloud-Base/Rawhide/20210622.n.0/data/logs/image/oz-x86_64.log so I'm not certain lorax is doing this on the ext4 cloud images.

Filed a releng bug to make sure fstrim is being passed down to backing image, making them sparse. This would make any dd zeroing command obsolete.

https://pagure.io/releng/issue/10801

This is done now: https://pagure.io/releng/issue/10801

I still end up with waste, but I'm not sure why:

First mount
892856 -rw-rw-r--. 1 chris chris 4294967296 Jul 21 14:45 Fedora-Cloud-Base-36-20220721.0.x86_64.raw
fstrim /boot (partition 2)
856796 -rw-rw-r--. 1 chris chris 4294967296 Jul 21 14:45 Fedora-Cloud-Base-36-20220721.0.x86_64.raw
fstrim / (partition 5)
429316 -rw-rw-r--. 1 chris chris 4294967296 Jul 21 14:46 Fedora-Cloud-Base-36-20220721.0.x86_64.raw

@kevin are the builders using discard=unamp only for Rawhide? Or also 35 and 36 nightly images?

All the buildvm's (that is... all builders that are vm's and not bare metal) are using discard=unmap. There's no seperate builders for each release, etc.

OK looks like it's an oz issue. Affects Cloud Base and Workstation aarch64.

https://github.com/clalancette/oz/issues/301

We made our images smaller by moving to kiwi. :tada:

Metadata Update from @ngompa:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

14 days ago

Login to comment on this ticket.

Metadata