For cloud installs of Fedora, we want to provide advanced file system features to users in a transparent fashion. Thus, we are changing the file system for the Cloud Edition to Btrfs so we can leverage its features and capabilities to improve the quality of experience for Cloud users.
+1 (change owner)
Just worth mentioning, this would mean that Fedora cloud images cannot be mounted on a RHEL host. RHEL does not enable or support BTRFS at all, the module is not built, there would be no way to mount and verify a Fedora cloud image on RHEL or CentOS without actually booting it. If your image is unbootable, for whatever reason, you may have to find a different host to mount and try to fix it on.
Irrespectively of this change, and whether btrfs is the default or not in other places, not being able to access a very popular file system at all seems very user-unfriendly. Not supporting it for root file system and normal storage on RHEL systems is one thing and completely reasonable, but completely preventing users from accessing a certain file system type is quite a big step. So maybe we could convince the RHEL maintainers to issue the module in an update.
I have to agree with @zbyszek here: being completely unable to even access a btrfs filesystem is imho not a great user experience and shouldn't be a reason why the Fedora Cloud images shouldn't switch. It is rather a reason why at least read support should be added to RHEL.
RHEL makes its decisions based on a variety of factors, even if they were to change this for RHEL 8 (seems unlikely) or RHEL 9, it could take a very long time before this appears in production environments. We have absolutely no control over what they decide to build and support. For workstation and such, this doesn't matter so much. For cloud, we offer a downloadable raw image that is likely to be used on RHEL hosts as much or more than on Fedora. It is certainly worth taking into consideration.
Metadata Update from @ngompa: - Issue tagged with: meeting
I don't think this is a chicken and egg problem. Fedora's mandate is to lead, not second guess itself as if it's a favor to Red Hat. Guessing is folly. Fedora needs to do what's in the best interest of the community given all available facts. And as we don't have many (or even most) of the decision making facts that go into RHEL, I choose to consider it out of scope. As in, it expressly should not be taken into consideration.
Red Hat can do anything they want. If they care about this use case they can at least provide a libguestfs appliance that supports btrfs, so that guestfish can inspect btrfs images.
So, I was pretty "meh" about this change... seemed fine, but wasn't really a big gain for cloud uses.
Thinking about it more however, would this result in our images no longer being able to be automatically resized? Or snapshotted outside the OS?
ie, https://docs.openstack.org/image-guide/openstack-images.html for openstack has no ability to auto resize btrfs (at least that it says there)
How does aws handle this?
cloud-init is capable of resizing btrfs (it works with openSUSE based images already), so I imagine AWS would use that same path anyway.
cloud-init
So, I was pretty "meh" about this change... seemed fine, but wasn't really a big gain for cloud uses. Thinking about it more however, would this result in our images no longer being able to be automatically resized? Or snapshotted outside the OS?
No, none of this changes with Btrfs.
I'm pretty sure OpenStack's documentation is out of date here, since it references cloud-init stuff.
We discussed this during today's fesco meeting: AGREED: We'll wait a week for more discussion and information about userspace access options. ACTION: Eighth_Doctor, cmurf, dcavalca, michel will provide some more information about userspace access options
Just for more information (I was in another meeting during the meeting, so couldn't comment as much). Yes, libguestfs does include a kernel for the appliance, but it is a buildreq, it is not building its own, meaning the RHEL libguestfs still does not support btrfs. From the spec:
%if !0%{?rhel} BuildRequires: btrfs-progs %endif
From a kernel standpoint, I don't care too much one way or the other for most spins. Cloud images, you do have to take into consideration the host though. Even if RHEL were to turn on btrfs in RHEL 9 for some reason (I don't see any indication that this is being considered, it is still off in ARK), it would literally take years for that to roll out to the majority of users. I would guess there is considerably more RHEL 7 out there than RHEL 8 at this point. I don't know when RHEL 9 is planning to release, but It isn't in the F35 time frame for sure. This is one area that it certainly will not be Fedora leads RHEL, because there is a lot more at play there., and there are plenty of features enabled in Fedora that RHEL is happy to keep turned off for a variety of reasons. As far as I am currently aware, btrfs is one of those features. Basically the decision needs to be made assuming nothing is going to change in RHEL.
So what do we give up by having no btrfs support on the host? The images should still boot, and function as normal. You do lose the ability to debug or modify the image from a RHEL host, and basically anything with the image that does not involve booting the included kernel. No read or write access. You can't try to use the image as a container image. You limit some forms of offline backup, though some others still work. I can not comment on the Openstack compatibility, because I haven't been much of a user of that, it would need someone else to verify.
Basically the decision needs to be made assuming nothing is going to change in RHEL.
Ack.
A possible read only solution is grub2-mount: as grub can read btrfs file systems, you can use its implementation to mount a btrfs based cloud image. Unfortunately you won't be able to modify it from that.
This should even be possible from RHEL, since GRUB there has support for Btrfs.
grub2-mount is explicitly disabled in CentOS Stream 8. I've filed https://bugzilla.redhat.com/show_bug.cgi?id=1970538 to get it added and put up https://git.centos.org/rpms/grub2/pull-request/1 with the necessary changes. CentOS Stream 9 already includes it, so we should be good there.
grub2-mount
Aslo, depending on the grub version zstd will be unsupported (see: https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_Btrfs.3F)
This will be discussed again during the meeting today, Tuesday June 15th, 2021 at 1700 UTC in #fedora-meeting on libera.chat.
I've updated the proposal wiki to include some details on userspace access options are requested by @zbyszek. Also copying it here for visibility:
Btrfs Cloud images will boot and function as normal even on hosts that do not support btrfs (e.g. RHEL). Resizing is also not impacted, as that's generally handled via cloud-init. However, the host will not be able to mount the image directly, nor read or write its contents. Libguestfs is also impacted, as it generally relies on an appliace built from the host kernel.
A standalone Fedora-built libguestfs container will allow image access regardless of host kernel support. The container was approved in https://bugzilla.redhat.com/show_bug.cgi?id=1970071 and a preliminary version is already up at https://src.fedoraproject.org/container/libguestfs . Productionization is pending on container build issues in https://pagure.io/releng/issue/10161 being resolved.
As grub can already read btrfs file systems, grub2-mount should be able to read only mount btrfs Cloud images. However, grub2-mount is currently disabled in CentOS Stream 8 (and hence in RHEL 8). We have filed https://bugzilla.redhat.com/show_bug.cgi?id=1970538 to get it enabled, and put up https://git.centos.org/rpms/grub2/pull-request/1 with the necessary changes to this end. CentOS Stream 9 already includes grub2-mount, so no change is needed there (nor should be needed for RHEL 9 as it inherits from CentOS Stream 9). Moreover, grub2 in CentOS Stream 8 currently does not include zstd support; this is being addressed in https://github.com/rhboot/grub2/pull/86.
Because btrfs-progs already includes a full filesystem implementation in userspace, it should be possible to write a Fuse-based tool leveraging it to mount btrfs filesystems on any host with no kernel involvement. This has been confirmed as feasible, and a design is currently being put together.
Metadata Update from @sgallagh: - Issue close_status updated to: Accepted - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.