Link to Bugzilla Entry
In what was, at least to the active members of the Fedora Cloud Working Group, a flash decision on October 29th, 2021, the Fedora cloud images were eliminated from consideration as an edition for Fedora 35. There was some historical foundation for making this decision, but it did not, in the opinion of the working group members, disqualify the actions and delivery requirements for the group as they had been actively working towards a successful edition release with modifications and updates. With the popularity of the cloud offering today, it still deserves to be available on the Fedora Website and as an edition.
This was an unexpected setback for the cloud working group. As a working group, we had an appointed a member to work on including the edition in the new website revamp and it was almost two-years ago that we were told to wait for the website revamp before the edition would be changing locations. The effort put forth to make the cloud images possible for public and private hyperscale environments is a distinctly complex action, but like every other edition, it is wrapped up in the common concerns of delivering the other server, workstation, and immutable OS editions.
The Fedora Cloud base image was removed as an edition from the F35 release of Fedora. The cloud working group's very existence has been challenged and there was a move from community members to force the Working Group to reassemble by updating the PRD and actitivies. As the Fedora Cloud base image is no longer considered an edition, it will not be included in the Website revamp which is a currently ongoing project.
Link to Reference Discussion
Matthew Miller was kind enough to collect a series of discussions that demonstrated a thread of shifting the focus of the cloud working group from the standard Fedora Cloud base images to work related to the Atomic image. Shifting of the Cloud WG to Atomic WG was formalized 4 years ago and there was an effort to stabilize the Atomic experience and fulfill the needs of the Cloud image user base with the Atomic Project. Before that was completed, Red Hat purchased CoreOS, a lightweight operating system purpose-built to support lightweight clustering systems. This purchase eclipsed the work being done on Project Atomic and brought with it a significantly large code base and distribution model that was not yet integrated with any of the Cloud WG's program. Meanwhile the Cloud SIG continued to provide a distinctly minimal Fedora base without the integration of OStree. Fedora CoreOS superseded the Project Atomic mechanics, but did not align with the Cloud WG goals. It is clear that there was great discussion when Atomic WG was being replaced with CoreOS, but at the same time, the goals of the Cloud WG and the Atomic WG were not yet unified so the Fedora Cloud WG remained active independently. The Fedora CoreOS mission and the Fedora Cloud mission have not aligned either and continue to represent distinct user bases.
For both the cloud working group and the FCOS working group, a significant amount of work continued, so there has been an operational split in what is delivered by the Cloud and FCOS working groups and updates to the goals related to the cloud images and the FCOS in separated programs. The cloud images maintain a focus on supporting distinctly different workloads than those of the lightweight containers workloads targeted by the FCOS working group. The Cloud Working Group continues improving the cloud image base to continue delighting users under the impression that they were still included in the editions.
From the discussion it was clear that there was a difference of opinion over the value of the cloud base edition delivered independent of other editions. Opinions were voiced identifying that there was an argument to be made for the cloud base image to be a secondary form of one of the other editions, such as Server or Workstation with no requirement to consider it specialized. There will need a new charter for the cloud images since there are a number of council members who have called for clarification and an updated review.
According to the statistics presented by Matthew Miller, Fedora Cloud base use makes up more than 15% of the total persistent<sup>1</sup> installations and more than 30% of the total ephemeral<sup>2</sup> installations today making it the second most popular effort in the portfolio after Desktop.
The first issue raised was one of value to users and developers. There was an opinion expressed that the goals of these, cloud edition are not sufficiently divergent from those of the Fedora Server edition to warrant a second series of goals and responsibilities. Additionally, another opinion was stated that at times, the goals of Cloud overlap with those of Workstation Edition. This was stated as a reason to consider delivering cloud based distributions of both of these editions rather than keeping this single edition that matches this overlap with additional specialization.
The argument that there is insufficient value in the Cloud edition because it has overlapping value in other projects is not, in our evaluation, sufficient to support giving up on highly specialized development with a sensitivity to the modifications based on functional requirements in the individual environments supported (more every day)
Matthew also identified himself as one of those people who no longer sees the variations in the versions. He was convinced, but now he isn't as to what kind of controls separate server from cloud editions. There were a couple of current considerations that came to mind immediately that determine those differences. Thanks to Matthew for helping us to tailor our discussion.
David Duncan mentioned in a recent Fedora Podcast with Grayson, that users who work with Cloud instances in public cloud have access to the latest hardware and platform specialities. Just like the flexibility users have in instance types, they require flexibility in what they do on their instances. That means advanced specialized functionality for the platform infrastructure with little to no additional overhead. The mission of server includes preserving support for legacy hardware and that's just not a critical focus for Cloud images.
Fedora Server has a strong requirement to remain stable, which is why, when the cloud working group found that moving to btrfs as a file system was a compelling position, Server edition working group members voiced concerns that users would not have access to advanced RAID types in an attempt to identify that this was not the decision they would make. For users adept in cloud configurations, it is well-known that the advanced RAID types are not generally useful. The virtual volumes presented as hardware and used by cloud consumers are typically already striped and redundant behind the hyperscaler storage. Virtual volumes are elastic as a result and extend to match larger sizes without requiring advanced RAID to achieve that outcome. What is useful to a typical cloud user is advanced partitioning with subvolumes and additional methods for handling dynamic snapshots and in cases where there are adjustments or updates to deliver for new images cloud is focused on using techniques for fast API transfer, like the ones found in coldsnap.
Cloud, not Server deals with hyperscaler platform variations, like the use of swap files and power management in opportunistic compute power. That isn't an issue for Fedora Server since the partitioning is expected to follow disk management practices. It might be something that you want to address on workstation, but even then, using a swapfile would be a strong deviation from the best practices for standard installations and deployment. Workstation WG and Server WG are keenly focused on different support goals. The FCOS WG is focused on building a foundation for lightweight clusters and basic batch efforts. The difference in focus differentiates them. The Cloud WG is valuable to both persistent and ephemeral users who are taking advantage of platform features in their modernization.
This is by no means an exhaustive list of the distinctions in mission or functionality.
The cloud working group, while still active, was intended to fold into the Atomic Working Group when it was stable. That never really happened. The Atomic Working Group was dissolved in favor of the Fedora CoreOS alignment with Red Hat CoreOS when Red Hat acquired CoreOS. The Cloud Working Group remained active throughout all of these changes and so while there was a plan of action in the council to make the FCOS working group replace the Atomic WG, there was no direct dissolution of the cloud working group because it had remained active throughout. Now, there is a BZ that states that Cloud is no longer an edition and that means the team will have to reestablish the status as an edition.
The Server Edition has a direct affiliation with the RHEL product line. It is a forgone conclusion that it exists as an asset of continued significance with planned use in the future. Therefore there is no way to remove it without creating a gap in the current sponsorship. FCOS also has that same direct relationship with Red Hat CoreOS and cannot deviate from the downstream alignment without creating uncertainty. The cloud image is not currently connected to a release by a direct line. It merely informs the builds for the cloud images for Red Hat Enterprise Linux. With the move to use BTRFS, there was a strong divergence from Server. Moving the project under server would be a forcing function for the group to revert changes that are beneficial to public and private cloud initiatives in favor of legacy expectations.
With Workstation, there are similar concerns regarding the alignment. For developers and site reliability engineers, there is no association with Workstation as a directive for a minimal installation. It would be a considerable change for this group that has historically been focused on Laptops and PC Hardware. The Target Audience would use easy to configure virtual machines for development that isn't container specific. It is more likely that those developers would benefit from the Fedora Cloud base images to achieve a lightweight development environment that meets their requirements for a minimal install environment and dynamic configuration through userdata and automation. Vagrant images are also the product of the Cloud WG.
Fedora Workstation was the first to make the move to btrfs and it was alignment with that existing direction that helped the Cloud WG to move the cloud images to deviate from what the Server working group was doing. It was not expected that the Fedora Server would be able to be consistent with these directives as there is not yet full support for RAID5 and RAID6. Fedora CoreOS moved to XFS before workstation moved to btrfs for reasons of alignment, but originally, they were openly using use of btrfs and that was simply discouraged based on concerns related to issues of consistency and negative perceptions of btrfs. With Fedora Cloud Base image, the advantages outweighed the alternatives in moving away from ext4.
The Cloud base image goals are not the same as Workstation or Server, but the mechanics are similar. Hyperscalers and public clouds don't need another server and do not typically require all the components for Workstation. The emphasis in cloud is on boot times and complex snapshot management or platform special requirements. In many cases, there are platform services that cannot be fully utilized using other editions without significant modifications and the Cloud Edition will continue to focus on these requirements to ensure that it remains popular with users.
This is mostly answered by the environments where customers choose to run their workloads. A considerable number of workloads are moving to cloud and hyperscaler environments without changing their fundamental runbooks. Fedora Cloud images are popular because they are available where customers require them. This will further increase as the cloud working group continues to extend the availability of the images and tailors them to various environments' specific needs as a principal goal for their target users.
The focus of the working group is the most important aspect of the published editions, but clarity on the status of the working group can retard the progress of even a strong working plan. The current cloud working group members believed that the dissolution of the Atomic WG returned the focus to the original charter when FCOS formed independently, with different goals and focus. Getting back to the expected status requires the cloud group to retrace their steps and lose ground gained over years of action.
As of this midnight hour change for the release of F35, effort will be required to reestablish the working cloud release as an edition. The Cloud Base image will not be included in the F35 releases or the website revamp. The cloud working group learned that de facto working programs, even when all processes align with the target goals previously agreed upon, cannot undo a procedural confusion that aligns with a well-intentioned, but inconsistent chain of decisions. The cloud working group intends to remedy this as soon as possible with an expectation that we can be back on track by the release of F36.
<sup>1</sup> persistent - installed for more than one week.
<sup>2</sup> ephemeral - installed for less than one week.
Metadata Update from @davdunc: - Issue tagged with: meeting
The Bugzilla bug isn't the key thing here. The key change here happened years ago, when we made Atomic Host an edition and Cloud not an edition. Cloud would not have been an edition for Fedora 35 whether or not that bug was filed. It wasn't one for F34, or F33, or F32, or etc. etc. There was no "consideration" for edition status going on; we haven't "considered" what the editions are since whenever we made IoT one (last cycle, I think).
At least as early as 2018, this was clearly explained on the wiki page - note that it says "The initial editions, at that time referred to as 'products', were Workstation, Server, and Cloud, and this split was initially implemented in the Fedora 21 release. An objective covered further development and refinement of the concept over the next few years, including the replacement of Cloud by Atomic, and settling on the name 'editions'."
Per https://web.archive.org/web/20170901000000*/getfedora.org ), getfedora.org switched from listing Cloud as an edition to listing Atomic as an edition between June 2016 and December 2016. Cloud base images were already buried (actually, more than they are now) at that point.
Aside from airy-fairy theoretical debates, those are the actual practical consequences of being an edition. If you want a practical answer to the question of "what is an Edition?" rather than a theological one, the answer is "they're the things up front in big letters on getfedora.org". So ipso facto, Cloud hasn't been one for about five years at this point.
I think it's worth considering that all the stuff you mention above about the install base of the Cloud images is true even though it hasn't been an Edition for five years. So it's reasonable to ask whether it really matters. I think you could note that, since the main practical consequence of "edition status" so far as Fedora users are concerned is the placement on getfedora.org, it doesn't necessarily matter much to Cloud because most people deploying in clouds do not do so by downloading image files from distribution download sites. They just pick an image to deploy from their cloud's deployment interface.
Thus the key question for Cloud, really, is "what Fedora image(s) do the major cloud vendors offer for easy deployment, and how prominent are they?" What's on getfedora.org is obviously less important (or else we'd have about as many Cloud users as we do LxQt users).
The main problem with not being on getfedora.org is the lack of visibility makes it difficult to attract contributors. And people who are new to the cloud don't even know we have a Cloud variant that they can use as they move from traditional server to cloud infrastructure.
We have a similar problem with the Fedora base container, too, but people seem to be less concerned about that at the moment (though it's on my radar of Things That Need To Be Dealt With).
Discussed in Cloud meeting today. The team has unanimously agreed that they would like to pursue Edition status targeting F36 as the earliest possible release for which we can make this change. Updates are planned to the PRD and we have additional work to be done. The issue is resolved however as we believe we have what we need to move forward on reclaiming the edition label with all that entails.
Metadata Update from @davdunc: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Metadata Update from @davdunc: - Issue untagged with: meeting
Login to comment on this ticket.