The Fedora Rawhide builds of the core libraries provided by glibc (C, threading, math, etc) are a rolling release of upstream glibc git master. The Fedora glibc maintainers feel they meet the goals of Rawhide as outlined (https://fedoraproject.org/wiki/Releases/Rawhide) and that all the releases are usable and at the same time represent the latest incremental improvements from upstream.
With a project of glibc's complexity there is a series of feedback loops that are important for both the upstream project and the downstream distributions that use glibc. The downstream distributions carry out important integration work that provides valuable feedback to the upstream project. The downstream distributions provide operational hours of testing for components in live environments which is otherwise impossible for upstream to carry out. Upstream in turn provides a platform for cross-distribution sharing of work and integration.
Upstream glibc has begun using downstream distributions to test out parallel and concurrency (P&C) changes in threading code; code that can't otherwise be tested without millions of hours of testing. Such code is reviewed, tested on as many machines as possible (Fedora glibc team tests on x86_64, i686, ppc64, ppc64le, s390, s390x, aarch64, arm), but it's final inclusion is dependent on an acknowledgement by the downstream distributions that it has operational integrity. To answer that request Red Hat and SUSE do the following: they integrate the patch into their rolling releases, SUSE into Factory/Tumbleweed and Red Hat into Rawhide. After a number of months of integration the patch is approved and it is checked in upstream for use by all the distributions. It could be said that upstream glibc is being too conservative here, and could checkin the patch, and just let everyone test the changes, but the reality is that only the mature and well-staffed distributions and hardware vendors (Intel, and IBM) are capable of triaging the eventual corner case problems with P&C changes. Therefore, commensurate with our community involvement and maturity, we assist upstream, with the hardware vendors involved if the changes reveal hardware-level defects. This process produces a very high quality upstream glibc that can be used by a wider assortment of smaller distributions that don't have the ability to triage P&C issues. Smaller distributions are still welcome to participate with SUSE and Red Hat in similar work, picking up patches from glibc's patchwork to validate as needed.
The point to clarify here is that Rawhide is being used by the Fedora glibc team as a means to facilitate upstream inclusion of features important to the Fedora glibc team. This appears to be slightly different than the intent of Rawhide which is to serve as integration for upstream changes which are already approved. The nature of the work is a singular cycle with distributions and upstream feedback going in both directions, and this feedback is important.
The optics of this are very important for our Rawhide users which may have volunteered to test Rawhide under a different understanding. Now it may be that FESCo considers this "normal" per the definition of Rawhide, but I am filing this ticket to clarify that what glibc is doing with Rawhide is an acceptable set of actions and in line with Rawhide practice, and if not, that some clarification be made for glibc usage. The language of the clarification would need to be discussed.
Thank you for taking the time to read this. If you have any questions please ask. If anything needs to be clarified, please ask.
The Fedora Rawhide builds of the core libraries provided by glibc (C, threading, math, etc) are a rolling release of upstream glibc git master. The Fedora glibc maintainers feel they meet the goals of Rawhide as outlined (https://fedoraproject.org/wiki/Releases/Rawhide) and that all the releases are usable and at the same time represent the latest incremental improvements from upstream. With a project of glibc's complexity there is a series of feedback loops that are important for both the upstream project and the downstream distributions that use glibc. The downstream distributions carry out important integration work that provides valuable feedback to the upstream project. The downstream distributions provide operational hours of testing for components in live environments which is otherwise impossible for upstream to carry out. Upstream in turn provides a platform for cross-distribution sharing of work and integration.
FWIW, this has worked fairly well for many years. I can only think of one case off the top of my head where something needed to be reverted.
Upstream glibc has begun using downstream distributions to test out parallel and concurrency (P&C) changes in threading code; code that can't otherwise be tested without millions of hours of testing. Such code is reviewed, tested on as many machines as possible (Fedora glibc team tests on x86_64, i686, ppc64, ppc64le, s390, s390x, aarch64, arm), but it's final inclusion is dependent on an acknowledgement by the downstream distributions that it has operational integrity. To answer that request Red Hat and SUSE do the following: they integrate the patch into their rolling releases, SUSE into Factory/Tumbleweed and Red Hat into Rawhide. After a number of months of integration the patch is approved and it is checked in upstream for use by all the distributions. It could be said that upstream glibc is being too conservative here, and could checkin the patch, and just let everyone test the changes, but the reality is that only the mature and well-staffed distributions and hardware vendors (Intel, and IBM) are capable of triaging the eventual corner case problems with P&C changes. Therefore, commensurate with our community involvement and maturity, we assist upstream, with the hardware vendors involved if the changes reveal hardware-level defects. This process produces a very high quality upstream glibc that can be used by a wider assortment of smaller distributions that don't have the ability to triage P&C issues. Smaller distributions are still welcome to participate with SUSE and Red Hat in similar work, picking up patches from glibc's patchwork to validate as needed. The point to clarify here is that Rawhide is being used by the Fedora glibc team as a means to facilitate upstream inclusion of features important to the Fedora glibc team. This appears to be slightly different than the intent of Rawhide which is to serve as integration for upstream changes which are already approved. The nature of the work is a singular cycle with distributions and upstream feedback going in both directions, and this feedback is important.
I agree that the intention in general is integration, but glibc is not the only component that does facilitation before something is upstream. The kernel has done similar in the past with uprobes, wireless drivers, and secure boot for example.
The optics of this are very important for our Rawhide users which may have volunteered to test Rawhide under a different understanding. Now it may be that FESCo considers this "normal" per the definition of Rawhide, but I am filing this ticket to clarify that what glibc is doing with Rawhide is an acceptable set of actions and in line with Rawhide practice, and if not, that some clarification be made for glibc usage. The language of the clarification would need to be discussed. Thank you for taking the time to read this. If you have any questions please ask. If anything needs to be clarified, please ask.
I believe the seriousness to approach and competency the glibc team has repeatedly shown make the current usage perfectly acceptable. I'm very supportive of this and would like to see it continue. The value Fedora provides to the Free Software community is immeasurable and it lines up very well with all four of our Fedora Foundations.
+1
Just to reiterate for those commenting on the issue:
It seems that jwboyer and sgallagh have agreed that it is acceptable behaviour.
The question remains: Do we need to clarify this anywhere?
Just to reiterate for those commenting on the issue: Is glibc's approach to using Fedora Rawhide acceptable? Do you think we need any clarifications in the description of Fedora Rawhide (https://fedoraproject.org/wiki/Releases/Rawhide)? It seems that jwboyer and sgallagh have agreed that it is acceptable behaviour. The question remains: Do we need to clarify this anywhere?
Is glibc's approach to using Fedora Rawhide acceptable? Do you think we need any clarifications in the description of Fedora Rawhide (https://fedoraproject.org/wiki/Releases/Rawhide)?
It seems that jwboyer and sgallagh have agreed that it is acceptable behaviour. The question remains: Do we need to clarify this anywhere?
In my opinion, no. This has been the status quo for many years already, with a proven track record of success. If you really feel you need to write something, I don't think FESCo would object but I don't beleive it's a requirement.
With that being said, if there is a particularly large or "worrisome" change coming in a particular version of glibc that you feel people should pay extra attention to, we can always use the Change process. That would allow you to highlight the feature/change using the existing process and give people an extra heads up. It might be a nice middle ground for such scenarios.
If the consensus on this ticket is that we don't need to clarify anything, then I'm happy with that. Personally I don't believe it needs to be called out. I consider this part-and-parcel of what Rawhide is all about, a connected feedback loop with upstream that cycles very quickly.
I've created a section here explaining the issue, and will redirect developer questions there: https://fedoraproject.org/wiki/Glibc
Absolutely agreed, we use the Change process every release to highlight the rebase and any other changes which may be disruptive.
With that being said, if there is a particularly large or "worrisome" change coming in a particular version of glibc that you feel people should pay extra attention to, we can always use the Change process.
AIUI, the Change process is for branched/future-release changes, not for transient situations within Rawhide.
I think the concern here was for occasions when something considerable is changing or a new feature is being added. (E.g. addition of a new name-service module or a change in a default, etc.)
AIUI, the Change process is for branched/future-release changes, not for transient situations within Rawhide. I think the concern here was for occasions when something considerable is changing or a new feature is being added. (E.g. addition of a new name-service module or a change in a default, etc.)
Exactly. In this case it would be the new POSIX condition variable implementation.
A month of operational testing and we've found one instance of undefined behaviour in Berkeley DB (https://bugzilla.redhat.com/show_bug.cgi?id=1394862), which underscores for me the value of operational hours of testing in Fedora Rawhide. I want to thank the Fedora glibc team on the speed and accuracy of their triage and analysis for this potentially blocking bug.
I would still like to see this issue come before FESCo to decide if our usage is acceptable.
To be clear, this ticket is equivalent to bringing the issue before FESCo. All FESCo members can see it and comment. A meeting is only required if there's actual high-bandwidth discussion needed.
FESCo members, please indicate your support or concerns in the ticket. This has been open for a month and we owe an answer.
I really thought we had made this plain earlier, but perhaps it wasn't clear. I'll suggest a Proposal:
"FESCo asserts that while in the general case Rawhide should not be used as a place to dump untested software, there is real value in allowing certain low-level packages, including (but not limited to) glibc and gcc to provide pre-release versions there. The benefits of the early real-world testing of and upstream collaboration on these key packages far exceeds the risks that they may introduce."
+1 from me, FWIW.
I really thought we had made this plain earlier, but perhaps it wasn't clear. I'll suggest a Proposal: "FESCo asserts that while in the general case Rawhide should not be used as a place to dump untested software, there is real value in allowing certain low-level packages, including (but not limited to) glibc and gcc to provide pre-release versions there. The benefits of the early real-world testing of and upstream collaboration on these key packages far exceeds the risks that they may introduce."
Where do you see this proposal being documented, communicated at? Just in this ticket, or are you envisioning it being placed somewhere more prominent?
Just in this ticket would be fine for me. Ideally I'd like the ticket to move out of OPEN into a resolved status.
I would still like to see this issue come before FESCo to decide if our usage is acceptable. To be clear, this ticket is equivalent to bringing the issue before FESCo. All FESCo members can see it and comment. A meeting is only required if there's actual high-bandwidth discussion needed. FESCo members, please indicate your support or concerns in the ticket. This has been open for a month and we owe an answer.
To be clear, this ticket is equivalent to bringing the issue before FESCo. All FESCo members can see it and comment. A meeting is only required if there's actual high-bandwidth discussion needed. FESCo members, please indicate your support or concerns in the ticket. This has been open for a month and we owe an answer.
I am also very much supportive to what glibc is doing in rawhide.
+1 to this proposal.
+1 of course.
We have 4 +1 FESCo votes. We need one more for consensus. I believe that consensus exists fully anyway, but we need people to vote.
+1 here.
I could see this being documented either on the Rawhide wiki page, or the Updates_Policy page (or both).
I've made the following wiki edits:
https://fedoraproject.org/w/index.php?title=Updates_Policy&diff=prev&oldid=480509
https://fedoraproject.org/w/index.php?title=Releases/Rawhide&diff=prev&oldid=480510
Feel free to adjust that if desired. :man_with_gua_pi_mao:
@kevin changed the status to Closed
Closed
Login to comment on this ticket.