CPE has announced that they decided to switch to gitlab as the dist-git implementation for Fedora. The changes for git-forge and centos uses are obviously important, but don’t directly impact the distribution functions of Fedora. The evaluation that led to the decision was not public, so we don’t know if and to what extent the functions specific to a dist-git forge and Fedora package ownership rules were weighted in the process.
Previous experience suggests that it is easy to undervalue this specialized functionality, and that the lack of that functionality negatively impacts the functioning of the distribution. The switch from pkgdb to (as it was then called) pagure-over-dist-git was announced in summer of 2017 [1]. Important parts of functionality were only finalized in 2019, over two years later. Between 2017 and 2019 ownership changes required opening tickets that were manually handled by the releng team. The details of how this functioned changed over time; the important part is that adding those management functionalities to an existing hosting solution can be very hard. Another important part is that the new implementation never quite reached feature parity [2] despite 2½ years of incremental improvements.
In particular, easy-to-use dist-git management and integration with the build and update systems is very important for new contributors. Long-time contributors can deal with cryptic command line invocations and the requirement to jump between disjoint services, when they are intimately familiar with all the different services that the distribution provides and they closely follow changes over time. New or non-full-time packagers find it daunting. The reduction in the number of new packagers and the reduction of input from non-power-packagers over the last few years is most likely at least partly caused by the complication of the packager workflow. The lack of self-service ownership changes also discouraged people from getting involved with new packages in the past [3].
The collection of “user stories” collected in Fedora [4] does include mentions of this workflow, but not in a comprehensive way. (It seems that the “user story” approach encourages people to describe operations and aspirations in high-level terms, and not boring details like “bodhi must know if the user is in the owner group in dist-git to decide whether they can edit an update”). They were later generified to the point of being hard to recognize [5], and completely removed in the final version [6].
Access to dist-git by groups and the transparency of ownership are a fundamental ingredient of how Fedora functions, even though they are not part of normal git-forge requirements, and any plan to switch to a different service must include an evaluation of the impact on those features.
=== Status quo (i.e. current dist-git functionality):
=== Status quo ante (i.e. parts that we lost with the last dist-git move, but that people would still like to see implemented):
Fedora functions through the Change process. Putting the motivation, scope, responsibilities, benefits, and contingency plans in writing, and public discussing and voting on each change allows us to make decisions publicly and transparently, give voice to anyone who wants to be involved in the process, avoid misunderstandings that happen in spoken and informal writing, seek clarifications and safeguards whenever the initial text comes up short, leave a “paper trail” for retrospection, and finally serve as documentation both internally in the project and for other distributions. The process is not perfect, but we don’t have a better one. In a project the size of a distribution any single person or group cannot be sure that they have taken into account all the interactions with other uses and users of the project. In fact, many decisions have mixed impact, and we proceed with changes that impact some groups negatively while being generally positive. We need to publicly voice and weigh and decide especially in such complicated cases.
== Proposed FESCo statement
We request that the change to dist-git service be handled in a manner akin to the Change process: with a point-by-point discussion of functionality used by the Fedora project, before the decision is made. [8] makes a step in the right direction. Other infrastructure changes (koji and bodhi enablement, pagure-dist-git switch, pkgdb retirement, etc.) were handled in this manner [9], and most recently ELN [10]. The list of features above may be used as a starting point, though it was written without having any specific solution in mind. Some functionality may not be feasible in the new scheme. That doesn’t rule out the switch, but we should recognize what needs to change in our processes and what non-essential functionality will be missing and any possible workarounds, and the timing. If we change how the Fedora projects functions, we should do that consciously, and not as an effect of realizing that an already deployed implementation is different than expected. Given that the dist-git system lies at the core of how Fedora is developed (with many services relying on it as a source of truth for package sources, package metadata, and for access management), this needs to be publicly evaluated prior to switching to another platform.
[1] https://fedoraproject.org/wiki/Infrastructure/WhatHappenedToPkgdb [2] https://pagure.io/fesco/issue/2115 [3] The initiative to restart the orphaning of long-time FTBFS packages happened while manual tickets were required to un-orphan packages, which resulted in the need open a large number of tickets, and some contributors were clearly reluctant. [4] https://lists.fedoraproject.org/archives/list/council-discuss@lists.fedoraproject.org/thread/OEPDGVKYAJIQ6YYZU5J3XT3LHXFROFI5/ [5] https://lwn.net/ml/fedora-devel/CA+voJeVX+usYB+sNnKxBXVw4P3Nba=-66spPPVxnbLNuHjTRhA@mail.gmail.com/ [6] https://hackmd.io/@My419-DISUGgo4gcmO1D6A/HJB74lcL8 [7] Assignees for fedora and EPEL bugs separately from pagure-dist-git is a pending feature in pagure 5.9. This partially obviates per-branch ownership by publicly showing who is responsible for Fedora branches, and who is responsible for EPEL branches. See https://src.stg.fedoraproject.org/rpms/granite (bottom of the left navigation pane). [8] https://fedoraproject.org/wiki/Git_forge_update#Purpose_.26_Plan [9] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/4GXT3VVQZOLWMBEYAAEPYSGVUFT2A4PI/ [10] https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose
edit: packers → packagers ;)
Hi @zbyszek thanks for putting that together this is quite useful.
I am starting to take a closer at GitLab CE with the particular use case of dist-git (ie starting to look at solutions). I am going to put the result of that work in a wiki page using the Change Process as a template. So overall I agree that this should be going through the normal Change policy before we make the switch, I also anticipate that getting early feedback on the proposed solutions will be really valuable and help design a better solution.
Also as you mentioned some of the current features might not be feasible, in such case having the discussion with FESCO and the wider community will also be useful to find way forward and maybe use this opportunity to rethink some of this features or some of the Policies we are currently using.
+1 to @zbyszek's statement in general, but I haven't checked the list (e.g. if the numbers match the cases etc.)
Thank you @cverna.
+1 for the statement.
+1 for the statement, thanks for putting this together.
(Typo fix maybe, where you say "packers", I think you mean "packagers".)
Typo fix maybe, where you say "packers", I think you mean "packagers".
I updated the text to fix the typo. (In Polish, "packer" or "paker" is used to dismissively describe someone who goes to the gym to be all muscles. I guess packers would be daunted by our packaging process too ;) )
After a week, this has +4 (counting @zbyszek as well). I feel like this deserves more people voting, so I'm not marking this approved yet, but technically, it is.
+1.
Metadata Update from @ignatenkobrain: - Issue tagged with: pending announcement
Announced: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/VKMR3PEHNMDOAPB6JOCCB4XUYKAPIMR3/
@bcotton please make sure that this propagates to the people who are taking care of dist-git replacements (such as @lgriffin). Thanks!
Metadata Update from @ignatenkobrain: - Issue close_status updated to: Accepted - Issue status updated to: Closed (was: Open)
I agree with the general sentiment here of using a process like the one we use for Changes going forward to evaluate different requirements and make changes thoughtfully, so that we have the best chance of covering all angles, and so that we have better and more transparent communication.
I'm concerned about one part, though. Let's recognize that the CPE team doing the hosting and work for our dist-git platform has already definitely decided to go with GitLab. I don't see any situation where going back and asking them to-revisit that actually provides value — most likely, it'd just be a charade accompanied by extra paperwork.
With community volunteers, we don't have the ability to force people to do a certain thing. For example, if people don't want to make modules, we can't make them do it. People will often do things not in their favorite way if they see the overall value to the project — we're a community, not a random collection of self-interested autonomous bots. Obviously, Red Hat and a Red Hat funded team with supporting Fedora deliverables is somewhat different from a volunteer, but many similarities apply. Red Hat funds Fedora because of the benefit to the ecosystem, including RHEL and CentOS. If we want to convince a Red Hat team that it's worthwhile to do something in what they see as a non-optimal way, we may be able to do that. But that comes with a cost.
As I know you all know, Fedora does not have a huge source of independent funding, and while we have many awesome volunteers including contributors to infrastructure, we don't have an excess. So even if we would ultimately conclude that this is the hill on which we want to make a stand for some reason and ask CPE to not use GitLab for Fedora even though they're doing that for CentOS Stream and Red Hat presumably will use it for the next RHEL, we'd be reducing resources available to the project, to us — and we have plenty of other priorities.
So, I think any process going forward should be around how GitLab-for-dist-git will work and be deployed, not going back to first principles. I know that's going to be frustrating for some people, and, sure, there's plenty that could have been better in retrospect. (See the Council statement.) But here we are now.
I also think that having this go through a series of unconnected changes may not be the best approach. This kind of longer-horizon effort is exactly what we designed the Objective process for. I hadn't considered this before, but let me ask: would FESCo support the establishment of an 12 to 18 month Objective for "Update dist-git to a new GitLab-based system"?
@mattdm there's so many things here that I don't agree with that it's hard to know where to start, but I guess I'll start at the top ...
I think the general sentiment here was not that this should go through "something like the Change process", but that it should go through the actual Change process, the way other infrastructure Changes have been handled in fedora. I think @cverna is definitely trying to make communication more open here, which I do appreciate, but that doesn't negate the fact that most community members seem to have been blindsided by a Decision that was seemingly made behind CPE's closed doors - with no communication happening between the gathering of requirements and the announcement of the decision.
And that's just something I don't understand. With some of the issues @cverna has laid out in his GitLab ticket, I would hope that at least a contingency plan is in place, in case that GitLab will not work out for fedora. (For example, I find it ridiculous that GitLab does not support projects/repositories with the "+" character in their names, which is something that KDE and GNOME apparently also had issues with, but which has not been addressed by GitLab for either of those projects.)
This whole endeavour is starting to sound like "Dear Community Memb... erm, Passengers, our train is headed towards a ravine, but we decided for this route hours ago so there's nothing we can do. Prepare for impact, thanks for travelling with CPE."
There seems to be a misunderstanding as to what the statement said. It doesn't say anything about GitLab or any other proposed solution. The word "gitlab" only appears in the introduction to specify that this was written in response to the announcement. The statement is about functionality, and trying to prevent us from inadvertently backsliding on functionality that is crucial to how Fedora is currently put together. In fact, in https://gitlab.com/gitlab-org/gitlab/-/issues/217350, a similar list of functional principles is discussed as wanted in the gitlab implementation. I don't even think that there's any disagreement on this.
With community volunteers, we don't have the ability to force people to do a certain thing.
Not exactly. We definitely cannot force people, but we have certain general agreement as to what community members should do and shouldn't do, and processes for communicating, reviewing, and accepting or rejecting bigger changes. Even community volunteers don't have complete freedom with parts of Fedora that they maintain. I would expect that for Red Hat employees, the bar should be at least as high.
For a community distro which Fedora is, public communication and public decision making for the big stuff is absolutely fundamental.
I also think that having this go through a series of unconnected changes may not be the best approach.
I certainly hope we don't have a series of "unconnected changes". The ask was for something "akin to the change process", and at least I understand this to be about the spirit, and not about the precise details of how we do changes that apply to a single release.
This kind of longer-horizon effort is exactly what we designed the Objective process for. I hadn't considered this before, but let me ask: would FESCo support the establishment of an 12 to 18 month Objective for "Update dist-git to a new GitLab-based system"?
For me at least, this is putting the cart before the horse. I don't know if this is a good idea, because there has been no public evaluation. Why would we commit to year or more of work before evaluating if this work a) is feasible, and b) will leave us in better state than before?
Note that the decision process cuts both ways: if we all publicly discuss and reach some consensus for some change, and things fall apart and in retrospect we see that something was a bad idea, we all share the blame, and can't complain because we accepted the tradeoffs. That will happen sometimes: if we do things that weren't done before, we can't always get everything right. But if something is done as fait accompli by a very narrow group, people will blame that group for any shortcomings. With a project the size of dist-git, any change will have drawbacks, so we need a public acceptance process.
I think the general sentiment here was not that this should go through "something like the Change process", but that it should go through the actual Change process, the way other infrastructure Changes have been handled in fedora. I
I'm not trying to be cute or anything -- the accepted statement literally says "a manner akin to the Change process".
With some of the issues @cverna has laid out in his GitLab ticket, I would hope that at least a contingency plan is in place, in case that GitLab will not work out for fedora.
That sounds absolutely reasonable and I don't disagree.
Metadata Update from @zbyszek: - Issue untagged with: pending announcement
Login to comment on this ticket.