Fedora depends on Pagure, a home-grown git forge that lacks important features and runs in an inconsistent and undermaintained capacity. From providing features a growing community needs to responding to security issues or incidents, reliance on Pagure represents a vulnerability for all concerned. Our goal is to eliminate this vulnerability by moving workloads to one or more maintained alternative tools, preferring free and open source software wherever possible.
In order to do this right, we need community involvement when gathering and reviewing the requirements that a forge must be able to provide, and when determining which options suit these needs the most. We must also acknowledge that not every requirement may be met by any one tool. To finally progress this work, we need to begin the conversation anew and agree on core and critical requirements the Fedora Project must have in order to continue to build and release on its current cadence, and to grow for the future. Therefore, I propose the following in order to begin this work in a transparent and open way:
I believe these steps are needed to guide both the conversation to work, and the work to delivery. There are nuances to be added and points to be fleshed out, but this is the general flow of work required at a very high level. The request to the Fedora Council is to advise on whether the above steps are the right way to proceed, and give suggestions and feedback on improving the initial plan so that the whole Fedora community can feel confident that these decisions are made in the proper Fedora way.
I've created a topic on Fedora Discussion for this ticket.
Please keep this ticket focused. Discuss there, and record votes and decisions here. Thanks!
Metadata Update from @jflory7: - Issue priority set to: Coming Up (was: Needs Review) - Issue tagged with: policies
For the record, other related discussion threads are under this tag:
https://discussion.fedoraproject.org/tag/git-forge-future
Metadata Update from @amoloney: - Issue assigned to amoloney
Discussed in our 2024-04-10 meeting.
The announcement following our 2024 Hackfest is now published on the Community Blog.
For next steps, the Council will review a list of high-level requirements we would like the investigation by Fedora Infrastructure / Red Hat Community Platform Engineering to use. @amoloney will share these soon. We are mindful of including stakeholders from specific key groups relying on our git forges, like QA, release engineering, Infrastructure, Design Team, and others.
To keep feedback in one place, we will continue using the #git-forge-future tag on Fedora Discussion primarily. As a reminder, please share feedback and comments in the Fedora Discussion tag. Thanks!
#git-forge-future
The set of requirements provided to CPE for the investigation are as follows:
Suitability for replacement of Pagure and bugzilla for release issues (change process, blocker bugs, etc)
Suitability for replacement of Pagure for SIG and Team ticket tracking (e.g. FESCo tracker)
These represent a generalised, top-level view of the requirements the project is most likely to need holistically and further refined requirements are expected to fall in under some of these high level requirements per 'department' .
Please ack these requirements, or provide feedback and suggestions on iterating/adding to the requirements set by no later than Friday 12th April.
cc: @mattdm @t0xic0der @ffmancera @rwright @jflory7 @sumantrom @jonatoni @dcantrel @bt0dotninja @smeragoel
Will the packagers as a whole also be surveyed for their requirements? It seems that no part of this effort has been brought to the devel list—i.e., the canonical place where packaging is discussed—which is unfortunate given that we are the primary users of distgit.
packager
Certainly, and thank you for adding where the best place to engage with this group is too. All of the discussions in recent months have been taking place on discussions.fpo, as I'm sure you're aware of the effort to move project discussions from email to here. From a quick read of the packager guidelines page, it mentions that the packaging email list is where the packager committee discuss guidelines, etc. Do you suggest using this email too when requesting input from the packaging group?
In general, development discussions don't happen on Discourse except for Change documents. So when soliciting feedback from the Fedora developer community, discussion.fp.o isn't the place to go.
Re the discussion piece: that's why I said effort and not mandate (or other word to convey this meaning) to move our discussions over to discourse :) This shift is something the project is moving towards slowly and afaik the change proposal process is/was the first in this effort, but there have been other discussions happening in that forum and the git forge evaluation conversations have been happening (quietly) there too, so there are conversations taking place there which is nice to see.
All that aside though, coz it's wildly off-topic and sorry for my part in it, the requirements shared above are not a finite list by any means and as said in the comment, they are to represent the needs of a project in a more generalised way for the council to review and ack on. They are based on a general understanding of what we know is needed for the project as a whole from previous discussions on this topic, but we fully expect the project to have way more requirements and quite specific ones too. The formal requirements gathering phase has not started yet, and there will be input required from all groups on what they would need from a git forge before the investigation begins. And the requirements gathering phase will include posting the call for requirements in as many venues as possible to maximise awareness, including mailing lists and discussions.fpo.
I am working with the CPE teams product owner, @humaton, who will be overseeing the investigation work and we have been fleshing out a message to share out to the community on what we need from the and how they can/need to/should get involved in this work. We need to make sure all requirements are recorded and complied. We need to re-share this formal set of requirements again to the community to validate they are captured correctly before the investigation starts. And then we need to plan the investigation itself - have it be community-driven, establish timelines and reasonable outcomes, information sharing, etc. Keep an eye out for an email and post from @humaton in the next few days (if not tomorrow, then early next week) asking for the more detailed requirements from the community, and please reply to the message with the needs your group has for git.
I hope this makes sense, and I didnt mean to write an essay but I was probably not clear enough at the start of where we truly are in our evaluation journey!
@amoloney, +1 to the list you've made above.
Add deez more points.
Do you suggest using this email too when requesting input from the packaging group?
No, I suggest requesting input from packagers via the devel list. I confess that I am not particularly aware of how the CPE investigation process usually works, but I am surprised that this discussion started here 4 months ago and still hasn't been brought to the packagers—the primary users of distgit. I understand that this is a larger, project-wide issue, as it seems the effort encompasses replacing both pagure.io and distgit, but I would strongly suggest additionally bringing the requirements before the devel list and a FESCo vote like we do for other engineering decisions to ensure that side of the project is accounted for. distgit is a crucial packager tool and the issue is somewhat sensitive, so please make sure that we are included as well.
Suitability for dist-git and src.fedoraproject.org replacement
This would include:
1-3 are implemented via a homegrown Pagure plugin (https://pagure.io/pagure-dist-git) that would need to be reimplemented in Gitlab or Forgejo (if they even allow this type of plugin) or as a separate webapp.
Metadata Update from @amoloney: - Issue tagged with: In Progress
Metadata Update from @amoloney: - Issue tagged with: Next Meeting
@mattdm @jflory7 @jonatoni @ffmancera @t0xic0der @asamalik @bookwar @rwright @jbrooks
Can you please vote +1 that you are happy with the below criteria we would like the git forge report to contain in order to assist us with a decision? If there are other criterion we want, please add it as a comment.
A one page summary for each forge options, answering the following:
The council would also like a more in depth analysis of the investigation and a walk through by the team at a council meeting, but the report should capture these key details as a succinct summary on each forge option.
Metadata Update from @amoloney: - Issue priority set to: None (was: 3)
I would like to express my frustration with this innitiative, and share how I wish this would be going.
I'm really struggling to understand how writing sentences like "As a developer I need to be able to push to a git repo" in a pagure ticket is useful...
But let me start start with the wishes:
We want to replace a git forge. We have identified two options, and need to choose one of them.
What I would like the investigation team to do:
Get those two forges running.
Identify groups of users in Fedora and approach them directly. You'll have a bunch of package maintainers, people writing Docs, QE, etc.
Figure out what they do with git forges, and set up whatever is necessary for them to try both. Have actual conversations with them. Don't write those tens of user stories in a discourse post. Figure out what they need to test, and make it testable for them.
Then ask them about how each git forge works for them, or if there's anything else they need to try.
And as they're testing it, focus on how easy/hard it is to run each option. What errors pop up. How upgrades work. What's the resource consumption like. You'll have people test their actual workflows, so this should help discover all sorts of hidden operations-related gotchas.
The output I want is something that tells me "We tried these two git forges with these 17 user groups. Here's how each forge worked, group by group. Forge A comes with these tradeoffs, and Forge B comes with these tradeoffs. The most affected groups are these ones, in this way."
Then we can make a sensible decision.
And now the frustration:
As I said, I don't thing listing these user stories in a ticket is helpful.
This is what I mean: https://pagure.io/fedora-infra/arc/issue/164
I understand the concept of requirements gathering. I don't agree these user stories, in the way we're having them right now, are helping.
For this specific task, I'm not even sure we need to start with requirements. We already have two git forge implementations, deployed, ready to be tested. Working with users on testing those should be the way this is done, in my opinion.
But this whole thing seems to have turned very abstract. With very little connection to actual people testing their actual workflows.
Can we forget these user stories and rather focus on comparing the two options, with people who'll need to use a git forge? And enable them to test rather than abstractly describe what they need to do?
@amoloney I agree with what you're proposing. But I'm not entirely sure we can get to a sensible version of that with this current approach.
I cant say I disagree with any of that and thank you for being so candid about things @asamalik . Were at a pivotal point in this investigation, so if we need to make changes, now is the time to do it. No one wants this to drag on, but if the way things are happening now will only result in an exercise in futility, lets look at what we should stop doing, keep what is working and figure out how to proceed better. I can be responsible for this effort, but I need collaboration from you all in what we want. Can we make this discussion a priority for tomorrows meeting please?
Hello there,
@mattdm @jflory7 @jonatoni @ffmancera @t0xic0der @asamalik @bookwar @rwright @jbrooks Can you please vote +1 that you are happy with the below criteria we would like the git forge report to contain in order to assist us with a decision? If there are other criterion we want, please add it as a comment. A one page summary for each forge options, answering the following: Can this (either git forge) be supported through CPE? Y/N Are there technical barriers to favour one over the other? What does the workflow impact look like for teams An estimate of resource consumption for each Hardware Storage People-Power An estimate of the relative effort for migration from current setup to new thing An estimate on the relative ongoing maintenance for each option What are the big gaps for each option when comparing them to the current git setup and user stories An estimate of complexity of migration plan for each The council would also like a more in depth analysis of the investigation and a walk through by the team at a council meeting, but the report should capture these key details as a succinct summary on each forge option.
@amoloney, for now, we have scoped this ARC investigation initiative to the comparison of the Git Forge choices concerning the usecases that the community members are likely to have with it. In the report, we aim to include points associated with the "possibility of deployment support", "the technical barriers associated", "estimate of resource consumption" and "estimate of ongoing maintenance efforts" with each of the curated options in great detail.
As for the "estimate of migration efforts" and "the complexity of migration plan", those are beyond the scope of this specific ARC investigation initiative. Please refer to the Dist Git Move ARC investigation (which can be found here https://fedora-arc.readthedocs.io/en/latest/dist-git-move/index.html) that covers that aspect in detail (as mentioned in the mailing thread with the Fedora Quality Engineering folks and Community Platform Engineering team).
Regarding the "workflow impact to the teams with the move" and "hurting omissions from the options", we have reached out to multiple subprojects (that might or might not be strictly involved with just packaging and releases) for them to represent their user stories. As such, this aspect would be made available in the report on a best-efforts basis depending on how well-represented the user stories are in the ARC tracker (here, https://pagure.io/fedora-infra/arc/issue/164).
I have been spearheading this both in the capacity of an "owner" of the initiative from Fedora Council as well as the initiative lead for the ARC investigation work from the Community Platform Engineering. While still in progress (here, https://fedora-arc.readthedocs.io/en/latest/dist-git-comparison/index.html), @humaton and I plan on delivering a presentation as a part of the Fedora Council video meeting immediately after the originally planned handover date.
@asamalik,
I would like to express my frustration with this innitiative, and share how I wish this would be going. I'm really struggling to understand how writing sentences like "As a developer I need to be able to push to a git repo" in a pagure ticket is useful... And now the frustration: As I said, I don't thing listing these user stories in a ticket is helpful. This is what I mean: https://pagure.io/fedora-infra/arc/issue/164 I understand the concept of requirements gathering. I don't agree these user stories, in the way we're having them right now, are helping. For this specific task, I'm not even sure we need to start with requirements. We already have two git forge implementations, deployed, ready to be tested. Working with users on testing those should be the way this is done, in my opinion. But this whole thing seems to have turned very abstract. With very little connection to actual people testing their actual workflows. Can we forget these user stories and rather focus on comparing the two options, with people who'll need to use a git forge? And enable them to test rather than abstractly describe what they need to do?
While I do not completely agree with your statement, given that the user stories gathered so far (around 150 of them so far) have been helping us understand the requirements (the "stories" part) in a succinct manner as well as the reason why it is needed by them (the "user" part) - I understand where you are coming from. It is extremely difficult to find a one-size-fits-all all kind of approach and we would be interested to know how we can do better going forward.
I appreciate your feedback on this. I just wished it was us (me taking off the Fedora Council hat temporarily to put on the Community Platform Engineering one) that this feedback was subjected to and in at least the foundational phases of the ARC investigation initiative. With the timeline closing in, there is little scope for being able to move things around in a way that makes subjective sense and doing that might end up being counterproductive as a whole.
We already have them for quite a while now.
GitLab - https://gitlab-gitlab-system.apps.fedora.cj14.p1.openshiftapps.com/users/sign_in
Forgejo - https://forgejoroute-communishift-forgejo.apps.fedora.cj14.p1.openshiftapps.com
They have been duly announced in a Fedora Discussion thread (which can be found here, https://discussion.fedoraproject.org/t/inviting-testers-for-git-forge-usecases/129016), in multiple Fedora Project mailing lists (here, https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/F3R2DNPL6LTJOP2ENKJ3VH5MSNOM6JJZ/ and here, https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/VKWVMAMFV2AI3BJ53EN4YQEGR7SQOTPN/) over six months back, in a talk in the Fedora Linux 40 Release Party (here, https://github.com/gridhead/gridhead/blob/master/FL40RP_1943IST_24May2024_GitForgeARCInvestigation.pdf), in a talk during Flock To Fedora 2024 (here, https://youtu.be/KiG9H7t7EHk?si=AIZU9fdR4DxGFBf8) among other places to ensure that these do not fly under the radar. After all, having these things available for a while and folks not knowing about them would not be of much use now, would it?
Identify groups of users in Fedora and approach them directly. You'll have a bunch of package maintainers, people writing Docs, QE, etc. Figure out what they do with git forges, and set up whatever is necessary for them to try both. Have actual conversations with them. Don't write those tens of user stories in a discourse post. Figure out what they need to test, and make it testable for them. Then ask them about how each git forge works for them, or if there's anything else they need to try. And as they're testing it, focus on how easy/hard it is to run each option. What errors pop up. How upgrades work. What's the resource consumption like. You'll have people test their actual workflows, so this should help discover all sorts of hidden operations-related gotchas.
We did.
We reached out to folks from a bunch of SIGs and Subprojects within Fedora Project. Either as a part of their Matrix channel or by reaching out to a representative of their team and asking them to share the word. We have asked them to head back to the original ARC investigation issue tracker where we are collecting the user stories (here, https://pagure.io/fedora-infra/arc/issue/164) in a format that suits the purpose the best and that has been met with great reception.
Our focus was to ensure that we did not lose focus of the requirements gathering phase while discussing "how one could get access to the testing deployment of GitLab/Forgejo" or "why SSH cloning is not enabled for the repositories" and that is why we purposefully limited the entry scope in the issue ticket to be in the format of a user story while the remaining things could be discussed either as a part of the Fedora ARC matrix channel or a Discussion Thread.
We attempted to ensure that our ARC investigation report is as in-depth as possible. There are likely multiple user stories that can be useful only in exceptional circumstances for a certain subproject or SIG but even those have been given an equivalent extent of attention. Having a focus superficially on a certain set of user groups might have made the report easy to digest at the cost of forsaking a bunch of user stories that might not be very common in usage.
I plan on conducting a retrospective survey at the end of the ARC investigation initiative. While that will not be able to un-mess up the things that were perceived to be messed up in this ARC investigation initiative - I want to take this as a learning experience for both myself and my team. As appreciative as I am of the feedback I get, I recognize that those can be only so helpful under the time, amount and circumstances that they have been provided in.
I'm seeking my inbox during my PTO - I need to get a life and I really should stop doing that.
@t0xic0der An approx of the migration effort should be in the report. It would be helpful to have something simple like '16 apps need to be rewritten from the ground up and 12 need to have API changes, etc for option A and 28 need this for Option B. Most rewrites for option A are in the critical path, less are in option B so we will need to work carefully around the release schedule for option A, etc'. Linking the previous investigation is fine for additional reading, but it is not out of scope to have this in the investigation report.
Things are messy, agreed. But they're not bad or broken. Some things might just need to change and that might take more time than we would have liked, but this is the opportunity now to figure out what we need need to make this decision and if what we need is not on the ARC teams radar, and/or we need QA to be more involved or other teams to get more involved, lets identify that. This might mean extending the investigation to cover other aspects. This might mean just going with what we have already. We've hit the Cost - Time - Scope triangle issue, with the cost being poeples time, the time being our desire to get a decision made, and the scope starting to creep to maybe needing to include Bugzilla.
We have instances, the user stories can potentially be grouped, but were missing some testing of each instance for performance, workflow compatibility, etc. If we need that to make a decision, and that comes with a cost of a few extra weeks to allow other teams time to plug in data and set up some tests to help validate user stories, I believe the delay is worth the gain. What do we need?
believe the delay is worth the gain. What do we need?
It may be. Let's ensure we document each delay we've encountered though to retro on it later. We've had a lot of delays and it would be awesome to be able to show how each has been helpful and possibly inform future planning.
I would like to express my frustration with the ignorance of this council member.
Yes it is and as we described multiple times this is an input form there is an output from it that is moderated to reflect use cases of different user groups: https://fedora-arc.readthedocs.io/en/latest/dist-git-comparison/stories.html It's not finished therefore it is not publicized yet.
But let me start start with the wishes: We want to replace a git forge. We have identified two options, and need to choose one of them. What I would like the investigation team to do: Get those two forges running.
The ignorance here, forges are running for months.
And we did exactly that and talked to different members of those groups, some even submitted stories on their own without our involvement.
Figure out what they do with git forges, and set up whatever is necessary for them to try both. Have actual conversations with them. Don't write those tens of user stories in a discourse post. Figure out what they need to test, and make it testable for them. There are rpms organizations in both deployments with package mirrors and we are working on mirroring the issue tracker as well. Then ask them about how each git forge works for them, or if there's anything else they need to try. And as they're testing it, focus on how easy/hard it is to run each option. What errors pop up. How upgrades work. What's the resource consumption like. You'll have people test their actual workflows, so this should help discover all sorts of hidden operations-related gotchas. The output I want is something that tells me "We tried these two git forges with these 17 user groups. Here's how each forge worked, group by group. Forge A comes with these tradeoffs, and Forge B comes with these tradeoffs. The most affected groups are these ones, in this way."
There are rpms organizations in both deployments with package mirrors and we are working on mirroring the issue tracker as well.
I am sorry you decided to voice it now and not months in the past or even that you didn't join the group to do the work. But writing comments like this weeks before we are supposed to be finished is not helpful.
Then we can make a sensible decision. And now the frustration: As I said, I don't thing listing these user stories in a ticket is helpful.
And I don't think this comment is helpful at all.
This is what I mean: https://pagure.io/fedora-infra/arc/issue/164 I understand the concept of requirements gathering. I don't agree these user stories, in the way we're having them right now, are helping. For this specific task, I'm not even sure we need to start with requirements. We already have two git forge implementations, deployed, ready to be tested. Working with users on testing those should be the way this is done, in my opinion.
Everyone has the right to an opinion, I chose this way its not what you like you could have voiced your concerns sooner.
That is just because you didn't look at the actual deployments.
I am more than happy to give you all the keys from this initiative to make it happen as you wish.
OK, something with what I wrote went very wrong. I'd like to apologise if this touched anyone's feelings. Especially to @humaton and @t0xic0der who are driving the technical work. I didn't mean it as a critisism of anyone.
I shouldn't have posted it like that, I should have slept on a draft and edit it in the morning.
Please let me try better.
First, please let me clarify the "my frustration with this innitiative" bit. What I meant here was purely the process that we're all part of, including me. It wasn't about any person or any group of people.
Some other clarifications: - I know we have the test intances of both forges — I even saw your talk at Flock. I think you've done great work, I just never said that. - I know you have identified user groups. - I know people are actually trying it.
Also, I wasn't on the Council when this innitiative started. I believe that joining a project that's been running for some time requires observation first.
I admit the tone of my very first comment about the innitiative wasn't great. Again, I'm sorry for that.
It probably all boils down to: I think this innitiative isn't very well organised.
We have people doing a lot of work deploying the two instances, people testing them, etc. That's all great.
But regardless all that, I saw people frustrated, I saw people confused, I've been part of conversations that weren't productive.
Why is that?
Most people I talked to in person (including myself) didn't like or understood the long list of user stories. We found it very hard to navigate, and questioned how effectively it can possibly capture all the possible things we, as a community building a Linux distro, need our git forge to do.
Again, please don't take this bit personally, I really don't mean it against any person. But some people found the user stories discouraging or even off putting. And I couldn't help myself but agree.
And don't take me wrong, I might have done the same thing at the start. And I'd be having the same reactions right now. It might not have been obvious at the start. All I'm saying is that's how it feels to me now.
So I thought: Can we do it differently now?
It's very easy to see different ways to do things after someone else tried it. So I'm not trying act like I'm so clever I'm gonna save the day.
I'm just trying to make everyone's life easier, and help us make a decision that's gonna work well for the project.
Anyway. Can we do it differently now?
Of course we can! Will it be better? I don't know. Here's my prpoposal:
We already have the instances running, groups of users identified. That feels like a huge amount of work has been already done. Excellent.
I'd like to forget the user stories entirely.
Rather, have the user groups try the two forges, and have them report back what worked well and what didn't work well. And try to see if you can make more of it work for them in the test environments.
And then write down how it went. What worked well in each forge for specific groups, and what didn't. Add what you learn running them. And that's your report to the Council.
And then we can have a discussion about specific tradeoffs, knowing how each option works for different kinds of users.
I believe that's the best way to reach a decision.
Do you think this would work?
I really believe that focusing on testing and completely leaving out the user stories would make it easier for everyone.
I cannot believe this, but I am CLOSING this ticket as RESOLVED :tada:
Following the announcement that the Fedora Project will choose Forgejo as our new Git Forge replacement, I will close this ticket now as the evaluation is complete.
Thank you EVERYONE!!!! :grinning:
Links: Forgejo Announcement Git Forge Evaluation - ARC Team
Metadata Update from @amoloney: - Issue close_status updated to: resolved - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.