I would expect that you're going to need to plan for Pagure support. Fedora sources are still on Pagure.

Probably for a lot of the early experimentation (where we don't necessarily need to be doing it to prod), we can work with src.stg.fedoraproject.org.

For some reason, my comments are not posting.

We should at least ask fedora infra for their input on this. We may or may not like the answer, but considering the scale of the project, I would rather get them involved early.

I mentioned this in the meeting today, but I'll add it to here as well: Using src.fedoraproject.org will simplify things for us in the form of being able to deploy custom extensions to enforce source-git workflow requirements and being able to do things like enforce DCO and FPCA membership.

Additionally, I'd like to bring this over to the openSUSE community (they've got code.opensuse.org) and other communities using Pagure, as they will be interested in this there.

I'd say src.fp.o is the solution for the long-run but I'd love we could start prototyping on pagure.io - would such iterative approach work?

I've already tried to comment here without success, so let's try again.

Regarding the implementation.
The code that manipulates the forge is interoperable thanks to our ogr library. (=We can share implementation with centos-stream on GitLab.)
* We already use Pagure implementation for our upstream->dist-git workflow known as propose-downstream.
* Some parts of the Pagure implementation are not used in production but the possible problems can be fixed on the way.
The information that gets from the forge to us needs to be implemented -- via webhooks or fedora-messaging.

So far, we haven't needed any git hooks for our services and everything is done using API or project/branch settings. And for stream, we need to support a similar workflow on this forge. So from that point of view, I don't think GitLab cannot be used. (I am not saying it must be. Just that it can be if we want.)

And lastly, some updates: @jpopelka is currently taking a look at this more deeply -- how we can support Pagure instances and what needs to be done to do so.

And lastly, some updates: @jpopelka is currently taking a look at this more deeply -- how we can support Pagure instances and what needs to be done to do so.

The difference is that CentOS Stream on GitLab.com has no committers except Red Hat employees. That drastically simplifies the problem domain. That won't work for Fedora, since we're ~80% non-Red Hat.

I spoke with the CPE team recently and they are still planning to have a GitLab instance in Fedora ecosystem.

Honestly, this issue prevents me to have a good sleep in the night :D

I'm wondering what we can do here: my biggest issue is that one forge will not fit everyone. Would it be a big problem if we had official namespaces in two different forges?

Not really. It should be fine with two of them as long as nobody screws up the setup for GitLab.com.

From my POV, I'd rather Fedora src-git always used (or at the very least permitted) GitLab (self-hosted or gitlab.com), principally because it is a service I'm using daily for upstream work, for CentOS Stream and for RHEL. There is a lot of value in consistent platforms across the board, since these distros are closely related and have many common maintainers and workflows.

As well as bringing familiarity with the platform, it also means we can benefit from a variety of tools and services that have built around GitLab's REST API. In particular I maintain the 'bichon' project which provides an interactive terminal UI for code review (https://gitlab.com/bichon-project/bichon), there is also the 'lab' command line tool for code review and other mgmt tasks (https://github.com/zaquestion/lab/), together these provide a nicer user experience for maintainers who prefer terminal interfaces over web interfaces. There are also bots people have created for various tasks which could potentially be reused. Using Pagure puts Fedora in its own garden where people have to re-create or port these tools. This is an undesirable burden unless there's some truly compelling unique feature only Pagure offers to outweigh the downside.

@berrange thank you for giving us your point of view!

This indeed seems we have 2 camps here and should cover both git-forges: gitlab.com and pagure.io

Since packit already supports gitlab and we have a ton of code for it, we'll get it almost for free. Hence the main task for us will be to start consuming pagure.io events (which shouldn't be such a big deal because we also have some pagure code)

This indeed seems we have 2 camps here and should cover both git-forges: gitlab.com and pagure.io

The criterion I would use is: "contributor should be able to submit a merge request to any Fedora package without relying on a proprietary forge." That implies that if you want to support gitlab.com, it should be possible for me to ignore it and still submit a MR to the package anyway. Is that how src-dist works? I suspect not?

We should probably support gitlab.fedoraproject.org, not gitlab.com. GitLab's open source edition is really good. Almost anybody who likes gitlab.com should probably be satisfied with a hypothetical future gitlab.fedoraproject.org. And if we don't want to take the time to set up an open source GitLab instance to really do this properly, then pagure.io is just fine too.

@catanzaro thanks for your opinion! I wanted to give others time to reply and 12 days is probably enough :D

"contributor should be able to submit a merge request to any Fedora package without relying on a proprietary forge."

This is a reasonable request but to some extent, we rely on the Fedora leadership - if they choose to adopt GitLab we won't have another option but support it as the official platform.

Nevertheless, our current plan is to establish support for pagure and gitlab since we are getting requests for support for both. We intend to mirror b/w the places - I'd say mirroring is gonna be a great solution to address the need for multiple forges. And in the end, dist-git is still gonna remain as a source of truth - once the source-git workflow will become proven, we can talk about making the primary workflow for certain packages. Until then, dist-git is still the king.

Related: https://pagure.io/fedora-source-git/sig/issue/7

To be clear, we can support GitLab without relying on any proprietary forge by deploying the open source edition of GitLab.

It would be strange for Fedora to reject the readily-available and widely-adopted open source solution here in favor of proprietary software. Debian, GNOME, freedesktop.org, and KDE are all using open source GitLab. Red Hat uses it internally. It has proven to be quite successful.

To be clear, we can support GitLab without relying on any proprietary forge by deploying the open source edition of GitLab.

It would be strange for Fedora to reject the readily-available and widely-adopted open source solution here in favor of proprietary software. Debian, GNOME, freedesktop.org, and KDE are all using open source GitLab. Red Hat uses it internally. It has proven to be quite successful.

That is not what CPE wants to do though as far as I know. They're working on GitLab.com (proprietary SaaS): fedora-infrastructure#10028

Red Hat uses it internally. It has proven to be quite successful.

While it is true Red hat has an internal self-hosted deployment (2 in fact), Red Hat has even more significant usage of the public hosted gitlab.com service, eg all CentOS dist-git & src-git work is on the public gitlab.com site. Taking on the task of maintaining a gitlab.com is of course possible, but it is a significant committment in terms of system administration, so it needs buy-in from whomever would end up doing that work.

Nevertheless, our current plan is to establish support for pagure and gitlab since we are getting requests for support for both. We intend to mirror b/w the places - I'd say mirroring is gonna be a great solution to address the need for multiple forges.

I'm not convinced having two copies of src-git for each package is a sensible thing. Accurate mirroring in both directions, with potential conflict resolution needs is a non-trivial task to get working reliably. Even a small rate of problems will turn into a large administrative burden as the number of packages using src-git increases. Having a choice of options for hosting src-git makes sense, but once a project has picked their choice, I don't see a need/benefit from introducing the mirroring to the discarded choice. The important thing is more how we inform contributors where the designated src-git repo actually is. This could be a README in dist-git, a link from the dist-git web UI, or a stub server with redirection srcgit.fp.org/<package> -> HTTP redirect to src-git, or all of the above and more.

I do not think it's a good idea for "source-of-truth" Git to not be self-hosted. I've been burned by having sources + packaging being outside of the distribution infrastructure disappearing and getting screwed on that in the process when working with Debian/Ubuntu packages over the years and I definitely don't want that problem in Fedora.

And it was extremely clear from last year that Fedora does not want to be dependent on proprietary SaaS stuff that nobody can work on. What upstreams choose to do, is of course their business, but Fedora itself is supposed to be a stalwart for a Free platform that anyone can build from.

That is not what CPE wants to do though as far as I know. They're working on GitLab.com (proprietary SaaS): fedora-infrastructure#10028

That's true, but CPE is exploring GitLab because we asked them not to use GitHub, because GItHub is proprietary software. Well, gitlab.com is too. Exact same reasoning applies there. Here's a pretty decent summary of our previous community discussion and how we somehow failed to communicate the headline requirement to CPE. Fedora Council has already promised not to rely on gitlab.com for dist-git but had no similar promise for src-git. Council said they're planning to use self-hosted GitLab for dist-git. But if we can self-host dist-git, why not src-git too? Do we really have resources to host one but not the other? (This seems unlikely.) Are any proprietary features of gitlab.com truly essential for src-git (surely not).

Putting proprietary software into the product pipeline is a massive defeat for open source, and in this case it really doesn't make sense if the only goal here is to avoid having to host our own GitLab instance. If we don't want to make the effort of hosting a gitlab.fedoraproject.org, that's perfectly fine, because Pagure is already perfectly fine and more than sufficient for our src-git needs. GitLab is nicer than Pagure, not hugely so.

@catanzaro the assumption is that src-git repos would be hosted alongside dist-git repos in a different namespace to have consistent experience, although we're far from that: right now we just want to set up namespaces on pagure.io and src.stg.fp.o to start exploring the workflow further

@berrange that's a very good point regarding mirroring. I'd be so happy if we could host the repos in a single namespace in future and everyone would be fine with the pick. So far we have received a lot of (conflicting) requirements and preferences that it seems it's impossible to do "one size fits all". I'm sure once we have the repos up, the scope would be narrow down and the need for mirroring would vanish.

I'm sure once we have the repos up, the scope would be narrow down and the need for mirroring would vanish.

It is much harder to take away an existing offered service, than to add a new service at a later date, so don't speculatively offer something unless it is obviously essential.

IOW I'd suggest explicitly not trying to setup bi-directional mirroring of src-git repos between a pagure & gitlab instance initially. Let maintainers choose pagure or gitlab for their src-git as they desire, only worry about the src-git <->dist-git sync. After src-git is a thing that people use for a while - say 6 months - it'll become much more obvious if mirroring is truly needed or not, or whether Fedora decides to standardize on only 1 src-git hosting option, or something else again. We just need the core src-git functionality available for use to start with.

I'm sure once we have the repos up, the scope would be narrow down and the need for mirroring would vanish.

It is much harder to take away an existing offered service, than to add a new service at a later date, so don't speculatively offer something unless it is obviously essential.

IOW I'd suggest explicitly not trying to setup bi-directional mirroring of src-git repos between a pagure & gitlab instance initially. Let maintainers choose pagure or gitlab for their src-git as they desire, only worry about the src-git <->dist-git sync. After src-git is a thing that people use for a while - say 6 months - it'll become much more obvious if mirroring is truly needed or not, or whether Fedora decides to standardize on only 1 src-git hosting option, or something else again. We just need the core src-git functionality available for use to start with.

I can give you the point of view from our side: the feedback we receive from maintainers is to not disrupt their workflow, hence the pagure/gitlab discussion. Also everyone has their own way of doing spec files, patches, history, testing, etc. A different requirement (we get from management/leads) is consistency for the contributors: everyone should be able to find all the repos in a single gitforge/location and would not need to go to github for project A, gitlab for project B, pagure for...

Daniel, I completely agree with you that we should avoid the mirroring b/w two src-git repos, that was short-sighted from my side to state it like that above. If we can agree on a single, primary git forge - YES! If not, we either need to figure out a solution or tell people to use what we have or bust.

Daniel, I completely agree with you that we should avoid the mirroring b/w two src-git repos, that was short-sighted from my side to state it like that above. If we can agree on a single, primary git forge - YES! If not, we either need to figure out a solution or tell people to use what we have or bust.

I don't think mirroring is a problem, it just has to be a one way mirror. If the project is maintained on gitlab, there is no harm in an automated one way mirror to pagure. But I don't see much of an advantage to two way mirroring anyway. If a project is hosting on pagure and someone creates an MR in gitlab, I would expect it to be completely ignored, or hopefully responded to with a "create a pull request on pagure, that is where this project is managed".

the assumption is that src-git repos would be hosted alongside dist-git repos in a different namespace to have consistent experience, although we're far from that: right now we just want to set up namespaces on pagure.io and src.stg.fp.o to start exploring the workflow further

If this is true, then there's no point in proceeding any further with gitlab.com, because gitlab.com for dist-git will not be approved by Council, even if for src-git it might. You might want to check in with Fedora Council to see what they would actually be willing to approve before continuing to spend more time on this....

If this is true, then there's no point in proceeding any further with gitlab.com, because gitlab.com for dist-git will not be approved by Council

I am not sure how you got this from the blog you linked.

I have spoken to the CPE team management and they understand our position. To be clear, no decision has been made as to what version of GitLab will be used. But regardless of the general CPE decision, we are looking at the possibility of using GitLab CE for our dist-git specifically. For a general purpose git forge, we may use another version.

Gitlab CE can be self-hosted or managed. Either way, requires being able to work with Gitlab. We wouldn't need to support the EE features, but I don't think there is anything in the basic source-git structure would benefit from them anyway.

Some more updates on this topic from our side. CPE is working on FAS integration for gitlab.com/fedora community namespace. This opens door to integrate that GitLab namespace with the rest of the Fedora infrastructure. I'd love to know details of that work to be honest, if anyone has links, please let me know.

We have been debating lately where to start the source-git prototype and are leaning towards gitlab.com/fedora for the initial prototype. We already have the CLI part, @csomh is finishing documentation for it this sprint so anyone can try it in their shell. My comment is about an automation which handles merge/pull requests - triggering CI and sync with dist-git.

This doesn't mean we intend to abandon pagure - it's just to prioritize gitlab over pagure: basically have the first version done on gitlab and follow with pagure[.io] soon after. The main argument is that CentOS Stream 9 is developed on GitLab so this would be consistent.

Please let us know your thoughts and even join the next SIG meeting to discuss. It's scheduled for December 8th now.

I'll just continue to predict that it's a waste of time. It will not be accepted by the Fedora community because gitlab.com is not open source.

@catanzaro thank you for your time providing your response here

I'd appreciate a more concrete set of arguments instead of just stating "it's a waste of time". I went briefly through the FESCo and Council tickets linked from the blog you posted above and I can see that both FESCo and Council provided a response to the proposal about gitforge change with a list of requirements to meet.

My ask here was to get feedback on our proposal to start our prototype in gitlab (followed by pagure.io). I accept that this prioritization is not right for you.

Council has said very clearly that a non-OSS solution for dist-git will not be accepted. It might be accepted for source-git, sure, but you're going to have to fight the community over it.

My ask here was to get feedback on our proposal to start our prototype in gitlab (followed by pagure.io). I accept that this prioritization is not right for you.

I much prefer GitLab to Pagure. GitLab is great, and Pagure suffers from highly-visible weird bugs. (Try editing any comment in this ticket.)

You have to deal with two controversies here: (a) people who prefer Pagure to GitLab, and (b) people who insist Fedora infrastructure must be open source only. I'm in the (b) group, and I suspect it is a very big group. The open source version of GitLab is surely good enough for Fedora's src-git needs, but for unclear reasons you are not pursuing that route, you're trying to use the proprietary software version hosted on gitlab.com instead. You're going to have to debate group (a) no matter what. Seems like it would be much easier to give up on gitlab.com and use a gitlab.fedoraproject.org instead, that way you would only face opposition from group (a) and not group (b). By insisting on targeting gitlab.com, you're going to have to argue against group (b) as well, and I predict that's going to be very tough and likely to result in failure. Hence, I consider it a waste of time to pursue this further.

I don't really have anything more to add, because I have no further arguments beyond "the infrastructure should be open source." That's all I have to say. Ultimately it's just going to come down to votes. You need a majority on FESCo and "consensus" (which really means unanimity) on Council. The election cycle is starting now, so it's a perfect time for the community to express its preference.

See also my previous comment in this thread.

I am not sure how you got this from the blog you linked.

Missed this, sorry. From the Council blog post:

Many of you have expressed a strong desire for Fedora to be built with free software at all layers of the infrastructure. The Council shares that desire, but we recognize that there are cases where viable free software options are not available. In this particular case, we believe that we do have those options.

Emphasis original. Council says proprietary software option will be accepted only if no open source alternative is suitable, and also says that open source alternatives for dist-git are suitable. This clearly implies gitlab.com for dist-git will not be accepted.

But regardless of the general CPE decision, we are looking at the possibility of using GitLab CE for our dist-git specifically. For a general purpose git forge, we may use another version.

Council couldn't be any clearer. They are leaving open the possibility for you to use gitlab.com for src-git, but clearly not for dist-git. I'm not sure why dist-git is their line in the sand, but whatever.

Just to be clear, we use services that are being offered or are available within the ecosystem. This SIG is certainly not going to champion the gitforge change - that's too big of a task outside of our scope. I hope that our role will be to participate in the discussion.

I also agree that GitLab has superior usability and feature set compared to Pagure (and we hear this argument from our users). It's exciting that Pagure is a true open source project, but it cannot compete with a company such as GitLab or GitHub.

I think it will be a great exercise to have both gitforges available to use and compare the workflows. We should be then more comfortable stating what works and what doesn't.

As for "only open-source infrastructure", I cannot argue there - it's up to the infrastructure operators to decide what they want to operate :)

@catanzaro Michael, I appreciate your detailed response.

Well it seems I misunderstood the purpose of src-git. I thought this was going to be a canonical packaging location and Fedora packagers would have to interact with src-git to make changes to Fedora packages that use src-git. I brought this up on devel@ and it seems that's not the case: even if a package uses src-git, I can still commit directly to dist-git and make my changes to the packaging? In that case, this seems much less controversial to me.

@catanzaro Michael, thank you for bringing this up on the mailing list and letting us know! (even though I browsed fedora-devel yesterday I missed the thread /o\ )

Let's discuss over there, Hunor is writing a response that covers our PoV.

Hey @catanzaro,

We intend to make source-git a canonical, alternative way to do packaging in Fedora. But it's not even proposed. This is what this SIG is about: to define the workflow and help with developing the tooling for it, so that we have a mature enough solution that worth proposing to the community. For this we would need to create some source-git repositories for packages which would like to try the workflow and give us feedback if it works for them. But these repos wouldn't be "final" source-git repositories, they are just experiments. This issue is about deciding where to host these repositories.

The way we plan to implement source-git right now is to create it as additional layer on top of dist-git. Deciding to adopt the workflow for a package would mean that:
- Maintainers commit to make all their packaging work in source-git and let the tooling sync these changes to dist-git.
- Dist-git would be still the canonical source for building.
- Proven packagers who are making mass-changes to spec-files during rebuilds, could keep using dist-git for this, and automation would make sure that these changes are synced back source-git. But this would be the only, very limited case when contribution through dist-git would be supported. Syncing back (from dist-git to source-git) changes to source-archives or patch-files would not be supported.

But again, this is optional. Participating in the experiment: is optional. Even if we get to the point that it becomes an official way of doing packaging in Fedora, adopting it would be optional.