no, cbs.centos.org is not designed to allow building from gitlab, and builders can only reach git.centos.org (per initial design, as all SIGs are supposed to build from git.centos.org which is where SIG should host sources)

We'll want to look into some adjustments to allow CBS to reach gitlab.com, this will take a bit of investigation though.

@lrossetti this is likely something we should look into :)

Commenting here as this interesting for the Kmods and possible other SIGs as well. This would SIGs to use MR/PR workflow which is currently not possible on https://pagure.io/centos-infra/issue/228 and probably the main reason why almost all SIGs store their primary repositories somewhere else and only sync to git.centos.org before building.

However it is obviously not as easy as simply allowing to point to git+https://gitlab.com/<namespace>/<sig>/rpms/<package>.git#<hash> when building in CBS.
Note: I'm calling the main group <namespace> here to better distinguish it from subgroup <sig>.

Some points that I think need to be discussed first:

<namespace>: The initial request was about namespace redhat. Afaik you need to be a RedHat employee to have a repository there. Hence this is a no go for CentOS SIGs, the official CentOS namespace centos should be the only one allowed instead. The centos namespace is currently still empty. Which is probably even good as we can agree on a structure first.

<sig>: It would be nice to have a subgroup <sig> in group centos and allow every SIG to manage its content within this group/namespace as they wish. Obviously there should be a suggestion to follow the <namespace>/<sig>/{rpms,src} structure. Probably even enforced by only allowing to build from <namespace>/<sig>/rpms/<package>.git in CBS.
However I'm not sure how well this works with the lookaside cache (git.centos.org/sources/<package>/c<release>-sig-<sig>*) currently used for builds. This lookaside cache uses the branch name (enforced to c<release>-sig-<sig>*) in its folder structure and for access control. Simplest solution might be to enforce using this same branch naming convention for repositories in <namespace>/<sig>/rpms/<package> and only allow building from these in CBS. Other branches, e.g. source and devel branches, could still have any arbitrary name.

dist-git layout: git.centos.org uses the exploded SRPM (I think that's what it is called) layout. There has been some talk about allowing the new standard dist-git (used by c9s, fedora, etc.) layout in git.centos.org. Probably allowing to build from gitlab.com/<namespace>/<sig>/rpms/<package> is a good opportunity to introduce this new layout to SIGs. I.e. support only the new standard dist-git layout on gitlab.com and keep the old layout on git.centos.org. Especially as it has been mentioned on the mailing list discussion that downstream sources dropped from RHEL to git.centos.org will not be changing layouts.

CC: @ngompa @dcavalca

Most of the restrictions around PRs in git.centos.org come from our old version of Pagure. Upgrading Pagure is tracked in #456 and is blocked on pagure#5219.

As for the Dist-Git layout, that's a restriction of the tools CBS uses. I believe that CPE will be fixing that soon, since @pingou mentioned it on the mailing list.

As for the Dist-Git layout, that's a restriction of the tools CBS uses. I believe that CPE will be fixing that soon, since @pingou mentioned it on the mailing list.

There is even a PR pending review for this: https://git.centos.org/centos-git-common/pull-request/9 :)

<namespace>: The initial request was about namespace redhat. Afaik you need to be a RedHat employee to have a repository there. Hence this is a no go for CentOS SIGs, the official CentOS namespace centos should be the only one allowed instead. The centos namespace is currently still empty. Which is probably even good as we can agree on a structure first.

This is very much the situation I woud like to end at: every SIG has their own namespace under the centos namespace on gitlab.com. The structure within that namespace can then be decided upon by the SIG but my suggestion would be something like:

sig-name
├── project_foo
├── groups
│   ├── group0
│   ├── group1
│   └── sig
├── rpms
│   ├── rpm0
│   └── rpm1
└── src
    ├── project0
    └── project1

but this structure makes sense if the sig is interested in leveraging something like packit, otherwise, the need for thesrc namespace is a lot less obvious.
At the end it will very much depend on the size of the SIG, how many project/package they have, how many people they have, how much they need to track ACLs...

However I'm not sure how well this works with the lookaside cache (git.centos.org/sources/<package>/c<release>-sig-<sig>) currently used for builds. This lookaside cache uses the branch name (enforced to c<release>-sig-<sig>) in its folder structure and for access control. Simplest solution might be to enforce using this same branch naming convention for repositories in <namespace>/<sig>/rpms/<package> and only allow building from these in CBS. Other branches, e.g. source and devel branches, could still have any arbitrary name.

That is a good point, I suspect the short fix will be to indeed use the same branch structure as on git.centos.org. Long term I don't know, should we move to a different structure there? I don't even know if it's possible but happy to investigate it if there is interest.

We spent a fair amount of time coming up with this structure, with stakeholders from kernel development, CentOS Stream broadly, and people involved in the new automotive effort. We did this under the impression that it would be workable and a relatively simple configuration change for the CBS builders.

Can someone enumerate what, if anything, technically and organizationally prevents CBS from being configured in this way to build using the proposed structure? It's not clear to me, reading the above, how much is reconfiguration, how much is documented policy, how much is assumed policy and how much is preference.

We spent a fair amount of time coming up with this structure, with stakeholders from kernel development, CentOS Stream broadly, and people involved in the new automotive effort. We did this under the impression that it would be workable and a relatively simple configuration change for the CBS builders.

Who was involved? Because as best as I can tell, there was no public discussion anywhere about this. When Hyperscale was being brought up, we regularly discussed with CentOS project and infra folks our needs at office hours, meetings, etc. We got a pretty good grip of the capabilities and limitations and communicate pretty often our needs.

This is the first time I'm seeing anything from Automotive on this front.

As far as I know, Automotive has not done anything of the sort. So where did these discussions happen?

Can someone enumerate what, if anything, technically and organizationally prevents CBS from being configured in this way to build using the proposed structure? It's not clear to me, reading the above, how much is reconfiguration, how much is document policy, how much is assumed policy and how much is preference.

It's all policy and code. Hyperscale wanted to use Fedora-style Dist-Git structures with git.centos.org and it wasn't supported, so we do it the "classic" way like the rest of CentOS.

There are no limitations in CBS that prevent us from using this structure. The project layout that pingou mentioned is a good one, and worth a careful look especially for SIGs coordinating with a number of contributors, but it is not something that we will be blanket requiring SIGs to adopt.

SIGs, by existing policy, will continue organize their own sources and projects how they choose.

To recap the work here, we will be doing the following things:
- Allowing CBS builders to contact gitlab.com, and adding certain namespaces to the koji allowlist. As soon as gitlab.com/centos is ready for content, SIG namespaces under there will be part of that allowlist
- Updating the CBS tools to understand the flat dist-git layout

This work is an addition to the existing git.centos.org + exploded SRPM layout, which will continue to work as long as folks still use it.

Hyperscale wanted to use Fedora-style Dist-Git structures with git.centos.org and it wasn't supported, so we do it the "classic" way like the rest of CentOS.

That is orthogonal to using gitlab though, the flat or exploded layouts can be used regardless of where the git repos are :)

The point I was making was that we talked publicly about what we wanted and needed instead of deciding something and doing it before anyone knew what we wanted and needed.

Neal, could you please describe where "talked publicly" happened and how to
include the "we", the "us" and the "anyone" you mentioned?

I just got off a call with @pingou where we synced up on this and cleared up some misunderstandings. We're gravy now. :v:

That said, what I'm referring to about talking publicly is:

• Going to the CentOS Stream Office Hours to talk about our concerns
• Giving talks about stuff us, stuff we've done, and stuff we're doing
• Talking on Matrix or IRC to folks
• Regular status reports (either to the blog or to the mailing list)

From my perspective, I don't know what you're even doing. @pingou cleared things up for me just a bit ago, but I wish it had not come to that.

Based on internal discussion/meeting, I'll start working on what's needed to allow building from gitlab.
At first sight, I see the following "tasks" :

• modify cbs/koji to allow gitlab.com/redhat in the allowed SCMs (easy and in git/ansible)
• Allow koji builders to reach gitlab (difficult as they are behind cloudflare so have to find someone who can help with that to directly point to gitlab infra and not cloudflare)

That's just for the koji/cbs part, but from what I see in other comments in this ticket, there is also work to be done to use the flat layout and new lookaside :

• flat-layout (already implemented and working from git.centos.org)
• different lookaside layout : currently not implemented and automotive SIG is testing currently on git.stg.centos.org to upload to different lookaside layout.
• when previous step is working, it needs to be deployed to git.centos.org
• get_sources.sh should also be adapted and centpkg-minimal package rebuilt in cbs to be able to then retrieve sources from the new directory structure from "new" lookaside cache layout

I'll update status when possible

• modify cbs/koji to allow gitlab.com/redhat in the allowed SCMs (easy and in git/ansible)

We likely don't want the entire gitlab.com/redhat and we'll want to restrict it
a little bit more than this, but as you say this is easy to do and adjust :)

• modify cbs/koji to allow gitlab.com/redhat in the allowed SCMs (easy and in git/ansible)

We likely don't want the entire gitlab.com/redhat and we'll want to restrict it
a little bit more than this, but as you say this is easy to do and adjust :)

Plus the structure you described in one of your previous comments 1 mentioned the namespace centos. Now we are back at namespace redhat again. Which, afaik, can only be used by Red Hat employees. Are both namespaces going to be allowed and redhat is simply the first one to be allowed to be used in CBS? Or am I missing something?

In case this issue here is simply just about allowing automotive SIG to build from redhat namespace and it is required to open an extra issue to get the proposed structure in 1 working for other SIGs, please let me know. The Kmods SIG is very much interested in being able to use the structure proposed in 1.

Plus the structure you described in one of your previous comments [1] mentioned the namespace centos. Now we are back at namespace redhat again. Which, afaik, can only be used by Red Hat employees. Are both namespaces going to be allowed and redhat is simply the first one to be allowed to be used in CBS? Or am I missing something?

The end goal is the 'centos' namespace but we're still working on getting the
authentication working, so until we get it done we want to enable a subsection
of the 'redhat' namespace which has the drawbacks you've mentioned :(

Down the line, I definitely want to move the automotive namespace under 'centos'
and then we can disable 'redhat' in CBS and only allow from the 'centos'
namespace :)
Migrating CBS from 'redhat' to 'centos' will be easy as it will just be a config
change in ansible, the heavy lifting that is needed on the network side will
have been done.

update status : just giving it a try in a small dev env (one kojid only to one kojihub) and it seems that git doesn't obey the inject env variable in mock to use http_proxy and https_proxy (but mock uses it for the dnf transactions, just git not using it). Can't find any doc about it but can still search.
If we can't find a solution, that will be problematic (still waiting for other info about how it was implemented for Stream 9 so once I'll have info we can see which method to test and use eventually)

We should continue the network enablement for the announced netblocks for gitlab and cloudflare mentioned here: https://docs.gitlab.com/ee/user/gitlab_com/#ip-range

GCP blocks are not necessary.

As far as the Allowed SCMs, we can handle requests for other namespaces in gitlab.com but will be starting with a select few for testing purposes. Ultimately, we'll want to allow gitlab.com/redhat and gitlab.com/centos

The network/config change was pushed in git and applied by ansible on the cbs kojid builders.
I launched a scratch build from automotive SIG kernel on gitlab and it was able to retrieve it sources and buildSRPMFromSCM task worked :

https://cbs.centos.org/koji/taskinfo?taskID=2729637

Worth knowing that used gitlab repo was still using the previous rpm style layout (but flat layout is now known to be working already) but also the previous lookaside cache layout/structure

So the next steps (probably for @bstinson) will be to :

• verify that new lookaside can work on git.stg (and validate with Red Hat RCM)
• have newer get_sources.sh working for new lookaside structure and then available through a newer centpkg-minimal (to be also rebuilt on cbs and tagged in the various buildsys tags but also in the centos9s-buildroot)

PS : the additional lookaside cache work is though parallel so technically, if one SIG wants to use gitlab with documented lookaside cache structure, it can work already. It's only if they want to also use a newer layout that it should be just available in get_sources.sh and also pushed to git.centos.org for then newer upload_sig.cgi

PS2: I'd still like to find another way to restrict kojid builders as actually that means that (previous isolated) builders can reach whatever is hosted behind cloudflare (and not only gitlab)

WRT PS2 : a config change was pushed earlier today on cbs.centos.org to ensure all traffic from builders is going through a proxy (where the needed ACL is in place)

That means at this stage, building from gitlab is still working (it was last week too) but we're still waiting on newer centpkg-minimal, assuming also that git.centos.org will get the newer upload_sig.cgi in place (it seems it's not merged from staging branch)

weekly update status : still waiting on newer centpkg-minimal to be available (built through kojihub.stream.centos.org) so that we can have it in the centos9s-buildroot repo (used on cbs.centos.org . That bumped centpkg-minimal is needed to get newer get_sources.sh that would be able to retrieve sources from modified lookaside structure

Per discussion with @bstinson we agreed to just bump and rebuild first on cbs.centos.org, to unblock SIGs and centpkg-minimal can then be rebuilt on https://kojihub.stream.centos.org

The following centpkg-minimal version is available in cbs for 8s/9s buildroot :100:
https://cbs.centos.org/koji/buildinfo?buildID=38884
https://cbs.centos.org/koji/buildinfo?buildID=38881

A test build was submitted to use newer get_sources.sh and was able to retrieve sources to build SRPM :

https://cbs.centos.org/koji/taskinfo?taskID=2760649

Just waiting for final ack from @pingou and we can then close this ticket

I've tried a scratch build on CBS for this commit: https://git.centos.org/rpms/abseil-cpp/tree/bcfa1179b7364c60e9a6d2097130fcf089c17bcd
which is:

• flat layout
• sources file
• sha512 hash
• we can see in https://git.centos.org/sources/abseil-cpp/ that there are no c9s-sig-automotive-main branch there

So :thumbsup: for me, thanks!

Thanks for the feedback. Closing ! :-)