@bstinson , @carlgeorge : now that cbs.centos.org is updated to koji 1.23 and builders/hub on 8-stream, that's something that can eventually be done.

Now the interesting question: do we want to import on regular basis 9-stream as external repo (like for 7/8/8-stream) and allow SIGs to start testing building pkgs against/for 8-stream ?

We'd just need to ensure that -release tags (and even probably -testing for now) would be blocked completely, to not have any pkgs going out for now and being signed by SIG key (talking about SIG pkg, nothing from 9-stream would go out through cbs.centos.org but available through composes.stream.centos.org anyway)

Opinions ? thoughts ?

We should wait at least until we have signed Stream 9 RPMs before we enable an externalrepo in CBS. Updates on timelines will go out to the -devel list.

By the way, now that mock and Copr point at the Stream 9 composes, I've built most of the Ceph dependencies in https://copr.fedorainfracloud.org/coprs/ceph/el9/

By the way, now that mock and Copr point at the Stream 9 composes, I've built most of the Ceph dependencies in https://copr.fedorainfracloud.org/coprs/ceph/el9/

That's interesting. That's a lot of packages — more than I remembered needing to build. I didn't go through all five pages, but of the ones I did check, many of them are already built in the stream.centos.org build system, and in one of the centos9s-appstream, centos9s-crb, or centos9s-base repos.

I guess there's still a non-zero chance that some of those could be pruned out.

E.g. cryptsetup is still in centos9s-base; gtest was in centos9s-crb earlier, but now appears to be gone; libtool is still in centos9s-appstream; libuv is still in centos9s-appstream; lua is still in centos9s-appstream; nasm is still in centos9s-crb; and ncurses is still in centos9s-base.

Is there a reason you don't want to use the packages from those repos? Do you have knowledge that they are going to be removed from the repos?

And IIRC, there's a pretty good chance we will be able to use EPEL to provide any other dependencies that aren't in -crb or -appstream

Yes, I've documented each reason in a Trello board. There's a Trello column "Request from RHEL program" that lists the packages that are present in RHEL but missing certain sub-packages. We need to add those packages to the CRB (mostly -devel packages). For example, cryptsetup is in the compose, but cryptsetup-devel is not, which means it won't be possible to build Ceph unless we make cryptsetup-devel available in some other way.

I wrote a script that compares the Copr NVRs with the compose's NVRs, so that we stay lined up on these particular packages.

Yes, I've documented each reason in a Trello board. There's a Trello column "Request from RHEL program" that lists the packages that are present in RHEL but missing certain sub-packages. We need to add those packages to the CRB (mostly -devel packages).

Quite some time ago I asked about making all the missing -devel packages available in -crb. The answer at that time was that they would not be provided. I won't be holding my breath. ;-)

For example, cryptsetup is in the compose, but cryptsetup-devel is not, which means it won't be possible to build Ceph unless we make cryptsetup-devel available in some other way.

At the time of this writing, cryptsetup-devel is still available in the -crb repo. I'm not following the centos9s/rhel9 process closely, so I'm not aware of which things are scheduled to be removed from the compose.

There was a chance that external repos like EPEL and COPR would be allowed to be used for CBS SIG builds, but until that actually comes to pass I am presuming that all the missing dependencies will need to be built in CBS. But if the COPR repos don't have aarch64 and pp64le packages then they won't be much use for CBS SIG package building. It looks like you have aarch64 at least.

At the time of this writing, cryptsetup-devel is still available in the -crb repo. I'm not following the centos9s/rhel9 process closely, so I'm not aware of which things are scheduled to be removed from the compose.

Oh, good catch, I missed that cryptsetup-devel is now present. I removed this from Copr today.

Bisecting with compose-utils' compose-diff-rpms, I found cryptsetup-devel was added between CentOS-Stream-9-20210803.0 and CentOS-Stream-9-20210808.0. I guess it could be any of the recent comps changes.

I'm having trouble following what is happening in CentOS Stream 9 as well. It's changing a lot week to week.

I am presuming that all the missing dependencies will need to be built in CBS

Right. You can get the package list with copr-cli list-packages ceph/el9, or with the Python API (see that copy-stream script get_copr_packages() method)

I'm not sure about the current state of ppc64le with Copr for el9. There's some discussion on the fedora devel list.

It's now possible to build against/for 9-stream so the requested tags were created :

* Checking distribution el9s configuration...
 -> Checking storage config...
Using default options for storage/ceph
Creating tag  : storage9s-ceph-pacific-candidate
Creating tag  : storage9s-ceph-pacific-testing
Creating tag  : storage9s-ceph-pacific-release
 -> creating storage9s-ceph-pacific-el9s
Added external repo centos9s-baseos to tag storage9s-ceph-pacific-el9s-build (priority 5)
Added external repo centos9s-appstream to tag storage9s-ceph-pacific-el9s-build (priority 10)
Added external repo centos9s-crb to tag storage9s-ceph-pacific-el9s-build (priority 15)

* Checking distribution el9s configuration...
 -> Checking storage config...
Using default options for storage/gluster
Creating tag  : storage9s-gluster-9-candidate
Creating tag  : storage9s-gluster-9-testing
Creating tag  : storage9s-gluster-9-release
 -> creating storage9s-gluster-9-el9s
Added external repo centos9s-baseos to tag storage9s-gluster-9-el9s-build (priority 5)
Added external repo centos9s-appstream to tag storage9s-gluster-9-el9s-build (priority 10)
Added external repo centos9s-crb to tag storage9s-gluster-9-el9s-build (priority 15)

* Checking distribution el9s configuration...
 -> Checking storage config...
Using default options for storage/nfsganesha
Creating tag  : storage9s-nfsganesha-3-candidate
Creating tag  : storage9s-nfsganesha-3-testing
Creating tag  : storage9s-nfsganesha-3-release
 -> creating storage9s-nfsganesha-3-el9s
Added external repo centos9s-baseos to tag storage9s-nfsganesha-3-el9s-build (priority 5)
Added external repo centos9s-appstream to tag storage9s-nfsganesha-3-el9s-build (priority 10)
Added external repo centos9s-crb to tag storage9s-nfsganesha-3-el9s-build (priority 15)

Asking for feedback here as -candidate and -testing can be used for now but -release is locked due to mirror reorg and so waiting on CentOS Stream 9 itself to know where to sign and push (so nothing for now)

lua-devel is not found in Stream9 CRB in https://cbs.centos.org/koji/taskinfo?taskID=2530585

But it appears it should per https://composes.stream.centos.org/test/latest-CentOS-Stream/compose/CRB/x86_64/os/Packages/

In #centos-devel arrfab says:
<arrfab> kkeithley: worth looking if it's in the tree that cbs is using
<arrfab> kkeithley: it's not
<arrfab> kkeithley: so there is a diff between that compose and the one that would be authoritative
<arrfab> one way to solve this would be to add the c9s-buildroot specific repo, that (I just looked) has it
<arrfab> kkeithley: if you can comment in you ticket, I can add it and it should be able to find lua-devel-5.4.2-4.el9.x86_64.rpm
<arrfab> but don't know if it should land in CRB proper, as clearly there is actually a mismatch
<arrfab> so something eventually for carlwgeorge and/or bstinson

@kkeithle so while we can eventually open a BZ request to explain the mismatch with latest composes, I have proposed you to add the centos9s-buildroot (straight from stream 9 koji infra) as external repo and that one has the pkg you need :

Added external repo centos9s-buildroot to tag storage9s-ceph-pacific-el9s-build (priority 20)

So can you give it a try ? it should build.
PS : worth mentioning that in initial mail sent to centos-devel I mentioned that possibility to add other repositories in buildroot, like HA or centos9s-buildroot (

Thanks. I didn't parse your suggestion correctly to add the centos9s-buildroot.

I am aware that I can request the addition of repos like the c9s HA and EPEL repos to the buildroot.

And it (i.e. luarocks) does build now.

We need libtool-ltdl-devel from HighAvailability / ResilientStorage . https://bugzilla.redhat.com/show_bug.cgi?id=1999315 tracks moving that to CRB.

For that missing lua-devel issue, I've opened #440

@ktdreyer for which tags ? @kkeithle asked me to add c9s-buildroot to storage9s-ceph-pacific-el9s-build , which was done
For which tags do you need HA and RS ? as said in original mail, SIG should just ask for other repositories to be added if they need something else than baseos/appstream or crb
Happy to add them as soon as you tell us for which tags they should be enabled :)

If it's moving to CRB then we can wait for that to occur. Unless it's going to take a long time to move.

@arrfab

storage9s-nfsganesha-3-el9s-build needs to inherit storage9s-ceph-pacific-candidate and storage9s-gluster-9-candidate

E.g. see https://cbs.centos.org/koji/taginfo?tagID=1922

done