So the CentOS8 stream servers are in one location and the CentOS9 stream servers are in another one. They are on different koji servers and the setup of each koji in CentOS is different from Fedora. You are probably going to need to detail more of your needs to make this possible.

One VM near each koji-like RPM repository would be needed. (If there is a mirror that contains reasonably timely union of both, that'd be fine too.) There's no problem federating multiple servers, one per stream, with one front-end.

Do you need NFS or just for the system to be near it and it picks it up via osmosis or HTTP :laughing:

Read-only NFS to a copy of the RPM content would be best. It does not have to be the standard koji-style directory hierarchy. The Fedora VMs have 500GB of fast local storage for holding the index + 24GB RAM + 4CPU or so, which should be ample here too.

discussed during "infra and releng" stand-up and per Stream policy (see https://wiki.centos.org/ReportBugs) it should be requested through Bugzilla first and then it can become a task in Jira (stream team)

@arrfab, can you be more specific about which bugzilla? bz.rh.com centos8/centos9 appear to contain only software problem reports, not infrastructure RFEs. Perhaps you can point me to an example to clone?

We were told internally to redirect all Stream related requests (including Infra like this) to Bugzilla and someone from the Stream team can then redirect to Jira if that's accepted or need to be discussed internally

@amoloney as you're the Stream PO, can you guide @fche for this ? (including eventually the Jira project for dedicated for Stream)

Please let me take liberty to reopen this, until someone can point me to the precise bug tracker where and how this issue should be transferred.

https://wiki.centos.org/ReportBugs says this is the right tracker, everything related to CentOS Infra, CBS, SIGs, mirror, CI, etc seems to have to reported here. Is there another tracker (or bugzilla) for this issue?

@fche Any Stream related tickets should be raised on bugzilla https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%209&version=CentOS%20Stream

@mobrien Thanks, what component is correct for CentOS infrastructure issues?

Also note that you may want to update https://wiki.centos.org/ReportBugs which says that everything related to CentOS Infra needs to be reported here. See also https://lists.centos.org/pipermail/centos-devel/2020-August/055980.html

@mobrien, @arrfab, @amoloney, thanks, but I just cannot find even a single bugzilla centos-stream component in bz that would cover this infrastructure. There is no software bug that requires a release. Please help us out and create whatever bug tracker object you believe would adequately track this request for centos operational infrastructure enhancement.

CentOS Stream infrastructure issues should be reported in JIRA: https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12326233

Hopefully, this direct link will work. If not, you need to select the “CS” project manually.

CentOS Stream infrastructure issues should be reported in JIRA: https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12326233

I don't think that is right for community infra issues. To use that Jira you need create an Red Hat account and accept an Enterprise Agreement.

OK, let's try https://issues.redhat.com/browse/CS-966 .

Any update on this?
It doesn't seem there is any activity on the other tracker either.

@fche we got news from Stream PO that it can be worked on (but we just need to scope this between other tasks) so we'd like to know about the needs.
It seems the best approach would be an EC2 instance, so not having NFS read access to koji instances (as they are isolated and different infra, and also one moving soon).
So if you have a EBS volume big that can rsync content, would that work for you ?
You'd be allowed to rsync content that one can find on debuginfo.centos.org (for 8-stream) and mirror.stream.centos.org (for 9-stream)

@fche we got news from Stream PO that it can be worked on (but we just need to scope this between other tasks) so we'd like to know about the needs.

Thanks!

It seems the best approach would be an EC2 instance, so not having NFS read access to koji instances (as they are isolated and different infra, and also one moving soon).
So if you have a EBS volume big that can rsync content, would that work for you ?
You'd be allowed to rsync content that one can find on debuginfo.centos.org (for 8-stream) and mirror.stream.centos.org (for 9-stream)

That could work, if it's affordable. When I mentioned NFS read to koji, I didn't mean to be that specific, so as to exclude another local copy of the .rpms. Perhaps there is another spot in the existing or post-move infrastructure where a moderate VM can access RPMs where they are already located?

Using a VM near the builder can eventually be a thing later, but after they'll all be migrated and consolidated (8-stream build infra is in one place and 9-stream a different one and both will have to move)

So let's just start (easier for us and faster for you) with a decent EC2 instance, from which you'll be authorized to pull content. Do you have an estimate about the instance type ? We can have OS on local EBS and second EBS volume attached to it, that will used to host the debuginfo pkgs ?

We'd still like to have our ansible baseline role applied, even if you'll be delegated rights on it, and so if possible also reuse our existing httpd/tls settings (to be compliant within the centos.org infra and with RH security guidelines, as we'll be PoC for this). That means we can have the cert automatically renewed (central process) and pushed automatically without a need to do that at your side. Happy to have a chat to discuss needs, instead of doing this through pagure ticket btw ;-)

OK, sounds good.

For EC2 instance, something like m4.2xlarge (32G RAM) would be about right. An EBS with all the centos -debuginfo and -debugsource (!) RPMS could be big, I'm afraid I don't really know how big it all is right now. For comparison, RH-internal server that covers all RHEL8 + RHEL9 is about 20TB today. The generated index is about 2% of that size, and should be on fast SSD, perhaps on the same volume as the OS. The bulk RPM content itself need not be on fast storage (cold hdd sc1).

Happy to have a chat to discuss needs, instead of doing this through pagure ticket btw ;-)

Thanks, can certainly do that too sometime!

Well, we don't have such fast storage for the production instance so don't put expectations too high :)

We'll start with a small EBS volume that can eventually be extended (easier than having to shrink)

Do you have an estimate of the total current size of centos[89]* -debuginfo -debugsource RPMs? For cold storage of the RPMs for that and near future, I'd suggest about 20% extra space beyond that total. For the index EBS volume, we'd need about 2% of the former. Its latency is important but localish HDD is fine if SSD is not affordable, just not NFS (1-2 orders of magnitude slower).

between other tasks I'll start deploying the EC2 instance and additional linked EBS volume.
I'll keep you informed but we'll probably have to discuss about the app that you'll deploy.
We'll use CentOS Stream 8 as OS for that node

node is deployed with two EBS volume:

• OS / 200 GB (io1/ssd)
• second disk (not mounted yet) : 2TiB (gp2)

Waiting (pinged internally) about some other questions

hey @fche ... had a little bit of time (between other running tasks) and got initial role for ansible (staging branch for now, so "manual" trigger) : https://github.com/CentOS/ansible-role-debuginfod/tree/staging

That means that https://debuginfod.centos.org is deployed and under monitoring but it now needs :

• debuginfo pkgs for 8-stream and 9-stream (let's discuss that)
• your code so that api calls to debuginfod is returning something

Thanks to @arrfab's labours, the server is up and running, thanks a lot!

NB: https://bugzilla.redhat.com/2052574 would let centos users automatically take advantage of this service.