We'd like to be able to rsync from buildlogs.centos.org to make it easier to consume CI-produced packages (stuff under https://buildlogs.centos.org/centos/8-stream/hyperscale/x86_64/). Right now this seems to be only exposed over HTTP -- would it be possible to enable rsync as well? Thanks!
Well, it was clearly stated when we open buildlogs that it was just for testing and so reason why -testing were pointing to https directly. As it's just a temporary place holder, and storage constraint so just some nodes behind, can you elaborate on why you'd like to have rsync support to just download unsigned pkg ? it's just a testing and ephemeral place before it's officially tagged (and so signed) and pushed to mirror CDN.
Also, SIGs are encouraged to do their testing through CI env, which ḧas internal buildlogs.centos.org mirror so not even leaving the vlan/DC :-)
Metadata Update from @arrfab: - Issue tagged with: need-more-info
Metadata Update from @arrfab: - Issue priority set to: Waiting on Reporter (was: Needs Review)
Sure, let's use systemd as an example to make things easier. Currently we have a CI job on openshift that builds a daily systemd package from the latest upstream git master. As part of the build, it runs the systemd test suite, which gives us reasonable confidence the package isn't completely broken. In addition, we plan to setup some VM-based tests to validate things like "can we actually boot a box with this new systemd".
These daily builds are, well, daily. We don't want to tag them -release, as it seems wasteful to distribute them to all the mirrors. OTOH, we do need them to end up in a place where they can be consumed, so we're tagging them -testing to get them on buildlogs.
-release
-testing
On the FB side, there's a bunch of additional tests we'd like to run against these daily builds. Some of these are tests that we can (and will) add to the openshift CI, but others aren't really doable anywhere else. Think stuff like "try to spin up our internal container infra against the daily systemd" or "put the daily systemd on a few production boxes and see what happens". The idea is then to run these sort of tests internally, and then publish the results somewhere public (as, e.g., we've found in the past that our internal container thing is pretty good at spotting genuine issues upstream).
So, we need a way to reliably pull in these daily builds. As we already have tooling to do this kind of thing via rsync, that seemed the easiest way to go, hence this ticket. Other options we're considered:
I spent some more time on this today: it looks like cbs download-build won't work for build tags, but it does work for destination tags, e.g. cbs download-build --latestfrom=hyperscale8s-packages-main-testing systemd. So we could keep tagging -testing, and pull artifacts down via cbs without having to hit buildlogs. I'm going to experiment with this some more and see if it's actually feasible, though if it isn't too much work to get rsync enabled on buildlogs I still think that would be useful regardless.
cbs download-build
cbs download-build --latestfrom=hyperscale8s-packages-main-testing systemd
I can have a look at the rsync RFE as it's clearly easy to add, but just had a quick look and the two nodes behind buildlogs.centos.org already sent out ~42TiB of data for the last 30 days and we already redirect for all rpms to a CDN77 sponsored cdn. So we'd like to keep it like that and not have these nodes suddenly allowing people to download more (directly) from buildlogs.centos.org (sponsored nodes), and risking sponsor to just cut access
So I'd still prefer people willing to keep local repository up2date using something like reposync, that would fetch from CDN77
Does it make sense ?
Yup I totally understand. I've just validated that the cbs download-build pipeline I rigged up will work for our usecase here, so we won't need to go through buildlogs.centos.org.
Metadata Update from @arrfab: - Issue close_status updated to: Invalid - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.