I'm facing a weird behavior with two different build tags:
kmods9s-packages-main-el9s
kmods9s-packages-rebuild-el9s
Apart from the external repos 'epel' and 'epel-next' both seem to be configured identically. I can not spot any differences. diff <(cbs taginfo kmods9s-packages-main-el9s-build) <(cbs taginfo kmods9s-packages-rebuild-el9s-build) seems to agree.
diff <(cbs taginfo kmods9s-packages-main-el9s-build) <(cbs taginfo kmods9s-packages-rebuild-el9s-build)
However currently all builds for *-rebuild-* fail , e.g. see kmod-aacraid, for ppc64le due to:
DEBUG util.py:444: No matching package to install: 'kernel-abi-stablelists = 5.14.0-51.el9' DEBUG util.py:444: No matching package to install: 'kernel-devel = 5.14.0-51.el9' DEBUG util.py:444: No matching package to install: 'kernel-devel-uname-r = 5.14.0-51.el9.ppc64le'
All these packages are available in the centos9s-buildroot external repo at the moment. This can be confirmed by checking the mirror directly.
My first assumption was that some sort of sync has to be done each time an external repo is updated. Hence it might be fixed after some time. However this error now persists for almost 24h and the weirdest part:
Builds in *-main-* which require these allegedly missing packages do work! See e.g. kmod-btrfs.
What's the difference between these two build tags *-rebuild-* and *-main-*? What's so special about the ppc64le centos9s-buildroot repository? For aarch64 and x86_64 it works as expected. Only ppc64le is causing issues.
Metadata Update from @zlopez: - Issue tagged with: low-gain, medium-trouble
Reading some tickets that were opened when I was away/off and wondering if you still have that particular issue ?
Worth knowing that when you relies on a external repository, the kojira process/service on kojihub is watching for external repositories changes and if it detects a change, it will trigger some regen-repo tasks on cbs, so one has to wait for these tasks to be complete before kicking a build that relies on specific pkg from that external repo.
kojira
regen-repo
Do you see that issue ? Worth verifying when it happens and so if there is also a regen-repo task just before or after your build as that would explains it. A pkg available in external repo doesn't mean that your build target can use it as cbs/koji has first to run the createrepo/mergerepos task (that you can see in each `newrepo' task on https://cbs.centos.org
Waiting for feedback and so if we need to investigate or if that's all clear for you now
Metadata Update from @arrfab: - Issue priority set to: Waiting on Reporter (was: Needs Review) - Issue tagged with: cbs
This issue still exists.
Here are tasks trying to build the same NVR each ~5 hours: https://cbs.centos.org/koji/taskinfo?taskID=2694205 https://cbs.centos.org/koji/taskinfo?taskID=2694305 https://cbs.centos.org/koji/taskinfo?taskID=2694383 https://cbs.centos.org/koji/taskinfo?taskID=2694496 https://cbs.centos.org/koji/taskinfo?taskID=2694658 https://cbs.centos.org/koji/taskinfo?taskID=2694828
And here it finally succeded: https://cbs.centos.org/koji/taskinfo?taskID=2695544
I expected that there is some process watching for external repositories changes. However I expected this to be done for all architectures at the same time. I was really suprised that the repository has been updated for x86_64 and aarch64, but the new packages are still not available for ppc64le for up to one day. This large time offset is especially bad as the external repository always only contains the latest version of a package, see #574. The package of interest, kernel, is often updated every 24 hours (on work days).
I also do not understand why this depends on the build tag. The above example is from build tag kmods9s-packages-rebuild-el9s. Here an example from build tag kmods9s-package-main-el9s: https://cbs.centos.org/koji/taskinfo?taskID=2694270 https://cbs.centos.org/koji/taskinfo?taskID=2694164
kmods9s-package-main-el9s
The timestamps are similar and it depends on the same missing package (kernel-5.14.0-62.el9) as the package from the other build tag.
At the first try again the ppc64le repository has not been updated yet, but it is already fixed in the next try ~5 hours later. But for the other tag it takes up to one day. Shouldn't the state of an external repository be the same for all build tags?
Metadata Update from @zlopez: - Issue priority set to: Needs Review (was: Waiting on Reporter)
[backlog refinement] This still needs to be investigated.
Metadata Update from @zlopez: - Issue assigned to arrfab
We briefly had a look at this ticket and in fact, all the intermediate steps to import, regen repodata and then have kojira doing the koji regen-repo task (in serial and automatic so no real control over that) would add confusion about when/how to trigger a build (apart from verifying first if specific package is there that is)
As we recently had to also had a proxy support for koji builders to reach gitlab, we were thinking about instead just switching url for centos9s-buildroot repo to origin, which should be always up2date (real source from where we were initially reposync'ing from) and kojira should then be ok with it.
centos9s-buildroot
Opinion ? if that works we can just take some action to then allow that and test ?
Metadata Update from @arrfab: - Issue priority set to: Waiting on External (was: Needs Review)
I'm fine with switching the url for centos9s-buildroot. If I understand it correctly the only difference for someone building in CBS should be that no scheduled sync happens in between, i.e., packages are available in CBS once they are in origin (and after running regen-repo task) and not delayed by a daily sync.
Would this by any chance also fix #574?
not really as what is in centos9-buildroot is managed by stream team so if it doesn't have older packages, you'd only have access to $latest (but at least we bypass the whole sync process so what is available in upstream centos9s-buildroot, straight from kojihub.stream.centos.org, will be available, nothing more, nothing less)
Ok, I thought it was worth asking. But it seems the change should at least fix this issue.
Thanks!
@pjgeorg : cbs is now pointing to real centos9s-buildroot (exposed from kojihub.stream.centos.org) and kojira detected the change so multiple regen-repo in progress. Can you give it a try ? just worth knowing that if that really happens "in sync" on upstream koji, that should fix the issue you had as it wouldn't be imported asynchronously on internal cbs. I had to allow the cbs builders to use existing internal proxy to reach external kojihub.stream setup .
I just did a scratch build which requires a package from centos9s-buildroot. This worked without any issues. However this obviously does not verify whether this issue is indeed fixed. Verifying an issue which only occurs at certain points in time is hard to verify.
In addition the kmods SIG automation process, which initially triggered this bug, is currently blocked by #574 for cs9.
Closing the issue for now as we expect that the issue has been fixed (at least on CBS side). I'll re-open in case the issue appears again. Thanks!
Metadata Update from @pjgeorg: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.