Since a few days there seem to be some issues how CBS gets updated repositories. E.g., see 1, which is failing for a few days now due to missing packages, although these missing packages have been added to mirror.stream.centos.org few days ago. Looking into details, it seems the koji internal repositories are not being updated anymore for some tags.
E.g., comparing kmods9s-packages-main-el9s-build to kmods9s-packages-rebuild-el9s-build the former shows 5.14.0-444.el9 as latest version for the kernel package, the latter 5.14.0-446.el9.
kmods9s-packages-main-el9s-build
kmods9s-packages-rebuild-el9s-build
5.14.0-444.el9
5.14.0-446.el9
Note: The available kernel version was obtained by running dnf --nogpgcheck --repofrompath cbs,https://cbs.centos.org/kojifiles/repos/kmods9s-packages-main-el9s-build/latest/x86_64/ --repo=cbs list kernel
dnf --nogpgcheck --repofrompath cbs,https://cbs.centos.org/kojifiles/repos/kmods9s-packages-main-el9s-build/latest/x86_64/ --repo=cbs list kernel
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue tagged with: cbs, centos-common-infra, high-gain, medium-trouble
I had a quick look and while the internal mirror itself is up2date (we have monitoring in place for that one too), it's more at the kojihub/kojira level that it seems it was confused, but don't know why. example :
May 16 07:19:52 kojira[483643]: 2024-05-16 07:19:52,510 [WARNING] {483643} koji.repo.regen:734 Tag name/id mismatch: directory: storage8-common-el8-build, name: virt8-ovirt-common-el8-build, id: 1932
So somehow it was confused about which repo to watch for which tags, and that for multiple tags. I just relaunched kojira itself and it processed all repositories in normal order, including kicking some "regen-repo" tasks Waiting for feedback but also keeping an eye on failing tasks if there is some underlying issues
hmm, some tasks are failing in a strange way : https://cbs.centos.org/koji/taskinfo?taskID=3960176
HTTP response code said error: The requested URL returned error: 503 Service Unavailable Warning: Downloading of repodata failed: https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/x86_64/
But I do see that it was generated so wondering if kojira detects some changes but at same time we have upstream stream kojihub also replacing files
that should have been the issue as kicked again and it worked https://cbs.centos.org/koji/taskinfo?taskID=3960234 .. Let me reprocess all *el9s-build tags and verify that it would contain up2date pkgs
All *-el9s-build tags which include epel seem to be affected. I remember reading about some issues due to changed compression used in Fedora infra, might this be related? I can not find the original post anymore, but here's one 1 related issue.
ouch : createrepo_c has an issue parsing epel9 repodata :
Critical: Cannot load repo: "https://dl.fedoraproject.org/pub/epel/next/9/Everything/x86_64/" : Error encountered while parsing:primary.xml parsing: Cannot open /tmp/koji/tasks/476/3960476/createrepo_c_tmp_repo_qLYHPA/repodata/4a517e26b96f3a03d135e5e58e93cafd21f912b5c72261520d6b53783f172859-primary.xml.zst: Cannot detect compression type
(from https://cbs.centos.org/kojifiles/work/tasks/476/3960476/mergerepos.log)
created ticket in Fedora infra to see what's possible
Upstream ticket closed (epel9-next now using correct compression method) and so all regen-repo tasks seem to have worked too : https://cbs.centos.org/koji/taskinfo?taskID=3961309
Metadata Update from @arrfab: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.