Many packit copr-builds and copr builds in general are failing for the centos-9-stream target, because there are errors fetching metadata for the baseos repo. The problem seems to have begun about 12 hours ago.
See the recent failed builds at: https://copr.fedorainfracloud.org/coprs/rhcontainerbot/packit-builds/builds/
it seems there was a new compose that was pushed out yesterday (centos infra doesn't control that but we can investigate, but ideally that would be reported at bugzilla.redhat.com
tried on a ec2 instance for stream 9 and it seems to work. Was there a problem at the mirrormanager (fedora infra hosted) side ? @adrian ?
@lsm5 , while waiting from @adrian's confirmation about mirrormanager being up2date, can you confirm that it seems working for you ?
@arrfab, looks like there's progress. Now I'm seeing a different error: https://download.copr.fedorainfracloud.org/results/rhcontainerbot/packit-builds/centos-stream-9-x86_64/05744865-container-selinux/builder-live.log.gz
yeah, I discussed that with people working on stream and there is a more important issue with gpg keys and wrong signatures on packages and repodata too.
in all cases, the correct tracker to use is bugzilla.redhat.com, and not centos-infra ;-)
will do. Thanks!
The checksum issue seems to be solved from what I see. I am not aware of any problems.
MirrorManager picked up the changes on the primary mirror at Wed Apr 5 06:27:28 UTC 2023 and the systems serving this information are reload at 20 after the hour. So since 7:20 UTC the new checksums should be available.
I can see that we skipped scanning the primary because content was being updated:
CentOS primary mirror sync in progress. Skipping scan at Tue Apr 4 07:42:01 UTC 2023 CentOS primary mirror sync in progress. Skipping scan at Tue Apr 4 07:57:01 UTC 2023 CentOS primary mirror sync in progress. Skipping scan at Tue Apr 4 08:12:01 UTC 2023 CentOS primary mirror sync in progress. Skipping scan at Tue Apr 4 08:27:01 UTC 2023 CentOS primary mirror sync in progress. Skipping scan at Tue Apr 4 08:42:01 UTC 2023 CentOS primary mirror sync in progress. Skipping scan at Tue Apr 4 08:57:01 UTC 2023
Between Tue Apr 4 08:57:01 UTC 2023 and Wed Apr 5 06:27:28 UTC 2023 I see no attempt to scan the primary mirror. I think the problem in the logic is that the scanning relies on the COMPOSE_ID file which is created during content creation.
Tue Apr 4 08:57:01 UTC 2023
Wed Apr 5 06:27:28 UTC 2023
The current COMPOSE_ID (for non SIG content) has Mon, 03 Apr 2023 11:02:24 GMT by the time it appears on the primary mirror it is between 24 and 48 hours old and the script sees that we scanned the primary mirror before that time stamp. So the primary mirror scanner falls back to the scan at least every 24 hours and the fast scanner checking (every 20 minutes) does not help.
Mon, 03 Apr 2023 11:02:24 GMT
We are re-using the logic from Fedora where we rely on the date of the file like fullfiletimelist-fedora which is generated after the primary mirror has been updated.
fullfiletimelist-fedora
I think re-using COMPOSE_ID for primary mirror changes detection is not the right solution. We would need a file that is updated once the primary mirror has been updated.
there is another issue with gpg and it's being checked by stream team
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue tagged with: centos-build-pipeline, centos-stream
We investigated the issue and it's not infra related but rather a rebase of gnupg2 that deprecates now SHA1, while the centosofficial key is still signed with that algo More info on https://bugzilla.redhat.com/show_bug.cgi?id=2184640
Closing at the infra side
Metadata Update from @arrfab: - Issue close_status updated to: Duplicate - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.