During the rsync operation from mirrors.c.o to OpenInfra of Mar 28 at 6:00 UTC, something went wrong which led to have a partial repo on OpenInfra side. Metadata was ok but some packages were not available, and the consumers hit HTTP/404 when trying to pull the packages. The issue got resolved after the next rsync iteration at 12:00 UTC.
The OpenInfra rsync logs live in https://mirror.dfw.rax.opendev.org/logs/rsync-mirrors/centos-stream.log (the file containing the valuable logs might be archived depending on the log rotation configuration)
When we check the incremental file list of the 2024-03-28T06:07:02, we can see lines like the one below:
2024-03-28T06:07:02
deleting AppStream/x86_64/os/Packages/perl-Scalar-List-Utils-1.56-462.el9.x86_64.rpm
perl-Scalar-List-Utils-1.56-462.el9.x86_64.rpm was one of the package missing during the download in some CI job.
perl-Scalar-List-Utils-1.56-462.el9.x86_64.rpm
However, in the 2024-03-28T12:07:01 run, we don't have those deletions. There is deletions but only for the temporary paths (i.e .~tmp~) like
2024-03-28T12:07:01
.~tmp~
deleting AppStream/aarch64/os/Packages/.~tmp~/systemd-journal-remote-252-32.el9.aarch64.rpm
Why the rsync client on OpenInfra side did see those files as deleted on CentOS Infra side ? Maybe there is more information on CentOS infra logs which would help us know what happened. Could you check out please ?
We don't have logs about what is pushed by the CentOS Stream team so it would be hard at our side to see what happened. Is that from same machine or one in the pool that was probably not in sync ? (at our side that is). The .~tmp~/ one seems to indicate that one machine in the pool wasn't in sync with upstream reference mirror (it happens from time to time due to multiple stream composes being pushed in parallel , each one is insanely high , and has duplicates too, like .iso images but that's another debate)
.~tmp~/
Closing this one for now but feel free to open a new one if that happens again and then we'll create a Jira ticket to discuss this with the Stream team (not following what's happening on this tracker)
Metadata Update from @arrfab: - Issue close_status updated to: Insufficient Data - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.