I recently built systemd-256.7-1.9 RPM builds at 12-05 ~2:30 UTC: https://cbs.centos.org/koji/builds
It is now 4:45 UTC (2+ hours later) and the RPMs are still not uploaded to the mirrors: https://mirror.stream.centos.org/SIGs/9-stream/hyperscale/x86_64/packages-facebook/Packages/s/
I tailed the MQTT broker and it seems the signing queue is quite slow, taking hours to sign a single build: https://sigs.centos.org/guide/delivery/#following-signingpush-process-status
I also noticed this with building systemd-256.7-1.8 but it took ~2 hours as those RPMs were the only ones in the queue. Now there are 10+ RPMs in the queue.
Can someone help look at this? Or give me pointers to how I can investigate? Thanks!
Looks like it completed around 3 hours (~5:30 UTC)
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue tagged with: cbs, centos-build-pipeline, investigation
I don't think there is anything to investigate, and by looking at signing logs, it seems other hyperscale pkgs were tagged before the one you mentioned :
[+] 20241205-02:16 stylo -> Processing koji_tag [hyperscale10s-packages-main-testing] for SIG [hyperscale] and release [10-stream] using GPG key [eb3dac40]
That's because systemd-257~rc3-20241205020455.hs.el10 was built and tagged to -testing. As there are multiple sub-packages (and multiple arches) the way it works is that each produced rpm is downloaded , gpg-signed and imported back into koji and that step finished at :
[+] 20241205-02:41 stylo -> Importing signed [systemd-257~rc3-20241205020455.hs.el10.src.rpm] back into koji ...
and then finished distRepo task and push to mirrors at :
[+] 20241205-02:44 stylo -> Finished processing [hyperscale10s-packages-main-testing]
Problem is that you also had other build with same amount of pkgs in the queue and so (same steps) :
[+] 20241205-02:45 stylo -> Verifying koji tag [hyperscale10s-packages-facebook-testing] .... [+] 20241205-03:16 stylo -> Finished processing [hyperscale10s-packages-facebook-testing]
And then again and again and again , always for your tags and systemd but multiple tags, builds ...
Your specific build was then treated at :
[+] 20241205-04:57 stylo -> Processing koji_tag [hyperscale9s-packages-facebook-release] for SIG [hyperscale] and release [9-stream] using GPG key [eb3dac40] ... [+] 20241205-05:29 stylo -> Incrementing COMPOSE_ID for 9-stream [+] 20241205-05:29 stylo -> Signing repomd.xml with gpg key [eb3dac40]
and systemd-256.7-1.9.hs+fb.el9 was then pushed to external mirrors Then same build but different disttag was also processed : systemd-256.7-1.9.hs.el9
systemd-256.7-1.9.hs+fb.el9
So if you think that one of your build, should be built and not even tested before being tagged to production (some kind of emergency ? ), maybe you can discuss within your own SIG to just not queue so many different builds for systemd, for multiple tags at same time and orchestrate that in a way that would do what you want ?
Tagging @dcavalca , @ngompa so that you can discuss that during your SIG meeting as in this specific example hyperscalebot submitted multiple builds, tagged to -testingand so , per your own request, are also signed and keeping the queue busy ...
hyperscalebot
-testing
@arrfab ah to clarify, the hyperscale bot builds are daily builds built for testing and the ryantimwilson builds are the real production build where I care about timing. So it looks like just bad timing where both were submitted at once.
hyperscale
ryantimwilson
As for no testing, yes its a bit of an emergency but I did build and test the RPM locally using mock.
Thanks for the explanation @arrfab, we can close this out and discuss within Hyperscale SIG
Metadata Update from @ryantimwilson: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.