#1553 RPM signing and mirror upload not done after 2+ hours - systemd-256.7-1.9
Closed: Fixed with Explanation 2 months ago by ryantimwilson. Opened 2 months ago by ryantimwilson.

I recently built systemd-256.7-1.9 RPM builds at 12-05 ~2:30 UTC: https://cbs.centos.org/koji/builds

It is now 4:45 UTC (2+ hours later) and the RPMs are still not uploaded to the mirrors: https://mirror.stream.centos.org/SIGs/9-stream/hyperscale/x86_64/packages-facebook/Packages/s/

I tailed the MQTT broker and it seems the signing queue is quite slow, taking hours to sign a single build: https://sigs.centos.org/guide/delivery/#following-signingpush-process-status

I also noticed this with building systemd-256.7-1.8 but it took ~2 hours as those RPMs were the only ones in the queue. Now there are 10+ RPMs in the queue.

Can someone help look at this? Or give me pointers to how I can investigate? Thanks!


Looks like it completed around 3 hours (~5:30 UTC)

Metadata Update from @arrfab:
- Issue assigned to arrfab

2 months ago

Metadata Update from @arrfab:
- Issue tagged with: cbs, centos-build-pipeline, investigation

2 months ago

I don't think there is anything to investigate, and by looking at signing logs, it seems other hyperscale pkgs were tagged before the one you mentioned :

[+] 20241205-02:16 stylo -> Processing koji_tag [hyperscale10s-packages-main-testing] for SIG [hyperscale] and release [10-stream] using GPG key [eb3dac40]

That's because systemd-257~rc3-20241205020455.hs.el10 was built and tagged to -testing.
As there are multiple sub-packages (and multiple arches) the way it works is that each produced rpm is downloaded , gpg-signed and imported back into koji and that step finished at :

[+] 20241205-02:41 stylo -> Importing signed [systemd-257~rc3-20241205020455.hs.el10.src.rpm] back into koji ...

and then finished distRepo task and push to mirrors at :

[+] 20241205-02:44 stylo -> Finished processing [hyperscale10s-packages-main-testing]

Problem is that you also had other build with same amount of pkgs in the queue and so (same steps) :

[+] 20241205-02:45 stylo -> Verifying koji tag [hyperscale10s-packages-facebook-testing]
....
[+] 20241205-03:16 stylo -> Finished processing [hyperscale10s-packages-facebook-testing]

And then again and again and again , always for your tags and systemd but multiple tags, builds ...

Your specific build was then treated at :

[+] 20241205-04:57 stylo -> Processing koji_tag [hyperscale9s-packages-facebook-release] for SIG [hyperscale] and release [9-stream] using GPG key [eb3dac40]
...
[+] 20241205-05:29 stylo -> Incrementing COMPOSE_ID for 9-stream
[+] 20241205-05:29 stylo -> Signing repomd.xml with gpg key [eb3dac40]

and systemd-256.7-1.9.hs+fb.el9 was then pushed to external mirrors
Then same build but different disttag was also processed : systemd-256.7-1.9.hs.el9

So if you think that one of your build, should be built and not even tested before being tagged to production (some kind of emergency ? ), maybe you can discuss within your own SIG to just not queue so many different builds for systemd, for multiple tags at same time and orchestrate that in a way that would do what you want ?

Tagging @dcavalca , @ngompa so that you can discuss that during your SIG meeting as in this specific example hyperscalebot submitted multiple builds, tagged to -testingand so , per your own request, are also signed and keeping the queue busy ...

@arrfab ah to clarify, the hyperscale bot builds are daily builds built for testing and the ryantimwilson builds are the real production build where I care about timing. So it looks like just bad timing where both were submitted at once.

As for no testing, yes its a bit of an emergency but I did build and test the RPM locally using mock.

Thanks for the explanation @arrfab, we can close this out and discuss within Hyperscale SIG

Metadata Update from @ryantimwilson:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 months ago

Log in to comment on this ticket.

Metadata