I had a quick look and it's happening at the haproxy (in front of the real kojihub.stream infra, not itself publicly reachable)
503 means that haproxy considered there was no backend available to server the request, and I confirmed this by looking at the haproxy log :

Apr  8 12:15:18 lb2 haproxy[832579]: 8.43.84.3:50289 [08/Apr/2024:12:15:18.782] https-in~ tls_kojihub/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 265/265/247/0/0 0/0 "GET /kojifiles/repos/c9s-build/latest/x86_64/toplink/packages/bash/5.1.8/9.el9/x86_64/bash-5.1.8-9.el9.x86_64.rpm HTTP/1.1"

When then that works, there is a backend defined and available and so it works :100:

Apr  8 13:31:31 lb2 haproxy[832579]: 8.43.84.3:44884 [08/Apr/2024:13:31:31.397] https-in~ tls_kojihub/kojihub-1 0/0/2/9/102 200 1739143 - - ---- 181/181/128/128/0 0/0 "GET /kojifiles/repos/c9s-build/latest/x86_64/toplink/packages/bash/5.1.8/9.el9/x86_64/bash-5.1.8-9.el9.x86_64.rpm HTTP/1.1

I see that koji was recently updated for Stream infra and causing a huge load on the postgresql DB host (investigating with koji maintainers why) but that can explain the issue you're seeing from time to time.
As a recap :
- koji was updated to 1.34 (stream, not cbs related)
- haproxy has a check to validate that backend is available and removes/adds it back when fully working
- due to DB load, and so check, haproxy considers it's not available all the time

I'll see how to fix that but it's clearly visible since the koji 1.34 upgrade on Stream infra side (checking in Zabbix the load to confirm this)

@tkopecek : do you have pointers ?

@amoralej : I think I was able to stabilize (for now but internal discussion ongoing about this) from external point of view.
I was monitoring current state after I applied a workaround earlier today and it seems back to usable state, from cbs/external repository PoV that is.

Closing so this ticket as from CBS infra PoV it should be ok now

Thanks Fabian!