#9480 AWS CDN for koji "latest" buildroots?
Opened 10 days ago by praiskup. Modified 7 days ago

In Copr, people often enable
https://kojipkgs.fedoraproject.org/repos/fXX-build/latest/
in their builds, but it is pretty complicated - so only a limited percent of
users are actually doing this.

I'm curious if there's a desire/need to have the CDN proxy; I don't
propose a mirroring system..., but something similar we have with
download.copr.fedorainfracloud.org.

The question comes from the idea that we could make it a bit easier to
enable the Koji repo in Copr. Would Koji be fine to serve the additional
Copr load?


The system that you are describing will look something like this. (ascii graphics are a bit off but best I can do)

            [dnf/yum]<-+
               |
            [mirrors]<-+               [other clients]
                       |                   |
[copr]  <-> [aws-cdn] <-> [dl.fpo] <-> [NFS-server]
                                           |   
                                  +--------+-------+----------+
                                  |        |       |          |
                             [some builders]   [koji]    [other backend tools]

The pipe that dl.fpo (and other parts) has is a 2 gigabit pipe and we are mostly using it for dnf/yum from users and mirrors. The aws cdn shows up for some of the user traffic but we have been at 100% utilization with mirrors not able to get traffic. The NFS server is on 40GB but the koji shares are a large share with a lot of clients asking stuff from it. The CPU/cache on the server is pretty loaded with this.

The aws cdn could help on this but it would need to balance out what can be done.

So, we actually already do this for ostree... and could for this... but the problem is that the content changes very often, so I am not sure cloudfront is going to help too much.

On the other side, I don't think copr builders could cause that much load. Builds would be staggered I would think right, not generally hitting all at the same time?

So, personally, I would say it would be ok to just use directly and if there's some issue we could try and add cloudfront, but as smooge notes we have had BW problems, so it might be nice to get those solved before enabling ?

So my summary: wait a bit to make this easier until we have more BW, then enable it directly and unless we see a issue we are done, if we do, we look at cloudfront?

Metadata Update from @zlopez:
- Issue priority set to: Waiting on External (was: Needs Review)
- Issue tagged with: aws

7 days ago

Login to comment on this ticket.

Metadata