#11634 Flathub reverse proxy for infra (openQA)
Opened 5 months ago by adamwill. Modified 2 months ago

For background, see https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/305 , specifically https://pagure.io/fedora-qa/os-autoinst-distri-fedora/pull-request/305#comment-195089 .

We are adding a Flatpak test to openQA. One thing the test does is check that installing flatpaks from Flathub works OK - which involves downloading several hundred megabytes from Flathub, because each time the test runs, it's on a fresh system which doesn't have any base flatpaks yet.

We have the all-clear from Flathub folks to run this for compose testing (i.e. a couple of times per day on average), but we'd ideally like to run it on updates too, which would involve running it a lot more than a couple of times a day. The flathub folks suggested that setting up a reverse proxy for flathub (as someone wrote up here) would be a good idea in that case, so I'm filing this ticket to see if we can have an infra-level reverse proxy. (Possibly this is even already covered by something existing? I'm not sure.)

There's no firm date on this, it's more of a would-be-nice-to-have.


This would likely require a rework of some httpd roles but should be feasible. Could I be assigned to this? I'd like to take a crack at it

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-trouble, medium-gain, ops

5 months ago

There's nothing like this currently. Our fedora flatpaks have aws cloudfront in front of them, but all internal stuff hits our registry directly.

I would think we would want it to only be allowed by internal openqa... we don't want to provide everyone a free flathub cache.

I don't think we would want this on our proxies at all, instead seperate. Perhaps we could look at something in openshift?

@leo feel free to look into this and propose something, but I think we probibly need to discuss this further before deciding too much about it.

I agree we wouldn't want to be a public cache, but it might be useful for more than just openQA internally? possibly other infra systems might want to hit up flathub regularly, and it might be more efficient to have that cached within infra?

I don't know of anything currently, but sure...

Hey all,
After some discussion on Matrix I have done some research and testing to setup a flathub proxy with varnish. The plan is to put varnish on openshift somehow. Either way, here's the varnish config:

vcl 4.1;

backend default {
    .host = "dl.flathub.org";
    .port = "80";
}

sub vcl_recv {
    set req.http.Host = "dl.flathub.org";
}

Either way, I'd like to help in this in any way possible. Let me know your thoughts!

P.S. This image seems useful: https://github.com/sclorg/varnish-container

Hi @adamwill,
After some testing and discussion in IRC we have concluded that varnish is not feasible. On your side for openqa, would it be possible to modify the flatpak remote to a new domain from openshift (the proxy)?
Thanks!

Ah, that's a shame. We could look at that, though we generally try to have openQA tests reflect 'normal' usage as much as possible - we want to test that the way users will actually deploy flatpaks (flathub and the Fedora flatpak remote) work...

@adamwill
Hey! Turns out we can use squid and this lets you keep the remote the same. Instead, you'd set the http_proxy environment variable and it'll hit the cache. Is this a better option?

that'd be better than modifying the remote, I think, yeah.

Metadata Update from @leo:
- Issue assigned to leo

4 months ago

@leo Did you have time to work on this? How it looks?

@zlopez Hi, yes, it’s still in development. There were a lot more hurdles than expected. It should be ready soon if everything goes well from here.

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog