#9272 Replace regindexer with flatpak-indexer
Closed: Fixed 2 years ago by kevin. Opened 3 years ago by otaylor.

flatpak-indexer is a new codebase that not only creates an index of Flatpaks, it also generates static deltas between old versions and the new version. This greatly reduces bandwidth usage for users and for Fedora.

A major difference from regindexer is that it doesn't query the registry to find out available Flatpaks, but instead uses data from Bodhi and Koji. This is necessary because the registry APIs don't allow finding out

Rough task list:

  • Set up flatpak-indexer in Fedora openshift. An older version without Fedora support is deployed within Red Hat on openshift as the backend for flatpaks.redhat.io, so the configs for that can be used as a starting point.
  • Update cron-sync-registry-index to sync content from flatpak-indexer rather than regindexer, and to sync deltas/ in addition to index/ and icons/, update reversepassproxy.registry-generic.conf to add deltas/.
  • Use the CDN to frontend deltas/ and maybe icons/

I'm willing to be primarily responsible for this application for now with limitations from limited access to Fedora infrastructure.


@otaylor Can it wait until we are out of beta freeze?

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble, ops

3 years ago

@otaylor Can it wait until we are out of beta freeze?

Yes, that's fine. I wasn't filing this as an immediate request but rather as a work item to be tracked.

(Most of the necessary work could be done in staging openshift first, given availability - the bulk of the work is adapting existing openshift configs to the Fedora ansible setup.)

Any chance this could be looked at now?

Perhaps. :) I think probibly the only way right now is if we schedule time to work on it.

I am out friday and all of next week, but happy to meet this week and work on it. It would be good if we could rope in at least another sysadmin to follow along and so they could keep working on it next week?

Perhaps @smooge @mobrien or @mohanboddu would like to work on it too?

I am out friday and all of next week, but happy to meet this week and work on it. It would be good if we could rope in at least another sysadmin to follow along and so they could keep working on it next week?

Sounds good - I'm available 1-5 eastern today, and most of the day on wed and thu.

Kevin has had a power outage so looks like he is out for the day today.

Metadata Update from @kevin:
- Issue tagged with: mini-iniative

3 years ago

We mostly have staging up and running now...

Just one remaining issue:

The tardiff image delta files are being created with a mode of 0600 which only worked for previous installations elsewhere because the httpd pod was running with the same UID as the pods creating the differ. On the proxies httpd can't read the files. I'll file a PR to update flatpak-indexer to a fixed version in a bit. (Also will pull in changes to actually clean up old unused deltas and icons.)

Beyond that, things seem to be working well:

$ flatpak --user remote-add fedora-stage 'oci+https://registry.stg.fedoraproject.org'
$ flatpak remote-list fedora-stage
Name                       Application ID                            Version         Branch
dconf-editor               ca.desrt.dconf-editor                                     stable
DOSBox                     com.dosbox.DOSBox                                         stable
[...]
$ flatpak install fedora-stage astromenace
        ID                                  Branch           Op           Remote                Download
 1. [✓] com.viewizard.astromenace           stable           i            fedora-stage          < 63.1 MB
[...]
# commit from curl -A 'Accept: application/vnd.oci.image.index.v1+json' https://registry.fedoraproject.org/v2/astromenace/manifests/master-3320201020164654.1 
$ flatpak update --commit=2abf0c9927a6e114789b377310e6c9a42453c167578d1e1ebf84db84f01e4672  com.viewizard.astromenace
$ flatpak -v update com.viewizard.astromenace
[...]
F: Loading https://registry.stg.fedoraproject.org/deltas/d6/b22d2f0bc342f66c48264eb3143870b5b25d49444b0b5587a5e24f1123b8e0.tardiff using libsoup
F: Received 330155 bytes
[...]

So updating between the last two versions, the deltas successfully reduced a 63MB download to a 322KB download.

I think this is now fully rolled out.

Please reopen and let us know if there's anything more to do here.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Done