#8579 Update regindexer to handle metadata-in-labels
Closed: Fixed 2 months ago by cverna. Opened 3 months ago by otaylor.

We're now building Flatpaks for Fedora with the metadata stored in both docker-compatible labels (for flatpak >= 1.6), and in annotations (for flatpak < 1.6). We want to set things up to create two sets of indexes - one for older flatpak clients and one for newer flatpak clients/

(This is a blocker for pushing flatpak-1.6 to Fedora. Cc: @kalev @amigadave)

Steps:

  • Update sundries to have regindexer-0.6
  • Update the regindexer config to build both sets of labels (roles/regindexer/build/templates/config.yaml)
  • Update the http config to redirect to the two indexes depending on the query (roles/httpd/reverseproxy/templates/reversepassproxy.registry-generic.conf)

I'll send a patch to the infrastructure list with updates forthe config files.


Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

3 months ago

Metadata Update from @cverna:
- Issue assigned to cverna

2 months ago

Hi @otaylor I am trying to apply your patch from https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/7Z7J6JIJQL45OADBOSQW2YOXAHINMMI3/ but I am having some trouble

Applying: regindexer: create two sets of indexes
.git/rebase-apply/patch:112: trailing whitespace.
daemon: 
error: patch failed: roles/httpd/reverseproxy/templates/reversepassproxy.registry-generic.conf:25
error: roles/httpd/reverseproxy/templates/reversepassproxy.registry-generic.conf: patch does not apply
error: patch failed: roles/regindexer/build/templates/config.yaml:7
error: roles/regindexer/build/templates/config.yaml: patch does not apply
Patch failed at 0001 regindexer: create two sets of indexes
hint: Use 'git am --show-current-patch' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

It might just be my git foo and email patches that let me down but could you check on you side, and maybe send the patch using git format-patch command.

Thanks.

Hmm, can't get the patch to apply for me either from the email. No idea why - my git-send-email foo is decidely weak.

0001-regindexer-create-two-sets-of-indexes.patch

It seems that the new regindexer is failing to start with the following error.

Feb 03 20:18:01 sundries01.phx2.fedoraproject.org systemd[1]: Started Daemon for creating an index of a container registry.
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: Traceback (most recent call last):
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: File "/usr/bin/regindexer-daemon", line 5, in <module>
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: from pkg_resources import load_entry_point
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3007, in <module>
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: working_set.require(__requires__)
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 728, in require
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: needed = self.resolve(parse_requirements(requirements))
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: raise DistributionNotFound(req)
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org regindexer-daemon[18710]: pkg_resources.DistributionNotFound: www-authenticate
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org systemd[1]: regindexer.service: main process exited, code=exited, status=1/FAILURE
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org systemd[1]: Unit regindexer.service entered failed state.
Feb 03 20:18:01 sundries01.phx2.fedoraproject.org systemd[1]: regindexer.service failed.

I have used the following command to update the sundries boxes

yum update --enablerepo=epel-testing regindexer

and the version of regindexer is regindexer-0.6-1.el7.noarch

Bah, I tried to clean up dependencies, and added something that wasn't used by regindexer at all, just by a script in my container-based testing framework.

https://koji.fedoraproject.org/koji/buildinfo?buildID=1456133
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-ff1afae47d

(or installing python2-www-authenticate works around.)

I have installed python2-www-authenticate for now and the service is running.

Closing this ticket since everything should be in place. Can you double check and reopen if needed.

Thanks

Metadata Update from @cverna:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 months ago

Metadata Update from @otaylor:
- Issue status updated to: Open (was: Closed)

2 months ago

Testing the resulting index revealed a bug in the new regindexer functionality - https://registry.fedoraproject.org/index/flatpak-testing-labels.json has every image with a org.flatpak.ref annotation, not just the ones that were rebuilt to have an org.flatpak.ref label.

regindexer-0.6.2 fixes:

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-8b55643030
https://koji.fedoraproject.org/koji/buildinfo?buildID=1456422

could we update sundries to that. Thanks!

Done, I have just manually installed the rpm for now, but once the update reach stable this version will be automatically installed by ansible.

Metadata Update from @cverna:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 months ago

Login to comment on this ticket.

Metadata