cc @otaylor @kalev

My guess would be regindexer. I don't know to debug it, sadly :(

https://registry.fedoraproject.org/index/static?label%3Aorg.flatpak.ref%3Aexists=1&architecture=amd64&os=linux&tag=latest is the index: I can't spot anything obviously wrong it (beyond missing the latest inkscape build, https://bodhi.fedoraproject.org/updates/FEDORA-FLATPAK-2021-ea03586ac8, but that should be unrelated here).

Maybe it is something else then, not the index :) I can poke at it a bit more in the evening and see what I can figure out.

So, the problem here is that the flatpak-indexer PR got merged, which did three things:

• Slightly modified the proxy httpd config to expect indexes as flatpak-indexer writes them
• Deleted regindexer cron jobs, etc.
• Added the openshift roles for flatpak-indexer

Then it seems that the playbooks got return to rebuild the proxy httpd config, and that's what's breaking things here. I switched:

[request index with annotations] => flatpak-<blah>.json
[request index with labels] => flatpak-<blah>-labels.json

[request index with annotations] => flatpak-<blah>-annotations.json
[request index with labels] => flatpak-<blah>.json

So current clients that request an index with labels get the index with annotations instead, and ignore all the images in their because they don't have the appropriate labels.

We really just wanted to merge the first part until we get flatpak-indexer up and running and proven on stage. Let me submit a PR to revert the stuff we're not ready for, and then the playbook will need to be rerun to regenerate the httpd configuration on the proxies.

Oops. I thought that was only affected staging. ;( sorry about that.

Thanks for tracking it down... I should have looked at that

https://pagure.io/fedora-infra/ansible/pull-request/399

Commit 67224d3b fixes this issue

merged and running. (It takes a while).

Note that we can still land these changes, we just need to make sure they are with when: env == 'staging' or '{% if env == staging ... %}' blocks.

Then we could test things better in stg... but we can do that down the road.

merged and running. (It takes a while).

Note that we can still land these changes, we just need to make sure they are with when: env == 'staging' or '{% if env == staging ... %}' blocks.

Then we could test things better in stg... but we can do that down the road.

Ack. Sorry that I didn't pick up that we were merging changes that would affect production ... I wrote the patch as if it were "final" for ease of understanding what was going on and review, but forgot that there was a need to strip it down before merging.

Thanks, Owen and Kevin!

Confirming that it's working OK now.

Yeah, back to all working. Many thanks for the quick revert, and all my fault for not looking before pushing. ;(