fedora-infrastructure

#10415 flatpak indexer seems to have fallen over
Closed: Fixed by kevin. Opened by kalev.

Closed: Fixed
Reopen Issue

Looks like flatpak indexer has fallen over again. None of the flatpaks that are currently in testing in Bodhi are showing up as updates on flatpak CLI.

@otaylor Could you take a look, please?

Metadata Update from @zlopez:
- Issue priority set to: Waiting on External (was: Needs Review)

Oh, I just got an idea what happened -- can it be related to F33 going EOL and we probably just need to update the indexer configuration to reflect that?

What basically happened is that the code to reconnect to fedora-messaging didn't work properly, so that we stopped seeing updates going from newly submitted to testing/stable, but I don't really understand why.

2021-11-17 22:33:13,248:ERROR:pika.adapters.blocking_connection:Unexpected connection close detected: ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2021-11-17 22:33:13,249:WARNING:flatpak_indexer.datasource.fedora.bodhi_change_monitor:fedora-messaging connection failure (ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"), sleeping for 1s and retrying
[...]
2021-11-17 22:33:14,276:INFO:flatpak_indexer.datasource.fedora.bodhi_change_monitor:Connected to fedora-messaging, queue=9b874dd9-085a-4570-b7d3-b1e7264fee24

But we never got any messages after the reconnection. I forced a restart (kill -INT 1) and on restart, the queue didn't exist any more:

2022-01-06 15:20:45,136:WARNING:pika.channel:Received remote Channel.Close (404): "NOT_FOUND - no queue '9b874dd9-085a-4570-b7d3-b1e7264fee24' in vhost '/public_pubsub'" on <Channel number=1 OPEN conn=<SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncSSLTransport object at 0x7f60409bfa00> params=<ConnectionParameters host=rabbitmq.fedoraproject.org port=5671 virtual_host=/public_pubsub ssl=True>>>
2022-01-06 15:20:45,147:INFO:flatpak_indexer.datasource.fedora.bodhi_change_monitor:Connected to fedora-messaging, queue=b7023f7b-cac3-41a2-8874-3ada9c67add3

I do see one hole in the flatpak-indexer logic - it looks like if the broker "cancels" the consumer then we don't try to reconnect (and get a new queue) - the notification thread just quits. Possibly that's what happened here, though it's odd that we successfully reconnected to our old queue but never got any new notifications on it.

I'll try modifying the logic to handle that case, and just maybe it will help the next time this happens :-)

Filed:

https://github.com/owtaylor/flatpak-indexer/issues/5

I think this issue can be closed, since the indexer is working OK again for now.

(It's processing a bunch of queued deltas, but will finish within the next hour or so and I expect it to write out a new index then.)

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Highest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.