fedora-infrastructure

#7900 compose-tracker consumer not receiving pungi compose messages
Closed: Fixed by dustymabe. Opened by mohanboddu.

Closed: Fixed
Reopen Issue

compose-tracker consumer is not receiving any messages on fedora messaging. @jcline helped us to debug:

[10:36:35] <jcline> dustymabe, the bridges seem fine and the service that checks AMQP and zeromq for missing messages hasn't seen any differences
[10:37:08] <jcline> It could be the pungi->gateway connection? noc is full of nagios messages about no messages

From the logs on the consumer side:

2019-06-14 11:15:29,363 INFO fedora_messaging.cli - Starting consumer with compose_tracker:Consumer callback
2019-06-14 11:15:29,364 INFO compose_tracker - Using detected token to talk to pagure.
2019-06-14 11:15:29,374 INFO fedora_messaging._session - Authenticating with server using x509 (certfile: /etc/fedora-messaging/fedora-cert.pem, keyfile: /etc/fedora-messaging/fedora-key.pem)
2019-06-14 11:15:29,390 INFO twisted - Starting factory FedoraMessagingFactoryV2(parameters=<URLParameters host=rabbitmq.fedoraproject.org port=5671 virtual_host=/public_pubsub ssl=True>, confirms=True)
2019-06-14 11:15:30,552 INFO fedora_messaging.twisted.protocol - Successfully registered AMQP consumer Consumer(queue=8000364d-2dea-489f-88ab-9f39e9d453ab, callback=<compose_tracker.Consumer object at 0x7f0ee1088d68>)

cc @dustymabe

Can we get someone to look at this today or tomorrow.? It's hard to discuss compose issues without the opened issues against the compose-tracker.

Getting composes working again was higher on my list. :)

Anyhow, can you tell us where the compose-tracker is ? Still in your home lab? or did we move it internal somewhere?

I see compose messages in #fedora-fedmsg and in #fedora-releng just fine, and datagrepper seems to see them. So perhaps it's something in the outgoing fedmsg->fedora-messaging queue?

@jcline can we tell if the public queue is getting copies of the messages as expected?

Setting up an AMQP consumer and then triggering a publish and seeing if it got it (just using the printer callback) would do it. The bridges looked okay and everything on the AMQP side looked correct. I'd start looking at the fedmsg side of things.
Edited by jcline

Getting composes working again was higher on my list. :)
Anyhow, can you tell us where the compose-tracker is ? Still in your home lab? or did we move it internal somewhere?

The one that opens issues against https://pagure.io/dusty/failed-composes is running in my personal openshift instance. We'd like to get it running in the prod fedora. There is an open RFR waiting on security review.

It is also currently running in stg and opening issues against the stage failed composes repo. You can see there that the issues stopped getting opened on the 12th of june.

hey @kevin - can we get together with @jcline sometime today (I can send out a meeting invite to block off calendars) to look into this deeper?

as you may note, today is/was not good for me. ;(

I might be able to do friday afternoon? Not sure what @jcline 's availability is.

Friday afternoon your time would be pretty late Friday my time. I'm not sure what value my presence work offer, though. If you see those messages getting to the gateway, it is some issue with the bridge into AMQP, otherwise it's somewhere in the tangle of fedmsg connections and I don't know anything about that specific setup.

some new information:

The instance running in stage is now seeing messages come through. You can see the new issues it has been creating: https://stg.pagure.io/releng/failed-composes/issues .

Any reason why my personal openshift (running in DigitalOcean) would be different than the fedora stage openshift for this issue?

There shouldn't be. This all might have been due to the expired CRL that caused all fedmsgs to be dropped for a day or so.

OK. I'm still seeing the outage with my personal openshift instance. Can others confirm that they can see messages come through on the org.fedoraproject.prod.pungi.compose.status.change topic from outside fedora infra?

FWIW my VPS (also running in digital ocean) stopped receiving these messages on Jun 11, but I suspect it was some problem on my side.

FWIW my VPS (also running in digital ocean) stopped receiving these messages on Jun 11,

good to know. Looks like the last automatically filed ticket against dusty/failed-composes was on the 12th at 07:45 UTC.

but I suspect it was some problem on my side.

Why do you suspect that?

Why do you suspect that?

Because if the app I'm running there loses connection to the bus, it does not reconnect automatically and needs manual restart. This happens during maintenance etc, and I have never seen a case where it would be a problem with something else. So it's possible it was some temporary networking issue and I haven't noticed since then.

Why do you suspect that?

Because if the app I'm running there loses connection to the bus, it does not reconnect automatically and needs manual restart.

Can you restart your app and see if you start getting messages again?

@dustymabe, I did that, and will let you know when I see a message. So far there has been none.

Because if the app I'm running there loses connection to the bus, it does not reconnect automatically and needs manual restart. This happens during maintenance etc, and I have never seen a case where it would be a problem with something else. So it's possible it was some temporary networking issue and I haven't noticed since then.

Is this with fedora-messaging? If so, is it with 1.7.0? If so, that's a bug, it should restart the connection automatically.

Is this with fedora-messaging? If so, is it with 1.7.0? If so, that's a bug, it should restart the connection automatically.

Nope, it's not even written in Python.

Ah, okay

I'm receiving messages correctly.

@lsedlar - but you aren't using fedora messaging at all, right?

My public instance continues to not receive messages, while the fedora stage instance continues to receive them. Does that piece of information help in the investigation?

Nope, I'm not using fedora-messaging. I'm just confirming that a VM in Digital Ocean can receive the messages.

If you're interested, the source code for that unfinished thing is at https://github.com/lubomir/talk-compose.

this seems to be working now - I believe there was a period of down time where the messages weren't being delivered at all but recently the stage compose box started working again (that's when the messages started to be delivered again). My other box running this compose-tracker wasn't hooked up to autorebuild on commits to the git repo so it was running an old commit without the change to handle the new format of the message. I never saw the stacktrace from the error because by the time I looked at the logs again a new container had been started in its place and the logs from the old container were gone, so it looked like everything was working but just not getting messages.

All is well now. The tracker just created it's first issue in a while: https://pagure.io/dusty/failed-composes/issue/2003

Metadata Update from @dustymabe:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.