I noticed today that there aren't any recent openQA results in staging ResultsDB. Investigating a bit more, it seems like the fedmsg consumer that should wake up every time an openQA 'test complete' message appears and send the result on to ResultsDB just isn't firing at all - it's not broken, it's just not doing anything. Even though there definitely are openQA tests running in staging, and the 'job complete' messages are at least showing up in datagrepper.
I poked about a bit for a while and I'm coming to the conclusion that things listening to the staging hub just aren't getting any messages at all. If I do this, on either openqa-stg01.qa or on my own desktop:
fedmsg-tail-3 --include stg
which should include any message with 'stg' in the topic, I get...nothing. Nothing at all. Even if I do something I know creates a message, like restarting an openQA job on staging. If I do:
fedmsg-tail-3 --include prod
I get lots of messages. Both the openQA server and my desktop have stg.fedoraproject.org:9940 in their endpoints, so they are listening to the hub, but they're just not getting any messages from it.
I think this stopped working around 2018-11-30, as that's about the last time openQA results were showing up in staging resultsdb.
Curiouser and curiouser - now openqa-stg01 doesn't seem to be seeing any fedmsgs at all. Not even prod ones. I just ran fedmsg-tail-3 on both openqa-stg01 and openqa01 and left them for a while; openqa01 saw lots of messages, openqa-stg01 saw nothing.
I don't know what could be causing this...
ah, seems prod messages went missing because @puiterwijk commented out the prod hub in the config as part of debugging the issue with staging messages. so we can get prod messages back easy enough, but the missing staging messages are still a problem.
Metadata Update from @bowlofeggs:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: staging
This now seems to have spread to prod - see https://pagure.io/fedora-infrastructure/issue/7493 .
Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
to comment on this ticket.