I configured Sentry for the Fedora Review Service (which listens to all Fedora Messaging messages) and noticed that it is getting flooded with thousands of errors that some message doesn't specify content encoding and/or severity. Unfortunately, I had to delete all the reports because it quickly went over my account quota, but I did some investigation afterward.
It seems that the culprit messages have org.centos.* topic. I am not sure if all centos topics are "broken" or only some. But the content encoding is 100% reproducible here: https://apps.fedoraproject.org/datagrepper/v2/search?topic=org.centos.prod.ci.koji-build.test.complete
org.centos.*
The centos topics are enabled in the default Fedora Messaging config btw: https://github.com/fedora-infra/fedora-messaging/blob/develop/configs/fedora.toml
Here is a minimal reproducer:
ERROR
consumer.py
def consume(message): if not message.topic.startswith("org.centos.*"): return print(f"{message.id} - {message.topic}")
FEDORA_MESSAGING_CONF=fedora.toml fedora-messaging consume --callback-file consumer.py:consume
You can find the relevant log lines here:
rg "Message arrived without a content encoding" /usr/lib/python*/site-packages/fedora_messaging rg "arrived without a severity" /usr/lib/python*/site-packages/fedora_messaging
I already contacted @arrfab and he said that services maintained by the Centos Infra don't publish that topic, so somebody else must be maintaining it, and we don't know who.
Contact the responsible teams to specify the missing fields.
At your convenience, I workarounded the issue on my side https://github.com/FrostyX/fedora-review-service/commit/87995e8ed867c07b49f9277f79f64faa5b19870f so this doesn't affect me anymore.
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, medium-gain
@abompard Could you look at this.
It's strange that there is Fedora CI content in org.centos.prod.ci
org.centos.prod.ci
It's probably the java-based sender, used in jenkins, that sends those messages: https://plugins.jenkins.io/jms-messaging/ All the senders that use the python fedora-messaging library for sending will add those headers automatically.
As for the topic specificity, I don't think the Jenkins JMS plugin is smart enough to choose a different topic depending on which code is being CI'ed.
I have opened a ticket upstream: https://github.com/jenkinsci/jms-messaging-plugin/issues/269
Log in to comment on this ticket.