#12353 Centos messages to Fedora Messaging don't specify content encoding and severity
Opened a month ago by frostyx. Modified a month ago

I configured Sentry for the Fedora Review Service (which listens to all Fedora Messaging messages) and noticed that it is getting flooded with thousands of errors that some message doesn't specify content encoding and/or severity. Unfortunately, I had to delete all the reports because it quickly went over my account quota, but I did some investigation afterward.

It seems that the culprit messages have org.centos.* topic. I am not sure if all centos topics are "broken" or only some. But the content encoding is 100% reproducible here:
https://apps.fedoraproject.org/datagrepper/v2/search?topic=org.centos.prod.ci.koji-build.test.complete

The centos topics are enabled in the default Fedora Messaging config btw:
https://github.com/fedora-infra/fedora-messaging/blob/develop/configs/fedora.toml

Here is a minimal reproducer:

  1. Download the default Fedora Messaging config
  2. Edit it and set all logging levels to ERROR to limit the amount of output
  3. Create this consumer.py file
def consume(message):
      if not message.topic.startswith("org.centos.*"):
          return
      print(f"{message.id} - {message.topic}")
  1. Run FEDORA_MESSAGING_CONF=fedora.toml fedora-messaging consume --callback-file consumer.py:consume
  2. Listen for a while (I usually see the first errors in less than 10 minutes)

You can find the relevant log lines here:

rg "Message arrived without a content encoding" /usr/lib/python*/site-packages/fedora_messaging
rg "arrived without a severity" /usr/lib/python*/site-packages/fedora_messaging

I already contacted @arrfab and he said that services maintained by the Centos Infra don't publish that topic, so somebody else must be maintaining it, and we don't know who.

Describe what you would like us to do:

Contact the responsible teams to specify the missing fields.

When do you need this to be done by? (YYYY/MM/DD)

At your convenience, I workarounded the issue on my side
https://github.com/FrostyX/fedora-review-service/commit/87995e8ed867c07b49f9277f79f64faa5b19870f
so this doesn't affect me anymore.


Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, medium-gain

a month ago

@abompard Could you look at this.

It's strange that there is Fedora CI content in org.centos.prod.ci

It's probably the java-based sender, used in jenkins, that sends those messages: https://plugins.jenkins.io/jms-messaging/
All the senders that use the python fedora-messaging library for sending will add those headers automatically.

As for the topic specificity, I don't think the Jenkins JMS plugin is smart enough to choose a different topic depending on which code is being CI'ed.

Log in to comment on this ticket.

Metadata