fedora-infrastructure

#7100 Fedora CI Pipeline send topics, but do not get to fedmsg
Closed: Fixed Opened by bgoncalv.

Closed: Fixed
Reopen Issue

The Fedora CI pipelines seems to send a message to fedmsg, but the message never gets to fedmsg.

Example:

Sending message for job 'fedora-build-pipeline-trigger'.
{"i":1,"msg":{"CI_NAME":"fedora-build-pipeline-trigger","CI_TYPE":"custom","branch":"27","build_id":"115720","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-build-pipeline-trigger/detail/fedora-build-pipeline-trigger/115720/pipeline/","comment_id":"null","message-content":"","namespace":"null","nvr":"","original_spec_nvr":"","ref":"x86_64","repo":"null","rev":"kojitask-28121611","scratch":"false","status":"SUCCESS","test_guidance":"''","topic":"org.centos.prod.ci.pipeline.allpackages-build.package.ignored","username":"releng"},"msg_id":"2018-fe611ae0-b5bf-4119-b9c2-837d32de86cb","timestamp":1531298861,"topic":"org.centos.prod.ci.pipeline.allpackages-build.package.ignored"}
$ curl --silent --output /dev/null --write-out '%{http_code}' 'https://apps.fedoraproject.org/datagrepper/id?id=2018-fe611ae0-b5bf-4119-b9c2-837d32de86cb&chrome=false&is_raw=false'
404

Checking the topics on https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/inventory/group_vars/centos-ci only Fedora Atomic pipeline topics are defined.

The topics for Fedora pipeline are:

org.centos.prod.ci.pipeline.allpackages-pr.package.ignored
org.centos.prod.ci.pipeline.allpackages-pr.package.queued
org.centos.prod.ci.pipeline.allpackages-pr.package.running
org.centos.prod.ci.pipeline.allpackages-pr.package.complete
org.centos.prod.ci.pipeline.allpackages-pr.image.queued
org.centos.prod.ci.pipeline.allpackages-pr.image.running
org.centos.prod.ci.pipeline.allpackages-pr.image.complete
org.centos.prod.ci.pipeline.allpackages-pr.package.test.functional.queued
org.centos.prod.ci.pipeline.allpackages-pr.package.test.functional.running
org.centos.prod.ci.pipeline.allpackages-pr.package.test.functional.complete
org.centos.prod.ci.pipeline.allpackages-pr.complete
org.centos.prod.ci.pipeline.allpackages-build.package.ignored
org.centos.prod.ci.pipeline.allpackages-build.package.queued
org.centos.prod.ci.pipeline.allpackages-build.package.running
org.centos.prod.ci.pipeline.allpackages-build.package.complete
org.centos.prod.ci.pipeline.allpackages-build.image.queued
org.centos.prod.ci.pipeline.allpackages-build.image.running
org.centos.prod.ci.pipeline.allpackages-build.image.complete
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.queued
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.running
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.complete
org.centos.prod.ci.pipeline.allpackages-build.complete

I think the problem is: https://koji.fedoraproject.org/koji/taskinfo?taskID=28121611 -- this is task. This is not package. It is image. We should ignore messages that are not packages.

Edited by astepano

Please make sure that the messages you are sending go through the correct fedmsg-relay at CentOS, so that they don't get ignored by being signed by the wrong system.
Could you provide the certificate that is signing the messages?

Note that if a topic is not listed anywhere in our configuration, it is processed as if everyone can send it, so not being listed means no problems.

Looks like we're missing these:

ci.pipeline.allpackages-build.image.queued
ci.pipeline.allpackages-build.image.running
ci.pipeline.allpackages-build.image.complete
ci.pipeline.allpackages-build.package.test.functional.queued
ci.pipeline.allpackages-build.package.test.functional.running
ci.pipeline.allpackages-build.package.test.functional.complete
ci.pipeline.allpackages-build.complete

Right, but missing things won't make messages disappear - they'll mean that anyone can send them.

Please make sure that the messages you are sending go through the correct fedmsg-relay at CentOS, so that they don't get ignored by being signed by the wrong system.
Could you provide the certificate that is signing the messages?
Note that if a topic is not listed anywhere in our configuration, it is processed as if everyone can send it, so not being listed means no problems.

@jbieren could you provide details about this question?

I just know this was working before, the problem started less than 24hrs ago...

@bgoncalv that would match with us adding a sender policy for those topics, yes.
So we need to make sure that all the messages come from the expected certificate.

To provide a little more info, this worked at 12:45:08AM UTC and did not work at 12:51:55AM UTC (both today, July 11). The messages still seem to properly be sent out on topic ci.pipeline.allpackages-pr.package.ignored, but ci.pipeline.allpackages-build.package.ignored is not working. The only place I know of to check for these things is
https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure/blob/master/fedmsg_meta_fedora_infrastructure/centos_ci.py
which has no recent commits. So I am not sure. I will ask Brian

So, as I said, this is related to us adding a policy for allowing for sending of these topics.

Some of the related commits:
- https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=71de532
- https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=0bdca4d

Note that this means that the messages with that topic must be signed by a trusted certificate, which is why I asked to please confirm with which certificate they are being signed.

Maybe the trusted certificate plain doesn't work then. @bstinson can you shed any light on this?

Also, that file has some typos.. for example it has "ignore" in some cases and "ignored" in others. That is probably why it is still working in some cases now. The topics we publish on from centos ci are (/me now sees that the atomic pipeline has no d in ignore which is probably where the confusion came from):

org.centos.prod.ci.pipeline.allpackages-build.package.ignored
org.centos.prod.ci.pipeline.allpackages-build.package.queued
org.centos.prod.ci.pipeline.allpackages-build.package.running
org.centos.prod.ci.pipeline.allpackages-build.package.complete
org.centos.prod.ci.pipeline.allpackages-build.image.queued
org.centos.prod.ci.pipeline.allpackages-build.image.running
org.centos.prod.ci.pipeline.allpackages-build.image.complete
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.queued
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.running
org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.complete
org.centos.prod.ci.pipeline.allpackages-build.complete
org.centos.prod.ci.pipeline.allpackages-pr.package.ignored
org.centos.prod.ci.pipeline.allpackages-pr.package.queued
org.centos.prod.ci.pipeline.allpackages-pr.package.running
org.centos.prod.ci.pipeline.allpackages-pr.package.complete
org.centos.prod.ci.pipeline.allpackages-pr.image.queued
org.centos.prod.ci.pipeline.allpackages-pr.image.running
org.centos.prod.ci.pipeline.allpackages-pr.image.complete
org.centos.prod.ci.pipeline.allpackages-pr.package.test.functional.queued
org.centos.prod.ci.pipeline.allpackages-pr.package.test.functional.running
org.centos.prod.ci.pipeline.allpackages-pr.package.test.functional.complete
org.centos.prod.ci.pipeline.allpackages-pr.complete
org.centos.prod.ci.pipeline.package.ignore
org.centos.prod.ci.pipeline.package.queued
org.centos.prod.ci.pipeline.package.running
org.centos.prod.ci.pipeline.package.complete
org.centos.prod.ci.pipeline.package.test.functional.queued
org.centos.prod.ci.pipeline.package.test.functional.running
org.centos.prod.ci.pipeline.package.test.functional.complete
org.centos.prod.ci.pipeline.compose.running
org.centos.prod.ci.pipeline.compose.complete
org.centos.prod.ci.pipeline.image.running
org.centos.prod.ci.pipeline.image.complete
org.centos.prod.ci.pipeline.image.test.smoke.running
org.centos.prod.ci.pipeline.image.test.smoke.complete
org.centos.prod.ci.pipeline.compose.test.integration.queued
org.centos.prod.ci.pipeline.compose.test.integration.running
org.centos.prod.ci.pipeline.compose.test.integration.complete
org.centos.prod.ci.pipeline.complete

I fixed up some config on our end... does this seem like it's better/working now?

It seems to be consistent now (all topics we publish on have the same behavior). I honestly think that cert may just be bad or expired or something. Will have to wait for @bstinson though as he is the SME on centos ci and how the relay works and has the perms for it all from the centos side

Seems working, our certs definitely aren't expired. I'm wondering if we just missed a message.

We have ci.pipeline.allpackages-build.package.ignored messages from 20 hours ago:

https://apps.fedoraproject.org/datagrepper/id?id=2018-044f1624-c4eb-4994-a1f7-3b25e0b658df&is_raw=true&size=extra-large

and again 2 hours ago:
https://apps.fedoraproject.org/datagrepper/id?id=2018-26da6cfe-4e40-47f3-837b-916d58a7a0b3&is_raw=true&size=extra-large

If the certs were expired, or there were other troubles with the relay infra we'd see many fewer messages here:
https://apps.fedoraproject.org/datagrepper/raw?category=ci

Yes, it seems to be working well.

Yeah, looks fixed to me now as well. Not sure what did it. I saw some failures still after those config changes 18 hours prior to this message, but it looks like it was fixed ~11 hours ago. Glad it is working again. Thanks for digging in @bstinson

Awesome. Yeah, I did some more fixing last night... sorry for the hassles here.

Let us know if you see anything further messed up.

:eyeglasses:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.