[12:00:12] <zodbot> PROBLEM - rabbitmq01.iad2.fedoraproject.org/Check queue toddlers is WARNING: RABBITMQ_QUEUE WARNING - messages WARNING (13), messages_ready OK (9) messages_unacknowledged OK (4) consumers OK (4) (noc01) [12:20:13] <zodbot> PROBLEM - rabbitmq01.iad2.fedoraproject.org/Check queue toddlers is WARNING: RABBITMQ_QUEUE WARNING - messages WARNING (25), messages_ready OK (21) messages_unacknowledged OK (4) consumers OK (4) (noc01) [13:20:12] <zodbot> PROBLEM - rabbitmq01.iad2.fedoraproject.org/Check queue toddlers is WARNING: RABBITMQ_QUEUE WARNING - messages WARNING (86), messages_ready OK (82) messages_unacknowledged OK (4) consumers OK (4) (noc01)
2020-12-22 20:32:50,336 - [WARNING fedora_messaging.twisted.protocol] Returning message id 2020-787c72a9-b31e-4b59-8499-dc89d660ab5d to the queue 2020-12-22 20:32:50,337 - [ERROR fedora_messaging.message] Message (headers={}, body=b'{"artifact":{"uid":"3e6fa6ba98ef4903bcfa3451c00d9bda","id":4,"type":"pull-request","repository":"https://src.fedoraproject.org/rpms/rubygem-curb","comment_id":0,"commit_hash":null,"issuer":"pvalena"},"pipeline":{"build":"6168","name":"dist-git","id":"852f5d442f3f1752120ebd6d5ebc94aa60d8bc084c8f5444e586c3fbb9b7bc21"},"notification":{"recipients":[""]},"test":{"result":"failed","namespace":"fedora-ci.dist-git-pr","type":"tier0","category":"functional"},"generated_at":"2020-12-22T19:51:41.227Z","contact":{"docs":"https://github.com/fedora-ci/dist-git-pipeline","name":"dist-git","team":"Fedora CI","irc":"#fedora-ci","url":"https://github.com/fedora-ci/dist-git-pipeline","email":"ci@lists.fedoraproject.org"},"run":{"rebuild":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/rebuild","debug":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/console","log":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/console","log_stream":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/console","url":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/","log_raw":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/consoleText"},"error":{"reason":"Infrastructure Failure","url":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/console"},"version":"0.2.1"}') arrived without a schema header. A publisher is misbehaving! 2020-12-22 20:32:50,337 - [ERROR fedora_messaging.message] Message (headers={}, body=b'{"artifact":{"uid":"3e6fa6ba98ef4903bcfa3451c00d9bda","id":4,"type":"pull-request","repository":"https://src.fedoraproject.org/rpms/rubygem-curb","comment_id":0,"commit_hash":null,"issuer":"pvalena"},"pipeline":{"build":"6168","name":"dist-git","id":"852f5d442f3f1752120ebd6d5ebc94aa60d8bc084c8f5444e586c3fbb9b7bc21"},"notification":{"recipients":[""]},"test":{"result":"failed","namespace":"fedora-ci.dist-git-pr","type":"tier0","category":"functional"},"generated_at":"2020-12-22T19:51:41.227Z","contact":{"docs":"https://github.com/fedora-ci/dist-git-pipeline","name":"dist-git","team":"Fedora CI","irc":"#fedora-ci","url":"https://github.com/fedora-ci/dist-git-pipeline","email":"ci@lists.fedoraproject.org"},"run":{"rebuild":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/rebuild","debug":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/console","log":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/console","log_stream":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/console","url":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/","log_raw":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/consoleText"},"error":{"reason":"Infrastructure Failure","url":"https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipeline/job/master/6168/console"},"version":"0.2.1"}') arrived without a severity. A publisher is misbehaving! Defaulting to INFO. 2020-12-22 20:32:50,337 - [ERROR fedora_messaging.message] Message arrived without a content encoding 2020-12-22 20:32:50,338 - [INFO fedora_messaging.twisted.protocol] Consuming message from topic org.centos.prod.ci.dist-git-pr.test.error (message id 2020-787c72a9-b31e-4b59-8499-dc89d660ab5d) 2020-12-22 20:32:50,338 - [INFO toddlers.runner] Toddlers processing: 2020-787c72a9-b31e-4b59-8499-dc89d660ab5d -- org.centos.prod.ci.dist-git-pr.test.error ^C2020-12-22 20:32:50,338 - [INFO toddlers.runner] Toddler 'flag_ci_pr' accepted to process message id: 2020-787c72a9-b31e-4b59-8499-dc89d660ab5d 2020-12-22 20:32:50,338 - [ERROR toddlers.runner] Toddler 'flag_ci_pr' failed to process message id: 2020-787c72a9-b31e-4b59-8499-dc89d660ab5d -- putting it back in the queue Traceback (most recent call last): File "/code/toddlers/runner.py", line 96, in __call__ toddler.process(self.toddler_config[toddler.name], message) File "/code/toddlers/plugins/flag_ci_pr.py", line 106, in process commit_hash_text = " for %s" % msg["artifact"]["commit_hash"][:8] TypeError: 'NoneType' object is not subscriptable
I guess toddlers needs to handle this, we need to fix the sender and/or drop that message from the queue.
This is the message:
{"artifact":{"type":"pull-request","id":11,"issuer":"lis","repository":"https://src.fedoraproject.org/rpms/pam","commit_hash":null,"comment_id":0,"uid":"4aed457e544845c08d2f65c91630d3cc"},"pipeline":{"id":"c95839fe-4a94-4e6d-9496-320b826c3862","name":"fedora-rawhide-pr-pipeline","stage":{"name":"prepare-environment"},"build":"5046"},"test":{"category":"static-analysis","namespace":"fedora-ci.dist-git-pr","type":"build"},"generated_at":"2020-11-12T10:02:27.664884Z","contact":{"name":"fedora-ci","team":"fedora-ci","docs":"https://pagure.io/standard-test-roles","email":"ci@lists.fedoraproject.org","irc":"#fedora-ci","url":"https://jenkins-continuous-infra.apps.ci.centos.org/"},"run":{"url":"https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/5046/pipeline/","log":"https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/5046/console","rebuild":"https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/5046/rebuild/parameterized"},"version":"0.2.1"}
And indeed we have "commit_hash":null, now we'll need to ping the CI folks next year to find out why such message made it to the bus.
"commit_hash":null,
Other ones are:
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
I have opened a bug to the fedora-ci team asking them if they could validate their messages before sending them to the bus: https://pagure.io/fedora-ci/general/issue/195
Let's close this as fixed for now and I'll re-open if needed depending on the discussion with the fedora-ci folks.
Metadata Update from @pingou: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Issue status updated to: Open (was: Closed)
Issue status updated to: Closed (was: Open) Issue close_status updated to: Fixed
Login to comment on this ticket.