This is blocking gating
See:
https://apps.fedoraproject.org/datagrepper/raw?topic=org.centos.prod.ci.pipeline.allpackages-build.complete&delta=127800
And:
https://taskotron.fedoraproject.org/resultsdb/results?&testcases=org.centos.prod.ci.pipeline.allpackages-build.complete&since=2019-07-30T00:00:00,2019-08-28T23:59:59
CC @pingou
As far as @kevin and I knew.. loopabull had nothing to do with the CI pipeline. Did this change and what would we look for?
I was wrong, loopabull is not involved. This is the service: https://pagure.io/ci-resultsdb-listener
Ticket https://pagure.io/fedora-ci/general/issue/66 speaks about PR flags, so that's done via loopabull, more specifically in: - https://pagure.io/Fedora-Infra/loopabull-tasks/blob/master/f/playbooks/org.centos.prod.ci.pipeline.allpackages-pr.package.running.yml and - https://pagure.io/Fedora-Infra/loopabull-tasks/blob/master/f/playbooks/org.centos.prod.ci.pipeline.allpackages-pr.complete.yml
But as you can already see from these links, neither one is watching the .error topic
So is this working? Or we still need to dig more?
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: ci
I'm still seeing problems. From my perspective -- as an ignorant end user with no clue of what greenwave, loopabull, or any of those terms are -- the issue is that between [1] and [2] the test status changed from "tests failed" to "test results missing". I am still seeing "missing" on [3] and [4], but those are only 2 and 3 hours old respectively, and in the past it has taken up to four hours to actually get results - so I can't be sure if the problem still exists, or if it's just the usual slowness from before.
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-bfff3112b5 [2] https://bodhi.fedoraproject.org/updates/FEDORA-2019-8d97c8b9e6
[3] https://bodhi.fedoraproject.org/updates/FEDORA-2019-8dce0aaefc [4] https://bodhi.fedoraproject.org/updates/FEDORA-2019-4670e94e9b
@pingou yeah sorry for the confusion with loopabull, this is about the regular builds ... so ci-resultsdb-updater ....
I deleted the confusing comments and stand corrected
The links from the description are still relevant though ... the last resultsdb results were created 4 days ago +-
the last resultsdb results were created 4 days ago +-
Hm that sounds like a different problem then, let me check this
Ok rdbsync had failed to restart after the last reboot.
I've restarted the service and it seems messages are back in resultsdb: https://taskotron.fedoraproject.org/resultsdb/results?testcases=org.centos.prod.ci.pipeline.allpackages-build.complete :)
Metadata Update from @pingou: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
@pingou is there a way we could monitor these services? We are about to setup prometheus in upstream to monitor some critical parts of the infra ...
@mvadkert I use this script https://pagure.io/fedora-ci/monitor-gating to monitor the entire pipeline, it's a start but not yet enough.
This is definitely something we'll want to improve/fix with automation and proper reporting.
Login to comment on this ticket.