Learn more about these different git repos.
Other Git URLs
When I tried to waive an update in staging just now, the Bodhi UI told me:
Bodhi failed to send POST request to Greenwave at the following URL "https://greenwave-web-greenwave.app.os.stg.fedoraproject.org/api/v1.0/decision". The status code was "500".
In Bodhi's logs, I see this, which isn't that useful, but perhaps we can correlate timestamps with Greenwave's logs:
[Tue May 22 14:28:28.963262 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] 2018-05-22 14:28:28,962 DEBUG [bodhi.server][Dummy-4] Attempting to waive test results for this update FEDORA-2018-6404a539e5, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.306381 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] 2018-05-22 14:28:32,306 ERROR [bodhi.server][Dummy-4] Bodhi failed to send POST request to Greenwave at the following URL "https://greenwave-web-greenwave.app.os.stg.fedoraproject.org/api/v1.0/decision". The status code was "500"., referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.308738 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] 2018-05-22 14:28:32,306 ERROR [bodhi.server][Dummy-4] Unhandled exception in waive_test_results, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.308823 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] Traceback (most recent call last):, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.308902 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] File "/usr/lib/python2.7/site-packages/bodhi/server/services/updates.py", line 576, in waive_test_results, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.308933 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] update.waive_test_results(request.user.name, comment), referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.308980 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] File "/usr/lib/python2.7/site-packages/bodhi/server/models.py", line 2652, in waive_test_results, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309013 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] decision = greenwave_api_post('{}/decision'.format(config.get('greenwave_api_url')), data), referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309037 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] File "/usr/lib/python2.7/site-packages/bodhi/server/util.py", line 1373, in greenwave_api_post, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309059 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] data=data, retries=3), referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309079 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] File "/usr/lib/python2.7/site-packages/bodhi/server/util.py", line 1305, in call_api, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309103 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] return call_api(api_url, service_name, error_key, method, data, headers, retries - 1), referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309124 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] File "/usr/lib/python2.7/site-packages/bodhi/server/util.py", line 1305, in call_api, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309146 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] return call_api(api_url, service_name, error_key, method, data, headers, retries - 1), referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309167 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] File "/usr/lib/python2.7/site-packages/bodhi/server/util.py", line 1305, in call_api, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309221 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] return call_api(api_url, service_name, error_key, method, data, headers, retries - 1), referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309246 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] File "/usr/lib/python2.7/site-packages/bodhi/server/util.py", line 1311, in call_api, referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309268 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] raise RuntimeError(error_msg), referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27 [Tue May 22 14:28:32.309289 2018] [wsgi:error] [pid 13:tid 139739764086528] [client 10.128.2.1:42256] RuntimeError: Bodhi failed to send POST request to Greenwave at the following URL "https://greenwave-web-greenwave.app.os.stg.fedoraproject.org/api/v1.0/decision". The status code was "500"., referer: https://bodhi.stg.fedoraproject.org/updates/python-rpdb-0.1.6-9.fc27
Sounds like WaiverDB is down:
<pingou> threebean: bowlofeggs: regarding this greenwave issue: {u'message': u'503 Server Error: Service Unavailable for url: https://waiverdb-web-waiverdb.app.os.stg.fedoraproject.org/api/v1.0/waivers/+by-subjects-and-testcases'}
WaiverDB outage reported to infra: https://pagure.io/fedora-infrastructure/issue/6967
However, I suggest that 500 is the wrong code for Greenwave to return when WaiverDB is down. It's not truly an "internal" error to Greenwave. Perhaps HTTP 502 (bad gateway) would make sense, since Greenwave is kind of acting as a gateway to WaiverDB here.
It would also be handy if the Greenwave error message told Bodhi that WaiverDB was down for debugging purposes - I could have more quickly identified and reported the issue if that info was passed along via the error message.
I believe it does, that's how I found it out
I have kicked waiverdb. Should be back soon.
Heh, +1 to moving to 502. URGENT PRIORITY! No rush, right?
See also #38.
PR #239
Metadata Update from @dcallagh: - Issue assigned to acorvin - Issue close_status updated to: Fixed - Issue set to the milestone: 0.8 - Issue status updated to: Closed (was: Open) - Issue tagged with: enhancement
Login to comment on this ticket.