About every 10 requests pdc is tossing back a 500 error with the following traceback:
[Wed Sep 22 15:38:13.571626 2021] [:error] [pid 14102] [client 192.168.1.191:35316] mod_wsgi (pid=14102): Exception occurred processing WSGI script '/usr/lib/python2.7/site-packages/pdc/wsgi.py'. [Wed Sep 22 15:38:13.571648 2021] [:error] [pid 14102] [client 192.168.1.191:35316] Traceback (most recent call last): [Wed Sep 22 15:38:13.571663 2021] [:error] [pid 14102] [client 192.168.1.191:35316] File "/usr/lib/python2.7/site-packages/pdc/wsgi.py", line 19, in <module> [Wed Sep 22 15:38:13.571692 2021] [:error] [pid 14102] [client 192.168.1.191:35316] application = get_wsgi_application() [Wed Sep 22 15:38:13.571698 2021] [:error] [pid 14102] [client 192.168.1.191:35316] File "/usr/lib/python2.7/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application [Wed Sep 22 15:38:13.571709 2021] [:error] [pid 14102] [client 192.168.1.191:35316] django.setup() [Wed Sep 22 15:38:13.571714 2021] [:error] [pid 14102] [client 192.168.1.191:35316] File "/usr/lib/python2.7/site-packages/django/__init__.py", line 17, in setup [Wed Sep 22 15:38:13.571723 2021] [:error] [pid 14102] [client 192.168.1.191:35316] configure_logging(settings.LOGGING_CONFIG, settings.LOGGING) [Wed Sep 22 15:38:13.571728 2021] [:error] [pid 14102] [client 192.168.1.191:35316] File "/usr/lib/python2.7/site-packages/django/utils/log.py", line 82, in configure_logging [Wed Sep 22 15:38:13.571737 2021] [:error] [pid 14102] [client 192.168.1.191:35316] dictConfig(DEFAULT_LOGGING) [Wed Sep 22 15:38:13.571741 2021] [:error] [pid 14102] [client 192.168.1.191:35316] File "/usr/lib64/python2.7/logging/config.py", line 803, in dictConfig [Wed Sep 22 15:38:13.571751 2021] [:error] [pid 14102] [client 192.168.1.191:35316] dictConfigClass(config).configure() [Wed Sep 22 15:38:13.571754 2021] [:error] [pid 14102] [client 192.168.1.191:35316] File "/usr/lib64/python2.7/logging/config.py", line 568, in configure [Wed Sep 22 15:38:13.571761 2021] [:error] [pid 14102] [client 192.168.1.191:35316] 'filter %r: %s' % (name, e)) [Wed Sep 22 15:38:13.571774 2021] [:error] [pid 14102] [client 192.168.1.191:35316] ValueError: Unable to configure filter u'require_debug_false': 'module' object has no attribute 'utils'
Perhaps @abompard might know off hand here, since I think he's used django...
I looked at it quickly and I didn't find where that error could come from, sorry. Should I investigate more?
If you have time/willingness to do so that would be great. :)
It's not super urgent, but it would sure be nice to fix... it's causing intermittent failures for people trying to fedpkg push things (since the pagure hook checks pdc for SLAs)
Does PDC have frozen dependencies? I would guess that there is now newer version of some package.
Metadata Update from @humaton: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: high-gain, medium-trouble
If you have time/willingness to do so that would be great. :) It's not super urgent, but it would sure be nice to fix... it's causing intermittent failures for people trying to fedpkg push things (since the pagure hook checks pdc for SLAs)
Looking at the pagure code, I don't see any SLA check in a pagure hooks, but the SLA check is in fedpkg itself.
I see that fedpkg is doing plenty of other PDC queries. Not sure what exactly is called for push.
It does not matter what request. Here is a reproducer:
$ while wget -O /dev/null 'https://pdc.fedoraproject.org/rest_api/v1/'; do N=$((N+1)); echo "$N"; done
@ppisar This really looks like the issue on PDC side then.
I tried your reproducer and I got 87 responses with 200 OK and then unable to resolve host address ‘pdc.fedoraproject.org’
200 OK
unable to resolve host address ‘pdc.fedoraproject.org’
So I logged to the machine where PDC is running and there is one particular command that was executed $ rm /usr/lib/python2.7/site-packages/fedora/django/utils.py and some changes to files responsible for logging in Django.
$ rm /usr/lib/python2.7/site-packages/fedora/django/utils.py
It does not matter what request. Here is a reproducer: $ while wget -O /dev/null 'https://pdc.fedoraproject.org/rest_api/v1/'; do N=$((N+1)); echo "$N"; done
I tried to run it a couple of times, but no 500, just DNS error after a lot of requests.
Interesting. Yesterday I got error 500 after 20, 30 requests. Today 100 attempts and still good.
Checking logs, I don't see any 500's for the last 5 days now. :)
So I guess it fixed itself somehow? Or it was timing or load related?
Feel free to re-open if you see this again.
Metadata Update from @kevin: - Issue close_status updated to: Insufficient data - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.