fedora-infrastructure

#12649 src.fp.o gives Internal Server Errors until I prune cookies, repeats after 1/2 day
Closed: Fixed with Explanation by kevin. Opened by churchyard.

Closed: Fixed with Explanation
Reopen Issue

Describe what you would like us to do:

After the DC move, I was getting Internal Server Errors on src.fedoraproject.org. I assumed it's not yet up but when it worked for others, I deleted my cookies. The problem was gone.

Except it happened again today around 12:00 UTC. I pruned the cookies again and it helped again.

But now it happened again. Deleting cookies still helps, but this is annoying.

At least @lzachar and @cstratak were having similar issues.

When do you need this to be done by? (YYYY/MM/DD)

Any time is good. It's not blocking me, but it is rather tedious.

For me deleting all cookies in firefox didn't do much, I had to go to the developer tools with F12 and delete the session cookie.

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: Needs investigation, low-gain, ops, src.fp.o

Added myself to CC as being affected by it too. It happened multiple times to me, today again.

This happened after the DC move and after deleting my Fedora cookies then it's been fine since, until today. Clearing my cookies again helped, but that shouldn't be necessary.

FWIW I was logged into src.fp.o last week, but when I first tried today, my TGT was expired. fkinit-ing again didn't help, I still had to clear my cookies, and then I could access src.fp.o and log back in.

Could folks seeing this perhaps share browser and if they have any cookie related extensions?

I'm confused as it why this seems to only affect some folks, but not others...

Ugh. ok. I see segfaults in the error logs... very possibly related somehow.

I'll try and get a core and see if that tells us whats going on. :(

It's Firefox 136.0.2 (64-bit) here. No special extension installed.

Firefox 140.0, no cookie extensions, just classic add-ons like uBlock.

I've just got it again, when I clicked on a URL https://src.fedoraproject.org/rpms/NetworkManager/pull-request/25 in a mail:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at webmaster@fedoraproject.org to inform them
of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.

I had been on that site (not precisely this page, but other pull requests) today few times before.

I'm not clearing the cookies, in case you'd like me to re-trigger it.

I saw it as well today on Brave Browser 1.80.120. I checked the logs on pkgs01.rdu3.fedoraproject.org and it seems that the requests aren't even in the access_log.

I needed to resolve something else, so I didn't investigate it deeper. I will try that next time it happens.

It seems like they never get to pkgs01...

on proxy01:

REDACTED - - [17/Jul/2025:02:55:49 +0000] "GET /theme/static/pagure-logo.png?version=5.14.1 HTTP/2.0" 500 612 "https://src.fedoraproject.org/rpms/blender/c/80304e79fa8e78b882871bec508f9dc7d
4289b08" "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0"

[Thu Jul 17 02:55:49.375608 2025] [proxy_http:error] [pid 1798579:tid 1798810] (20014)Internal error (specific information not available): [remote REDACTED:61296] AH01102: error reading status line from remote server localhost:10057, referer: https://src.fedoraproject.org/rpms/blender/c/80304e79fa8e78b882871bec508f9dc7d4289b08

So, it seems like it's getting some error between httpd and the backend, in haproxy. But I don't understand why clearing cookies would change anything... so perhaps this is not the right case/something else.

In haproxy logs: 

Jul 15 10:18:45 proxy01.rdu3.fedoraproject.org haproxy[1304589]: [WARNING]  (1304589) : Server src-backend/pkgs01 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Jul 15 10:18:45 proxy01.rdu3.fedoraproject.org haproxy[1304589]: [ALERT]    (1304589) : backend 'src-backend' has no server available!
Jul 15 10:20:05 proxy01.rdu3.fedoraproject.org haproxy[1304589]: [WARNING]  (1304589) : Server src-backend/pkgs01 is UP, reason: Layer7 check passed, code: 302, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

So, I wonder... is clearing the cookies really needed? Or if you want a few minutes and retry does it start working again?

and we should fix it giving connection refused of course.

When I said I can reproduce on will above, I left it in that state until yesterday. I've got the error consistently, during those days. I wanted to actually do something in the src.fp.o, thus I cleared the cookies and then I could login to the page with not problem. Thus yes, it's about cookies, waiting (for days) does not help.

From your log:

Layer4 connection problem, info: "Connection refused"
backend 'src-backend' has no server available!

Can it be the reason for the Internal Server Error? It failed to login (use saved "credentials" in the cookie), then it deciphered it as no server available, instead of "ask for login or open unauthenticated".

When I said I can reproduce on will above, I left it in that state until yesterday. I've got the error consistently, during those days. I wanted to actually do something in the src.fp.o, thus I cleared the cookies and then I could login to the page with not problem. Thus yes, it's about cookies, waiting (for days) does not help.

ok. I just wanted to confirm that... thanks.

From your log:

Layer4 connection problem, info: "Connection refused"
backend 'src-backend' has no server available!

Can it be the reason for the Internal Server Error? It failed to login (use saved "credentials" in the cookie), then it deciphered it as no server available, instead of "ask for login or open unauthenticated".

Could be... could you let me know your ip address? either here or privately? I want to see if I can line things up any in logs.

Although, looking at logs, I don't see any of these for the last 2 days.
We fixed a mtu issue 2 days ago.

hurray. It finally happened to me.

I got a traceback:

[Sat Jul 19 02:43:21.641926 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586] mod_wsgi (pid=649073): Exception occurred processing WSGI script '/var/www/pagure.wsgi'.
[Sat Jul 19 02:43:21.647082 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586] Traceback (most recent call last):                                                      
[Sat Jul 19 02:43:21.647124 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__          
[Sat Jul 19 02:43:21.647128 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     return self.wsgi_app(environ, start_response)                                       
[Sat Jul 19 02:43:21.647132 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/whitenoise/base.py", line 85, in __call__      
[Sat Jul 19 02:43:21.647136 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     return self.application(environ, start_response)                                    
[Sat Jul 19 02:43:21.647148 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/pagure/proxy.py", line 74, in __call__         
[Sat Jul 19 02:43:21.647160 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     return self.app(environ, start_response)                                            
[Sat Jul 19 02:43:21.647164 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1993, in wsgi_app          
[Sat Jul 19 02:43:21.647167 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     ctx.auto_pop(error)
[Sat Jul 19 02:43:21.647170 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/flask/ctx.py", line 387, in auto_pop
[Sat Jul 19 02:43:21.647174 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     self.pop(exc)                                                                       
[Sat Jul 19 02:43:21.647177 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/flask/ctx.py", line 353, in pop
[Sat Jul 19 02:43:21.647181 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     self.app.do_teardown_request(exc)                                                   
[Sat Jul 19 02:43:21.647184 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1879, in do_teardown_request
[Sat Jul 19 02:43:21.647187 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     func(exc)
[Sat Jul 19 02:43:21.647190 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/pagure/flask_app.py", line 536, in end_request
[Sat Jul 19 02:43:21.647194 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     flask.g.session.remove()
[Sat Jul 19 02:43:21.647201 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]   File "/usr/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__
[Sat Jul 19 02:43:21.647209 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586]     return getattr(self._get_current_object(), name)
[Sat Jul 19 02:43:21.647220 2025] [wsgi:error] [pid 649073:tid 140610590394112] [remote 10.16.163.75:33586] AttributeError: '_AppCtxGlobals' object has no attribute 'session'
[Sat Jul 19 02:43:22.040889 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622] mod_wsgi (pid=649066): Exception occurred processing WSGI script '/var/www/pagure.wsgi'.
[Sat Jul 19 02:43:22.041626 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622] Traceback (most recent call last):
[Sat Jul 19 02:43:22.041660 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
[Sat Jul 19 02:43:22.041664 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     return self.wsgi_app(environ, start_response)
[Sat Jul 19 02:43:22.041669 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/whitenoise/base.py", line 85, in __call__
[Sat Jul 19 02:43:22.041672 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     return self.application(environ, start_response)
[Sat Jul 19 02:43:22.041676 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/pagure/proxy.py", line 74, in __call__
[Sat Jul 19 02:43:22.041682 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     return self.app(environ, start_response)
[Sat Jul 19 02:43:22.041686 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1993, in wsgi_app
[Sat Jul 19 02:43:22.041692 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     ctx.auto_pop(error)
[Sat Jul 19 02:43:22.041699 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/flask/ctx.py", line 387, in auto_pop
[Sat Jul 19 02:43:22.041724 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     self.pop(exc)
[Sat Jul 19 02:43:22.041728 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/flask/ctx.py", line 353, in pop
[Sat Jul 19 02:43:22.041731 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     self.app.do_teardown_request(exc)
[Sat Jul 19 02:43:22.041735 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1879, in do_teardown_request
[Sat Jul 19 02:43:22.041738 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     func(exc)
[Sat Jul 19 02:43:22.041741 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/pagure/flask_app.py", line 536, in end_request
[Sat Jul 19 02:43:22.041744 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     flask.g.session.remove()
[Sat Jul 19 02:43:22.041748 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]   File "/usr/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__
[Sat Jul 19 02:43:22.041751 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622]     return getattr(self._get_current_object(), name)
[Sat Jul 19 02:43:22.041760 2025] [wsgi:error] [pid 649066:tid 140610582001408] [remote 10.16.163.75:33622] AttributeError: '_AppCtxGlobals' object has no attribute 'session'

This commit should have prevented the issue: https://pagure.io/pagure/c/b50d32b7c92f131ebcc3b633de6c6e91e28297ec?branch=master
The pagure.io instance has a similar fix, not sure why the distgit instance does not have it.
I'll hotfix it and investigate a bit more.

It is possible that we (or I) hotfixed our IAD2 instance and didn't add the patch to the pagure RPM.
Since the Pagure RPM comes from EPEL and I'm not sure we want to carry our own in the infra repo, I've added a system in Ansible to patch Pagure, as we already have for Ipsilon.
This should make sure this problem doesn't happen again: https://pagure.io/fedora-infra/ansible/c/be82a1ee19c8adf6e428c6ca1686b296fcfa76b7

The ansible one should be enough for now.

Yes.

I guess then I will close this? And if any folks on this issue see it happening again, please re-open or file a new issue.

Sorry for the hassle.

Metadata Update from @kevin:
- Issue assigned to abompard
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)
Metadata

Needs investigation low-gain ops src.fp.o

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.