fedora-infrastructure

Clone
Source Code
GIT

#12845 502/503 when reaching https://koji.fedoraproject.org/kojihub/
Closed: Duplicate a month ago by nphilipp. Opened a month ago by zlopez.

Closed: Duplicate
Reopen Issue

Describe what you would like us to do:

This started happening few days back, when trying to reproduce with

curl -v -X POST --header "Content-Type: text/xml" --data '<?xml version="1.0"?><methodCall><methodName>getAPIVersion</methodName><params></params></methodCall>' https://koji.fedoraproject.org/kojihub/

I was able to reproduce the problem on my machine.

I then tried the same from proxy machine with

curl -v -X POST --header "Content-Type: text/xml" --data '<?xml version="1.0"?><methodCall><methodName>getAPIVersion</methodName><params></params></methodCall>' koji01.rdu3.fedoraproject.org/kojihub

and didn't saw this issue.

But I can see a lot of errors in httpd log on proxy for koji. My assumption is that something is wrong with the proxy itself as there are no errors in log on koji side.

When do you need this to be done by? (YYYY/MM/DD)

This is blocking a lot of builds and PRs, so it would be nice to resolve it quickly

Metadata Update from @zlopez:
- Issue assigned to zlopez
a month ago

So we spent some time with @nphilipp debugging this. Here is what we tried:

  • Enabling debug log level on one of the proxies (but there wasn't anything interesting in the logs even with that)
  • We found out that there is a lot of crashes of httpd in the journal (but as only koji is having these issues I don't think it's related)
  • The errors in proxy httpd log for koji were there even before people started complaining about the koji issues (the errors are happening regularly even in the oldest logs on the machine 08.10.2025 Connection timed out: AH00957: http: attempt to connect to 10.16.169.105:80 (koji02.rdu3.fedoraproject.org:80) failed and the users started reporting issues 2 days ago)
  • The error doesn't happen when trying the curl directly from proxy machine to koji01-02 machine, I only noticed that when reaching through proxy
  • There doesn't seem to be any connectivity issue between koji01-02 and proxy01,proxy10 that could cause it

There were few updates happening on proxy10 on 13th October. Maybe some of them are causing the issue we see:

$ dnf history info 34
Transaction ID : 34
Begin time     : 2025-10-13 18:28:51
Begin rpmdb    : 7cd1988054b9c8c3143d05709bf14be75f70a9efd46870f097ca90826b537ba0
End time       : 2025-10-13 18:29:28
End rpmdb      : d10c66b2c122fa52296109e5dae7dfa03ce2d6d7f571964fb5c3f041abd3bf6e
User           : 0 Super User <root>
Status         : Ok
Releasever     : 42
Description    : dnf --refresh -y update
Comment        : 
Packages altered:
  Action   Package                                       Reason        Repository
  Upgrade  hwdata-0:0.400-1.fc42.noarch                  Dependency    updates
  Upgrade  json-glib-0:1.10.8-1.fc42.x86_64              Dependency    updates
  Upgrade  nspr-0:4.37.0-3.fc42.x86_64                   Dependency    updates
  Upgrade  nss-0:3.116.0-1.fc42.x86_64                   Dependency    updates
  Upgrade  nss-softokn-0:3.116.0-1.fc42.x86_64           Dependency    updates
  Upgrade  nss-util-0:3.116.0-1.fc42.x86_64              Dependency    updates
  Upgrade  nss-softokn-freebl-0:3.116.0-1.fc42.x86_64    Dependency    updates
  Upgrade  nss-tools-0:3.116.0-1.fc42.x86_64             User          updates
  Upgrade  nss-sysinit-0:3.116.0-1.fc42.x86_64           Dependency    updates
  Upgrade  procps-ng-0:4.0.4-6.fc42.1.x86_64             Group         updates
  Upgrade  selinux-policy-0:42.12-1.fc42.noarch          Dependency    updates
  Upgrade  selinux-policy-targeted-0:42.12-1.fc42.noarch Group         updates
  Upgrade  strace-0:6.17-1.fc42.x86_64                   User          updates
  Upgrade  zabbix-1:7.2.13-1.fc42.x86_64                 External User updates
  Upgrade  zabbix-agent-1:7.2.13-1.fc42.x86_64           External User updates
  Upgrade  zabbix-selinux-1:7.2.13-1.fc42.noarch         External User updates
  Upgrade  zsh-0:5.9-17.fc42.x86_64                      External User updates
  Replaced hwdata-0:0.399-1.fc42.noarch                  Dependency    @System
  Replaced json-glib-0:1.10.6-2.fc42.x86_64              Dependency    @System
  Replaced nspr-0:4.37.0-2.fc42.x86_64                   Dependency    @System
  Replaced nss-0:3.115.1-1.fc42.x86_64                   Dependency    @System
  Replaced nss-softokn-0:3.115.1-1.fc42.x86_64           Dependency    @System
  Replaced nss-softokn-freebl-0:3.115.1-1.fc42.x86_64    Dependency    @System
  Replaced nss-sysinit-0:3.115.1-1.fc42.x86_64           Dependency    @System
  Replaced nss-tools-0:3.115.1-1.fc42.x86_64             User          @System
  Replaced nss-util-0:3.115.1-1.fc42.x86_64              Dependency    @System
  Replaced procps-ng-0:4.0.4-6.fc42.x86_64               Group         @System
  Replaced selinux-policy-0:42.9-1.fc42.noarch           Dependency    @System
  Replaced selinux-policy-targeted-0:42.9-1.fc42.noarch  Group         @System
  Replaced strace-0:6.16-1.fc42.x86_64                   User          @System
  Replaced zabbix-1:7.2.12-1.fc42.x86_64                 External User @System
  Replaced zabbix-agent-1:7.2.12-1.fc42.x86_64           External User @System
  Replaced zabbix-selinux-1:7.2.12-1.fc42.noarch         External User @System
  Replaced zsh-0:5.9-16.fc42.x86_64                      External User @System

$ dnf history info 35
Transaction ID : 35
Begin time     : 2025-10-13 23:00:39
Begin rpmdb    : d10c66b2c122fa52296109e5dae7dfa03ce2d6d7f571964fb5c3f041abd3bf6e
End time       : 2025-10-13 23:00:40
End rpmdb      : 5404fbd784d2b5bb8b9c2399ae7afdc288e723b6b3a4f3f8248a5895976da6e9
User           : 0 Super User <root>
Status         : Ok
Releasever     : 42
Description    : dnf downgrade openssl*
Comment        : 
Packages altered:
  Action    Package                            Reason     Repository
  Downgrade openssl-1:3.2.4-3.fc42.x86_64      Dependency fedora
  Downgrade openssl-libs-1:3.2.4-3.fc42.x86_64 Dependency fedora
  Replaced  openssl-1:3.2.6-2.fc42.x86_64      Dependency @System
  Replaced  openssl-libs-1:3.2.6-2.fc42.x86_64 Dependency @System

$ dnf history info 36
Transaction ID : 36
Begin time     : 2025-10-13 23:08:07
Begin rpmdb    : 5404fbd784d2b5bb8b9c2399ae7afdc288e723b6b3a4f3f8248a5895976da6e9
End time       : 2025-10-13 23:08:07
End rpmdb      : ace9005ca8443cc646f33dcd8849b33b881a9668bdb6258689de662d44b204c0
User           : 0 Super User <root>
Status         : Ok
Releasever     : 42
Description    : dnf --refresh -y update
Comment        : 
Packages altered:
  Action   Package                            Reason     Repository
  Upgrade  openssl-1:3.2.6-2.fc42.x86_64      Dependency updates
  Upgrade  openssl-libs-1:3.2.6-2.fc42.x86_64 Dependency updates
  Replaced openssl-1:3.2.4-3.fc42.x86_64      Dependency @System
  Replaced openssl-libs-1:3.2.4-3.fc42.x86_64 Dependency @System

It seems that the openssl library was downgraded and upgraded again in the same day. I'm not sure why the downgrade happened.

Hmm. If I run the request internally in a loop (“flooding”), it hangs after a couple of seconds (about 5, where I interrupted):

[root@proxy01 ~][PROD-RDU3]# while curl -v -X POST --header "Content-Type: text/xml" --data '<?xml version="1.0"?><methodCall><methodName>getAPIVersion</methodName><params></params></methodCall>' koji01.rdu3.fedoraproject.org/kojihub; do true; done
…
* Connection #0 to host koji01.rdu3.fedoraproject.org left intact
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host koji01.rdu3.fedoraproject.org:80 was resolved.
* IPv6: (none)
* IPv4: 10.16.169.104
*   Trying 10.16.169.104:80...
* Connected to koji01.rdu3.fedoraproject.org (10.16.169.104) port 80
* using HTTP/1.x
> POST /kojihub HTTP/1.1
> Host: koji01.rdu3.fedoraproject.org
> User-Agent: curl/8.11.1
> Accept: */*
> Content-Type: text/xml
> Content-Length: 101
> 
* upload completely sent off: 101 bytes
< HTTP/1.1 200 OK
< Date: Wed, 15 Oct 2025 09:50:19 GMT
< Server: Apache
< Upgrade: h2,h2c
< Connection: Upgrade
< Koji-Version: 1.35.3
< Content-Length: 121
< AppTime: D=1363
< X-Fedora-AppServer: koji01.rdu3.fedoraproject.org
< Content-Type: text/xml
< 
<?xml version='1.0'?>
<methodResponse>
<params>
<param>
<value><int>1</int></value>
</param>
</params>
</methodResponse>
* Connection #0 to host koji01.rdu3.fedoraproject.org left intact
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host koji01.rdu3.fedoraproject.org:80 was resolved.
* IPv6: (none)
* IPv4: 10.16.169.104
*   Trying 10.16.169.104:80...
^C

Perhaps we’re just overwhelming the backend somehow? Or something else in between throwing a spanner in?
Edited a month ago by nphilipp

This seems all like https://pagure.io/fedora-infrastructure/issue/12814

can you look at all the investigation there?

Looks the same to me. @zlopez, shall we close this as duplicate?

Closing as duplicate of #12814.

Metadata Update from @nphilipp:
- Issue close_status updated to: Duplicate
- Issue status updated to: Closed (was: Open)
a month ago

Log in to comment on this ticket.

Metadata

koji high-gain high-trouble Needs investigation

None
None
Waiting on Assignee
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.