#811 Git lookaside_upload script returns 500 server error
Closed: Fixed with Explanation 2 months ago by arrfab. Opened 2 months ago by jcapitao.

In Cloud SIG, we are hitting same issue as [1]

+ /tmp/scratch_dir/centos-git-common/lookaside_upload -f SOURCES/poetry-core-1.0.8.tar.gz -n python-poetry-core -b c9s-sig-cloud-openstack-zed
[+] CentOS Lookaside upload tool -> Checking if file already uploaded
[+] CentOS Lookaside upload tool -> Initialing new upload to lookaside
[+] CentOS Lookaside upload tool -> URL : https://git.centos.org
[+] CentOS Lookaside upload tool -> Source to upload : SOURCES/poetry-core-1.0.8.tar.gz
[+] CentOS Lookaside upload tool -> Package name: python-poetry-core
[+] CentOS Lookaside upload tool -> sha1sum: 25f17e15925423fca9ac4144fe7d1b0d1675f091
[+] CentOS Lookaside upload tool -> Remote branch: c9s-sig-cloud-openstack-zed
[+] CentOS Lookaside upload tool ->  ====== Trying to upload =======
                         15.8%
#######                  78.9%
####################### 100.0%
curl: (22) The requested URL returned error: 500 Internal Server Error
[+] CentOS Lookaside upload tool -> [ERROR] Something didn't work to push to https://git.centos.org/sources/python-poetry-core/c9s-sig-cloud-openstack-zed/25f17e15925423fca9ac4144fe7d1b0d1675f091
[+] CentOS Lookaside upload tool -> [ERROR] Verify at the server side

[1] https://pagure.io/centos-infra/issue/801


Metadata Update from @arrfab:
- Issue assigned to arrfab

2 months ago

Metadata Update from @arrfab:
- Issue tagged with: centos-common-infra, high-gain, high-trouble

2 months ago

I had a quick look and it's ... amusing ?

[Thu Jun 16 09:50:18.536257 2022] [cgid:error] [pid 4097304:tid 140634565150464] [client 109.89.109.151:52940] requests_gssapi.exceptions.SPNEGOExchangeError: stepping context failed: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529638944): Ticket expired: /var/www/cgi-bin/upload.cgi

So it's using a keytab (through python3-requests-gssapi) but after some time ticket is expired (normal) but while same code was working on python3, it seems here it doesn't renew expired ticket. I just cleared all cache and restart httpd process (under which that cgi script is running) and then it was working again :

[Thu Jun 16 10:04:51.808785 2022] [cgid:error] [pid 4128373:tid 140258263955200] [client <removed_ip>:35530] [username=jcapitao] Processing upload request: NAME=python-poetry-core BRANCH=c9s-sig-cloud-openstack-zed CHECKSUM=25f17e15925423fca9ac4144fe7d1b0d1675f091/srv/cache/lookaside/python-poetry-core/c9s-sig-cloud-openstack-zed/srv/cache/lookaside/python-poetry-core/c9s-sig-cloud-openstack-zed/25f17e15925423fca9ac4144fe7d1b0d1675f091Matched sig-cloud against c9s-sig-cloud-openstack-zedACL validated : /var/www/cgi-bin/upload.cgi
[Thu Jun 16 10:04:51.808869 2022] [cgid:error] [pid 4128373:tid 140258263955200] [client <removed_ip>:35530] [username=jcapitao] Stored /srv/cache/lookaside/python-poetry-core/c9s-sig-cloud-openstack-zed/25f17e15925423fca9ac4144fe7d1b0d1675f091 (414941 bytes): /var/www/cgi-bin/upload.cgi

@jcapitao while you confirmed that it works for you, let's keep that one open for now as I'd like to test tomorrow if that still works without a httpd restart (to force a new kerberos init)
It that works, I'll just close ticket but something to still have a look at

Yes it's working now, thank you @arrfab for troubleshooting this weirdness

I'm also hitting an issue trying to upload sources to lookaside using lookaside_upload_sig.

The exact failing command (part of lookaside_upload_sig) is:
curl "https://git.centos.org/sources/upload_sig.cgi" --fail --cert ~/.centos.cert --form "name=btrfs-progs" --form "hash=sha512" --form "sha512sum=a2dc30d2bc402ea2370fd093b9816e1414c72af5a3184efa8aa5e537bcb8a05b06bfb02cdc43ad2f81b041f58a0682d86742065e6cc60771abf956c7ab37a294" --form "file=@btrfs-progs-v5.14.2.tar.sign" --progress-bar

Error: curl: (22) The requested URL returned error: 500 Internal Server Error

I'm getting the same issue with command:

lookaside_upload -f dpdk-21.11.tar.xz -n openvswitch -b c9s-sig-nfv-openvswitch-2.17

Error:

  • curl https://git.centos.org/sources/upload.cgi --fail --cert /home/amoralej/.centos.cert --form name=openvswitch --form branch=c9s-sig-nfv-openvswitch-2.17 --form sha1sum=17331a86759beba4b6635ed530ce23b0b73c0744 --form file=@dpdk-21.11.tar.xz --progress-bar
  • tee /dev/null
################################################################################# 100.0%curl: (22) The requested URL returned error: 500

That was my fear (reason why I let this ticket open) : now that I restarted httpd, I asked @amoralej on irc and he confirmed that it was working (what I saw directly in the httpd log as no error was showing up anymore about "Ticket Expired").
I can implement a nasty workaround (to restart httpd on a daily basis) so always be under the 24h but still needs to be fixed correctly so ticket will remain open

Just pushed what should be a proper workaround for now, but investigating moving to gssproxy for this would be ideal but further down the road.

Explanations : on el7 (previous deployment) kerberos setup was using keyring but on el8, as sssd-kcm pkg is configuring it, it defaults now to KCM cache, which seems to be the problem for our httpd/cgi script to obtain/renew kerberos ticket when trying to reach https://fasjson.fedoraproject.org. Ensuring now using previous method seems to be working for now (multiple tests) and so shouldn't block SIGs when uploading to lookaside.

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 months ago

Login to comment on this ticket.

Metadata
Boards 1