I had a quick look and it's ... amusing ?

[Thu Jun 16 09:50:18.536257 2022] [cgid:error] [pid 4097304:tid 140634565150464] [client 109.89.109.151:52940] requests_gssapi.exceptions.SPNEGOExchangeError: stepping context failed: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529638944): Ticket expired: /var/www/cgi-bin/upload.cgi

So it's using a keytab (through python3-requests-gssapi) but after some time ticket is expired (normal) but while same code was working on python3, it seems here it doesn't renew expired ticket. I just cleared all cache and restart httpd process (under which that cgi script is running) and then it was working again :

[Thu Jun 16 10:04:51.808785 2022] [cgid:error] [pid 4128373:tid 140258263955200] [client <removed_ip>:35530] [username=jcapitao] Processing upload request: NAME=python-poetry-core BRANCH=c9s-sig-cloud-openstack-zed CHECKSUM=25f17e15925423fca9ac4144fe7d1b0d1675f091/srv/cache/lookaside/python-poetry-core/c9s-sig-cloud-openstack-zed/srv/cache/lookaside/python-poetry-core/c9s-sig-cloud-openstack-zed/25f17e15925423fca9ac4144fe7d1b0d1675f091Matched sig-cloud against c9s-sig-cloud-openstack-zedACL validated : /var/www/cgi-bin/upload.cgi
[Thu Jun 16 10:04:51.808869 2022] [cgid:error] [pid 4128373:tid 140258263955200] [client <removed_ip>:35530] [username=jcapitao] Stored /srv/cache/lookaside/python-poetry-core/c9s-sig-cloud-openstack-zed/25f17e15925423fca9ac4144fe7d1b0d1675f091 (414941 bytes): /var/www/cgi-bin/upload.cgi

@jcapitao while you confirmed that it works for you, let's keep that one open for now as I'd like to test tomorrow if that still works without a httpd restart (to force a new kerberos init)
It that works, I'll just close ticket but something to still have a look at

Yes it's working now, thank you @arrfab for troubleshooting this weirdness

I'm also hitting an issue trying to upload sources to lookaside using lookaside_upload_sig.

The exact failing command (part of lookaside_upload_sig) is:
curl "https://git.centos.org/sources/upload_sig.cgi" --fail --cert ~/.centos.cert --form "name=btrfs-progs" --form "hash=sha512" --form "sha512sum=a2dc30d2bc402ea2370fd093b9816e1414c72af5a3184efa8aa5e537bcb8a05b06bfb02cdc43ad2f81b041f58a0682d86742065e6cc60771abf956c7ab37a294" --form "file=@btrfs-progs-v5.14.2.tar.sign" --progress-bar

Error: curl: (22) The requested URL returned error: 500 Internal Server Error

I'm getting the same issue with command:

lookaside_upload -f dpdk-21.11.tar.xz -n openvswitch -b c9s-sig-nfv-openvswitch-2.17

Error:

• curl https://git.centos.org/sources/upload.cgi --fail --cert /home/amoralej/.centos.cert --form name=openvswitch --form branch=c9s-sig-nfv-openvswitch-2.17 --form sha1sum=17331a86759beba4b6635ed530ce23b0b73c0744 --form file=@dpdk-21.11.tar.xz --progress-bar
• tee /dev/null

################################################################################# 100.0%curl: (22) The requested URL returned error: 500

That was my fear (reason why I let this ticket open) : now that I restarted httpd, I asked @amoralej on irc and he confirmed that it was working (what I saw directly in the httpd log as no error was showing up anymore about "Ticket Expired").
I can implement a nasty workaround (to restart httpd on a daily basis) so always be under the 24h but still needs to be fixed correctly so ticket will remain open

Just pushed what should be a proper workaround for now, but investigating moving to gssproxy for this would be ideal but further down the road.

Explanations : on el7 (previous deployment) kerberos setup was using keyring but on el8, as sssd-kcm pkg is configuring it, it defaults now to KCM cache, which seems to be the problem for our httpd/cgi script to obtain/renew kerberos ticket when trying to reach https://fasjson.fedoraproject.org. Ensuring now using previous method seems to be working for now (multiple tests) and so shouldn't block SIGs when uploading to lookaside.