Is KojiHubCA set in config? I'm thinking about some impact of #2450 or #2448 respectively.

yes, it's set, per documentation.
Also, the https cert exposed on hub isn't signed by that CA, as we have valid cert for https, but the TLS certs used for auth are the ones signed from central IPA auth .
But that's always been the case : https using "official and trusted" CA, while for auth, it's using other auth CA, signing the client certs

@tkopecek happy to apply "hot-patch" on that .dev. instance to validate some theory about the web code that change.
Can even just upgrade whole code to 1.24 and revert web/init to 1.23 to confirm which code is causing us troubles.

@tkopecek I don't mind updating that dev instance against to 1.24, and then only revert /usr/share/koji-web/scripts/index.py to the 1.23 version to confirm that it's indeed the mentioned PR that is the issue ?

Yes, it would be helpful!

I tested the following and confirm that issue is in index.py from koji-web (and python3-koji-web-1.24.1-1.el8.noarch) :

rpm -q koji-web
koji-web-1.23.0-1.el8.noarch
cp /usr/share/koji-web/scripts/index.py /usr/share/koji-web/scripts/index.py.1.23
yum update koji\* ; rpm -q koji-web
koji-web-1.24.1-1.el8.noarch

At this point it has the issue mentioned above, and then :

[root@cbs ~]# cp /usr/share/koji-web/scripts/index.py /usr/share/koji-web/scripts/index.py.1.24
cp: overwrite '/usr/share/koji-web/scripts/index.py.1.24'? y
[Dev - cbs.dev.rdu2.centos.org]
[root@cbs ~]# cp /usr/share/koji-web/scripts/index.py.1.23 /usr/share/koji-web/scripts/index.py
cp: overwrite '/usr/share/koji-web/scripts/index.py'? y

and then it works again .. so is there a way we can just revert the introduced change that actually breaks TLS between web and hub ?

happy to give other details or test something if you want

Ok, I think I understand the problem now (missed that line that KojiHubCA != https CA. Unfortunately we don't support it on client level. Solution is to add kojihubCA as system-wide trusted CA, so KojiHubCA option needn't be used at all.

It is almost the same as #2618 (similar problem on builder, https://pagure.io/koji/pull-request/2724#_1__36)

Can you try this?

Basically my comment here: https://pagure.io/koji/issue/2448#comment-673128
@mikem Do you think it is worth to introduce another option for this (I'm more against it as it is an unusual setup).

Ok, I think I understand the problem now (missed that line that KojiHubCA != https CA. Unfortunately we don't support it on client level. Solution is to add kojihubCA as system-wide trusted CA, so KojiHubCA option needn't be used at all.

It is almost the same as #2618 (similar problem on builder, https://pagure.io/koji/pull-request/2724#_1__36)

Can you try this?

Well, you so mean importing the kojihubCA that signs the certs as trusted by default ? I thought that it was what that line was used for ? and so why suddenly after several years it doesn't work anymore between 1.23 and 1.24 ? as the exposed https cert is already trustead (it's a letsencrypt cert)

This "new" behaviour is in fact regression fix (#2448). If you've had globally non-trusted CA for login and for https in hub, CA was used only for login. Such login would fail on not being able to establish https connection.

Problem is on requests side - you can specify only one certificate bundle there. So, so system-wide for everything or user-specified one. There is no simple way to merge them together. Separate option would be a way to go, but it seems to me easier to have those additional CAs merged into system-wide bundle. Anyway, open to solutions.

so you mean importing the CA that was used for auth (not for https itself) system-wide and then remove the option from web.conf ? I can try that
Worth documenting somewhere after ?

I would try that - if it will work, we definitely should include it in docs in few places.

but what's the diff between ClientCA and KojiHubCA ?

Ouch, ClientCA is dead for years (1.11, #ffcf1a30ebb720dcf721f04630aae171c058ca4f), it stayed in docs/example :-(

ok, so tested then something else : removed completely ClientCA and KojiHubCA from /etc/kojiweb/web.conf and put back in place index.py from 1.24

And it seems to now work. I'll now test normal auth and kojid but if that's working, worth tagging as doc and so removing unneeded option ?

PR #2888

and FWIW, just also using it but pointing directly to KojiHubCA = /etc/pki/tls/certs/ca-bundle.trust.crt works too (and it makes sense) , so it's just the CA that signed the https cert on kojihub that it's verifying, and nothing wrt auth at all
And it makes sense as the real verification for auth is done at the kojihub level itself

@tkopecek feel free to close when you want , as you have PR open for the doc part.
Thanks for the help and fwiw, cbs.centos.org is now running 1.24

Commit 42af695 fixes this issue

Commit a77e340 relates to this ticket