We still have apps that are not yet using OIDC for authentication. It would be nice to add support to them, so we are no longer blocked by this tech debt.
Here is the list of the apps that still needs to be ported: Apps CPE owns and are critical pagure (Already implemented, but needs changes in ipsilon. See https://pagure.io/fedora-infrastructure/issue/7377 for more details) mirrormanager bodhi noggin PDC (PDC will be probably retired in future) FMN
Apps CPE hosts and are critical Mailman3 / HK OSBS * MBS
Apps CPE hosts and aren't critical COPR nuancier * testdays
It's possible that some of them are already ported over to OIDC (especially those we don't host), it needs to be checked.
Not urgent.
FWIW, the upstream bodhi ticket is at:
https://github.com/fedora-infra/bodhi/issues/1180
also, not sure about noggin -- it uses neither openid or openid connect
the MBS readme states that it uses OIDC:
https://pagure.io/fm-orchestrator#setting-up-kerberos-ldap-authentication
or is there something i am missing here?
Ok, some of these are not actaully needed, see inline comments
Describe what you would like us to do: We still have apps that are not yet using OIDC for authentication. It would be nice to add support to them, so we are no longer blocked by this tech debt. Here is the list of the apps that still needs to be ported: Apps CPE owns and are critical pagure (Already implemented, but needs changes in ipsilon. See https://pagure.io/fedora-infrastructure/issue/7377 for more details) mirrormanager bodhi noggin
Here is the list of the apps that still needs to be ported: Apps CPE owns and are critical pagure (Already implemented, but needs changes in ipsilon. See https://pagure.io/fedora-infrastructure/issue/7377 for more details) mirrormanager bodhi noggin
Noggin doesnt use openid or oidc
PDC (PDC will be probably retired in future) FMN Apps CPE hosts and are critical * Mailman3 / HK OSBS OSBS doesnt use openid it seems MBS
Apps CPE hosts and are critical * Mailman3 / HK
MBS uses oidc already apparently
Apps CPE hosts and aren't critical COPR nuancier * testdays doenst use openid or oidc It's possible that some of them are already ported over to OIDC (especially those we don't host), it needs to be checked. When do you need this to be done by? (YYYY/MM/DD) Not urgent.
Apps CPE hosts and aren't critical COPR nuancier * testdays doenst use openid or oidc
Okay, here is the curated list:
OK -- opened a PR for mirrormanager to add OIDC support:
https://github.com/fedora-infra/mirrormanager2/pull/301
PR opened for nuancier here:
https://github.com/fedora-infra/nuancier/pull/136
[backlog refinement] Here is the up to date list:
Bodhi was migrated to OIDC https://github.com/fedora-infra/bodhi/issues/1180
[Backlog refinement] There is a CPE initiative team working on the rewrite of FMN, which should address this issue as well.
testdays
I guess that'd be https://pagure.io/fedora-qa/testdays-web ?
And maybe bba should be added to the list?
blockerbugs (I am planning to port it).
Also, is there a plan to drop basic oauth2? I am Maintaining FAS integration on forum.mojefedora.cz, and it's using oauth plugin instead of oidc (I had really hard time getting oidc work there, so I fell back to oauth).
@frantisekz I don't know about any deadline for dropping basic oauth2, at least not for now.
Here is the up to date list:
Pagure is still sadly blocked. ;(
There's OIDC support in pagure, but ipsilon doesn't support variable scopes that we need to enable it. As far as I know.
Pagure is still sadly blocked. ;( There's OIDC support in pagure, but ipsilon doesn't support variable scopes that we need to enable it. As far as I know.
Wasn't that for the API though? We may be able to migrate the web UI part and leave the API to using the current API token system. Not ideal, but could work
Yeah, that was the api.
[backlog refinement] I'm currently working on https://pagure.io/fedora-infrastructure/issue/10372
Metadata Update from @kevin: - Issue tagged with: blocked
[backlog refinement] Nuancier is being deprecated now - https://pagure.io/fedora-infrastructure/issue/11371 PDC is being deprecated as initiative Pagure is being ported to RHEL 9 (this should solve the OIDC issue) COPR implemented https://github.com/fedora-copr/copr/issues/2422 Mailman 3/HK are being updated which should solve the OIDC support as well
Nuancier is now decommissioned.
FYI, kerneltest is still using openid also. ;)
[Update] Mailman 3/HK is now authenticating using OIDC. PDC is decommissioned
[Update] Currently working on OIDC support for release-monitoring.org. Pagure is almost ported, only outage is needed COPR is still missing the support, the change is already merged, but not deployed @frostyx @praiskup Could you look into that?
There's a tracker for this in Copr https://github.com/fedora-copr/copr/issues/3483
The release-monitoring.org is now using OIDC.
If there are still a number of things not ported over I recommend we drop those non critical items and open specific tickets for those that will be worked on. I fear this could stay open for another 3 years in it's current state.
I think all we are waiting on now is copr deployment... then setting a sunset date and perhaps trying to notify anyone we can identify thats still using it.
Found another one today blockerbugs, they deployment is still accessing https://id.fedoraproject.org/openid/
https://id.fedoraproject.org/openid/
@kparal ^^^
I've opened a ticket there to track it: https://pagure.io/fedora-qa/blockerbugs/issue/288
What is our expected deadline? 2 weeks? 1 month?
It looks like the COPR one is done ... is that accurate?
@smilner Yes, @abompard help COPR team to move this forward.
Metadata Update from @zlopez: - Issue untagged with: blocked
Here is the plan for moving this forward:
So here are the domains using OpenID for authentication I found in the logs:
We should now decide the sunset date for OpenID and let them know that they should migrate.
Thanks for finding these @zlopez. Is this something that will come up in the next infra meeting to discuss date and communication?
Yeah, we could discuss then. How about a time right around when Fedora 40 goes EOL (but not the exact same day)?
That's roughly 3 months from now. That seems like a reasonable susnet time frame to me!
+1 from me for the F40 EOL time.
Found out that https://lists.pagure.io/ is already using OIDC, somebody was just trying to use openid endpoint with it, but it wouldn't work anymore. So we can ignore that one.
The EOL date for OpenID was decided to be 2025-05-20 on Fedora weekly meeting.
I'm preparing an announcement on Fedora community blog and here is the draft for review https://communityblog.fedoraproject.org/?p=14508&preview=1&_ppp=6a0c6ffb3f
I've written a small parser script for the Ipsilon apache logs and ran it on ipsilon01 to identify which apps still use OpenID, and I'm getting the following results:
abrt.fedoraproject.org 2 copr.fedorainfracloud.org 5269 faf.lab.eng.brq2.redhat.com 1 listat.jyps.fi 33 lists.ncf.ca 471 lists.openldap.org 1148 lists.ovirt.org 8 lists.podman.io 148 openqa.fedoraproject.org 7 openqa.stg.fedoraproject.org 6 retrace.fedoraproject.org 28 retrace03.rdu-cc.fedoraproject.org 2
The second column is the number of time the hostname is using openid. This is on logfiles that date back to March 8th, so about a week ago.
Thanks for the list, that really helps. I found only the mailing lists in the recent ones. I noticed the COPR, they still have both option and OpenID is really the default. But we can ignore that as they already moved.
I checked quickly abrt.fedoraproject.org and it fails with net::ERR_CERT_COMMON_NAME_INVALID error and the cert itself points to retrace03.rdu-cc.fedoraproject.org, so I assume those two are connected.
abrt.fedoraproject.org
net::ERR_CERT_COMMON_NAME_INVALID
retrace03.rdu-cc.fedoraproject.org
I will publish the announcement next week and after that start contacting the projects.
the retrace.fedoraproject.org and abrt.fedoraproject.org should be the same machine... that retrace03.rdu-cc.fedoraproject.org box. We host it, but don't normally manage it. We should make sure it's moved if we can...
CC: @msuchy
I defer my cc to @msrb
The blog post announcement is scheduled for Thursday. After that I will also sent it to Fedora Infra and Devel Announce. So people are aware.
And start contacting the projects.
I saw the Community Blog article. Does this have any impact on Libravatar? I use the Open ID option there to authenticate and log in. I'm wondering if I will lose the ability to log into my Libravatar account because of this?
Yes, it could affect them. ;(
@oliver thoughts?
abrt.fedoraproject.org seems like an alias for retrace.fedoraproject.org, and I didn't even know about it. In any case, abrt.fedoraproject.org doesn't seem to be used, otherwise we would have discovered the problem much earlier (I assume).
retrace.fedoraproject.org
Yeah, but retrace is still using openid I guess? at least I don't see any OIDC config for it...
I sent the announcement mails to devel-announce and infra mailing lists as well, so it reaches more people.
So I tried to contact out the mailing lists we identified using the Fedora authentication option to log in and here is the result:
I tried to login to faf.lab.eng.brq2.redhat.com and it works. It seems to be some ABRT Analytics instance, is this something related to abrt.fedoraproject.org?
for openQA, sorry, I had overlooked this, for some reason I thought we were using OIDC. I'll try and look into switching it today.
That's ok! We made a good attempt, said we're making the change publicly, and this ticket has been open for 3 years. I believe we've made every reasonable attempt to notify folks.
Filed https://pagure.io/fedora-infrastructure/issue/12459 for openQA. We might need a bit of work on the config side of things here and possibly a bit of patching to openQA to make it possible to use the underlying Mojolicious plugin's OIDC 'mode' (specifically, I think we need to make it possible to set well_known_url in the custom hash instead of authorize_url and token_url.)
well_known_url
custom
authorize_url
token_url
I'm a bit worried we might have to do more work on the openQA side too, based on notes in the mojolicious plugin doc like "When Mojolicious::Plugin::OAuth2 is being used in OpenID Connect mode this helper allows you to decode the response data encoded with the JWKS discovered from well_known_url configuration." Unless Ipsilon can act more like the kind of oauth2 provider the current openQA implementation is set up to expect?
@smilner yeah, it's certainly my fault, I was fully aware of the change, it was just somehow in my head that openQA was already using OIDC, no idea why.
@adamwill My post was meant as a response for @zlopez on his work to try to directly notify folks but I failed to tag him directly. But still, no worries, we have a lot of stuff and it happens!
Currently waiting for magazine article to go live as well. So we can reach more folks.
Metadata Update from @zlopez: - Issue assigned to zlopez
I got openqa stg/lab more or less working today (just need to ansiblize the config now I know what works). will try and get prod moved over tomorrow.
The Fedora magazine article is now public.
OK, I've converted both lab and prod openQA now.
We are still missing:
Discussed the services with user defined OpenID endpoint (for example https://www.libravatar.org/openid/login/) today on stand up with @abompard and he had a great idea to keep one instance of ipsilon running with OpenID enabled that we will redirect any OpenID request to, but add warning to header to let people know that they should move to something else.
That would allow us to replace ipsilon with something else, but still allow people to login with OpenID for some time. We will just need to decide for how long we would like to keep this instance running.
Yeah, there's a HTML header that can be added to a domain's landing page to redirect clients to another OpenID provider, we could use that: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/YAANRQAEPRT2PFSWJBK2CD2RBZTIQ3UT/
I suppose we could do this... I'm not sure overjoyed to keep running ipsilon once we move away from it, but I agree it would be good to give a longer runway here.
If we do run an ipsilon service let's ensure we have a hard decom date and stick to it. Is it fair to say that it would be a "best effort, low priority" service? For timing I'd recommend not more than 6 months but ideally less.
I agree with the 6 months for this to be running and just put a big red header on the page saying that the OpenID will stop working on that date.
In the meantime we can replace the main ipsilon instance with whatever we want.
Yes, it could affect them. ;( @oliver thoughts?
Again, appreciate the nudge @kevin ! Yes, most of the Fedora Users using Libravatar also use OpenID to sign in, so this has a larger impact on that group of users.
I see the following topics: - OpenID Login to Libravatar - Fetching Libravatar-Images using OpenIDs - Implementing OIDC in Libravatar - Automatic (?) migration of the Fedora-Users who use OpenID to login currently
In no particular order.
So, first of all: Implementing OIDC in Libravatar could be quite some effort, esp. because I haven't done any OIDC implementation yet. The code is F/LOSS and available here: https://git.linux-kernel.at/oliver/ivatar - if anyone wants to take a stab at implementing it, I'm open to that.
Once we have implmeneted OIDC (and connected Fedora), does anyone have an idea how we could potentially automate the migration? Again, I've never been in that situation before.
Just to give you some data to better understand the impact.
libravatar=# select count(*) from django_openid_auth_useropenid where claimed_id like '%fedoraproject%'; count ------- 6205 (1 row) libravatar=# select count(*) from django_openid_auth_useropenid; count ------- 6670 (1 row)
Conclusion: 93% of the users using OpenID are in fact Fedora(project) users. :-(
@oliver I see you're already using python-social-auth's Django integration, it shouldn't be too hard to enable the OIDC module, I'll see if I can get it to work.
@abompard This will probably need some database changes as well. As the OpenID username is usually different than the OIDC one.
I'm thinking about trying to do a test deployment of ipsilon on staging with keeping one instance with OIDC support only and redirecting all the OpenID requests to other one with the injected header. What do you think about that? Should I wait till the freeze end with that?
Yes, social auth is in use and good point about potentially reusing it. Thanks for offering a helping hand!!
Log in to comment on this ticket.