#10241 Port apps to OIDC
Opened 3 years ago by zlopez. Modified 19 days ago

Describe what you would like us to do:


We still have apps that are not yet using OIDC for authentication. It would be nice to add support to them, so we are no longer blocked by this tech debt.

Here is the list of the apps that still needs to be ported:
Apps CPE owns and are critical
pagure (Already implemented, but needs changes in ipsilon. See https://pagure.io/fedora-infrastructure/issue/7377 for more details)
mirrormanager
bodhi
noggin
PDC (PDC will be probably retired in future)
FMN

Apps CPE hosts and are critical
Mailman3 / HK
OSBS
* MBS

Apps CPE hosts and aren't critical
COPR
nuancier
* testdays

It's possible that some of them are already ported over to OIDC (especially those we don't host), it needs to be checked.

When do you need this to be done by? (YYYY/MM/DD)


Not urgent.


also, not sure about noggin -- it uses neither openid or openid connect

the MBS readme states that it uses OIDC:

https://pagure.io/fm-orchestrator#setting-up-kerberos-ldap-authentication

or is there something i am missing here?

Ok, some of these are not actaully needed, see inline comments

Describe what you would like us to do:


We still have apps that are not yet using OIDC for authentication. It would be nice to add support to them, so we are no longer blocked by this tech debt.

Here is the list of the apps that still needs to be ported:
Apps CPE owns and are critical
pagure (Already implemented, but needs changes in ipsilon. See https://pagure.io/fedora-infrastructure/issue/7377 for more details)
mirrormanager
bodhi
noggin

Noggin doesnt use openid or oidc

  • PDC (PDC will be probably retired in future)
  • FMN

Apps CPE hosts and are critical
* Mailman3 / HK

  • OSBS
    OSBS doesnt use openid it seems
  • MBS

MBS uses oidc already apparently

Apps CPE hosts and aren't critical
COPR
nuancier
* testdays
doenst use openid or oidc

It's possible that some of them are already ported over to OIDC (especially those we don't host), it needs to be checked.

When do you need this to be done by? (YYYY/MM/DD)


Not urgent.

Okay, here is the curated list:


[backlog refinement]
Here is the up to date list:

[Backlog refinement]
There is a CPE initiative team working on the rewrite of FMN, which should address this issue as well.

I guess that'd be https://pagure.io/fedora-qa/testdays-web ?

And maybe bba should be added to the list?

blockerbugs (I am planning to port it).

Also, is there a plan to drop basic oauth2? I am Maintaining FAS integration on forum.mojefedora.cz, and it's using oauth plugin instead of oidc (I had really hard time getting oidc work there, so I fell back to oauth).

@frantisekz I don't know about any deadline for dropping basic oauth2, at least not for now.

Here is the up to date list:


Pagure is still sadly blocked. ;(

There's OIDC support in pagure, but ipsilon doesn't support variable scopes that we need to enable it.
As far as I know.

Pagure is still sadly blocked. ;(

There's OIDC support in pagure, but ipsilon doesn't support variable scopes that we need to enable it.
As far as I know.

Wasn't that for the API though? We may be able to migrate the web UI part and
leave the API to using the current API token system. Not ideal, but could work

Yeah, that was the api.

Metadata Update from @kevin:
- Issue tagged with: blocked

2 years ago

[backlog refinement]
Nuancier is being deprecated now - https://pagure.io/fedora-infrastructure/issue/11371
PDC is being deprecated as initiative
Pagure is being ported to RHEL 9 (this should solve the OIDC issue)
COPR implemented https://github.com/fedora-copr/copr/issues/2422
Mailman 3/HK are being updated which should solve the OIDC support as well

Nuancier is now decommissioned.

FYI, kerneltest is still using openid also. ;)

[Update]
Mailman 3/HK is now authenticating using OIDC.
PDC is decommissioned

[Update]
Currently working on OIDC support for release-monitoring.org.
Pagure is almost ported, only outage is needed
COPR is still missing the support, the change is already merged, but not deployed @frostyx @praiskup Could you look into that?

The release-monitoring.org is now using OIDC.

If there are still a number of things not ported over I recommend we drop those non critical items and open specific tickets for those that will be worked on. I fear this could stay open for another 3 years in it's current state.

I think all we are waiting on now is copr deployment... then setting a sunset date and perhaps trying to notify anyone we can identify thats still using it.

Found another one today blockerbugs, they deployment is still accessing https://id.fedoraproject.org/openid/

Found another one today blockerbugs, they deployment is still accessing https://id.fedoraproject.org/openid/

@kparal ^^^

What is our expected deadline? 2 weeks? 1 month?

It looks like the COPR one is done ... is that accurate?

@smilner Yes, @abompard help COPR team to move this forward.

Metadata Update from @zlopez:
- Issue untagged with: blocked

2 months ago

Here is the plan for moving this forward:

  1. Identify projects that are using openid endpoint on ipsilon
  2. Announce sunset date
  3. Reach to the remaining projects and ask them to migrate (help them with migration if needed)
  4. (After reaching the sunset date) Disable openid on ipsilon

So here are the domains using OpenID for authentication I found in the logs:

We should now decide the sunset date for OpenID and let them know that they should migrate.

Thanks for finding these @zlopez. Is this something that will come up in the next infra meeting to discuss date and communication?

Yeah, we could discuss then. How about a time right around when Fedora 40 goes EOL (but not the exact same day)?

That's roughly 3 months from now. That seems like a reasonable susnet time frame to me!

+1 from me for the F40 EOL time.

Found out that https://lists.pagure.io/ is already using OIDC, somebody was just trying to use openid endpoint with it, but it wouldn't work anymore. So we can ignore that one.

The EOL date for OpenID was decided to be 2025-05-20 on Fedora weekly meeting.

I'm preparing an announcement on Fedora community blog and here is the draft for review https://communityblog.fedoraproject.org/?p=14508&preview=1&_ppp=6a0c6ffb3f

I've written a small parser script for the Ipsilon apache logs and ran it on ipsilon01 to identify which apps still use OpenID, and I'm getting the following results:

abrt.fedoraproject.org 2
copr.fedorainfracloud.org 5269
faf.lab.eng.brq2.redhat.com 1
listat.jyps.fi 33
lists.ncf.ca 471
lists.openldap.org 1148
lists.ovirt.org 8
lists.podman.io 148
openqa.fedoraproject.org 7
openqa.stg.fedoraproject.org 6
retrace.fedoraproject.org 28
retrace03.rdu-cc.fedoraproject.org 2

The second column is the number of time the hostname is using openid. This is on logfiles that date back to March 8th, so about a week ago.

  • Copr is still defaulting to OpenID (if I understand correctly) which explains the high count.
  • We should probably contact the Mailman/HyperKitty instances in this list to ask them to switch the Fedora provider to OIDC.
  • Other apps need to be investigated as well I think.

Thanks for the list, that really helps. I found only the mailing lists in the recent ones. I noticed the COPR, they still have both option and OpenID is really the default. But we can ignore that as they already moved.

I checked quickly abrt.fedoraproject.org and it fails with net::ERR_CERT_COMMON_NAME_INVALID error and the cert itself points to retrace03.rdu-cc.fedoraproject.org, so I assume those two are connected.

I will publish the announcement next week and after that start contacting the projects.

the retrace.fedoraproject.org and abrt.fedoraproject.org should be the same machine... that retrace03.rdu-cc.fedoraproject.org box. We host it, but don't normally manage it. We should make sure it's moved if we can...

CC: @msuchy

The blog post announcement is scheduled for Thursday. After that I will also sent it to Fedora Infra and Devel Announce. So people are aware.

And start contacting the projects.

I saw the Community Blog article. Does this have any impact on Libravatar? I use the Open ID option there to authenticate and log in. I'm wondering if I will lose the ability to log into my Libravatar account because of this?

Yes, it could affect them. ;(

@oliver thoughts?

I checked quickly abrt.fedoraproject.org and it fails with net::ERR_CERT_COMMON_NAME_INVALID error and the cert itself points to retrace03.rdu-cc.fedoraproject.org, so I assume those two are connected.

abrt.fedoraproject.org seems like an alias for retrace.fedoraproject.org, and I didn't even know about it. In any case, abrt.fedoraproject.org doesn't seem to be used, otherwise we would have discovered the problem much earlier (I assume).

Yeah, but retrace is still using openid I guess? at least I don't see any OIDC config for it...

I sent the announcement mails to devel-announce and infra mailing lists as well, so it reaches more people.

So I tried to contact out the mailing lists we identified using the Fedora authentication option to log in and here is the result:

  • https://lists.ncf.ca/ - 502 Bad Gateway when redirecting back (bad callback, so it doesn't work anyway). I don't think it makes sense to even notify them as it seems they don't use it anyway.
  • https://lists.ovirt.org/ - I subscribed to infra@ovirt.org, but my subscription is pending.
  • https://lists.openldap.org/ - 502 Bad Gateway when redirecting back (bad callback, so it doesn't work anyway). I don't think it makes sense to even notify them as it seems they don't use it anyway.
  • http://listat.jyps.fi/ - I subscribed to ohjaajat-lista@jyps.fi, but my subscription is pending.
  • https://lists.podman.io/ - I subscribed to podman@lists.podman.io, but my subscription is pending.

I tried to login to faf.lab.eng.brq2.redhat.com and it works. It seems to be some ABRT Analytics instance, is this something related to abrt.fedoraproject.org?

for openQA, sorry, I had overlooked this, for some reason I thought we were using OIDC. I'll try and look into switching it today.

That's ok! We made a good attempt, said we're making the change publicly, and this ticket has been open for 3 years. I believe we've made every reasonable attempt to notify folks.

Filed https://pagure.io/fedora-infrastructure/issue/12459 for openQA. We might need a bit of work on the config side of things here and possibly a bit of patching to openQA to make it possible to use the underlying Mojolicious plugin's OIDC 'mode' (specifically, I think we need to make it possible to set well_known_url in the custom hash instead of authorize_url and token_url.)

I'm a bit worried we might have to do more work on the openQA side too, based on notes in the mojolicious plugin doc like "When Mojolicious::Plugin::OAuth2 is being used in OpenID Connect mode this helper allows you to decode the response data encoded with the JWKS discovered from well_known_url configuration." Unless Ipsilon can act more like the kind of oauth2 provider the current openQA implementation is set up to expect?

@smilner yeah, it's certainly my fault, I was fully aware of the change, it was just somehow in my head that openQA was already using OIDC, no idea why.

@adamwill My post was meant as a response for @zlopez on his work to try to directly notify folks but I failed to tag him directly. But still, no worries, we have a lot of stuff and it happens!

Currently waiting for magazine article to go live as well. So we can reach more folks.

Metadata Update from @zlopez:
- Issue assigned to zlopez

a month ago

I got openqa stg/lab more or less working today (just need to ansiblize the config now I know what works). will try and get prod moved over tomorrow.

The Fedora magazine article is now public.

OK, I've converted both lab and prod openQA now.

We are still missing:

  • retrace.fedoraproject.org - @msrb as this is abrt.fedoraproject.org alias do you know what needs to be done here?
  • listat.jyps.fi - I reached to them, but didn't get any reply yet
  • lists.ncf.ca - We can ignore that as it throws 502 when trying to use the login option even with OpenID
  • lists.openldap.org - We can ignore that as it throws 502 when trying to use the login option even with OpenID
  • lists.ovirt.org - I reached to them, but didn't get any reply yet
  • lists.podman.io - We can ignore that as it throws 502 when trying to use the login option even with OpenID

Discussed the services with user defined OpenID endpoint (for example https://www.libravatar.org/openid/login/) today on stand up with @abompard and he had a great idea to keep one instance of ipsilon running with OpenID enabled that we will redirect any OpenID request to, but add warning to header to let people know that they should move to something else.

That would allow us to replace ipsilon with something else, but still allow people to login with OpenID for some time. We will just need to decide for how long we would like to keep this instance running.

Yeah, there's a HTML header that can be added to a domain's landing page to redirect clients to another OpenID provider, we could use that: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/YAANRQAEPRT2PFSWJBK2CD2RBZTIQ3UT/

I suppose we could do this... I'm not sure overjoyed to keep running ipsilon once we move away from it, but I agree it would be good to give a longer runway here.

If we do run an ipsilon service let's ensure we have a hard decom date and stick to it. Is it fair to say that it would be a "best effort, low priority" service? For timing I'd recommend not more than 6 months but ideally less.

I agree with the 6 months for this to be running and just put a big red header on the page saying that the OpenID will stop working on that date.

In the meantime we can replace the main ipsilon instance with whatever we want.

Yes, it could affect them. ;(

@oliver thoughts?

Again, appreciate the nudge @kevin !
Yes, most of the Fedora Users using Libravatar also use OpenID to sign in, so this has a larger impact on that group of users.

I see the following topics:
- OpenID Login to Libravatar
- Fetching Libravatar-Images using OpenIDs
- Implementing OIDC in Libravatar
- Automatic (?) migration of the Fedora-Users who use OpenID to login currently

In no particular order.

So, first of all: Implementing OIDC in Libravatar could be quite some effort, esp. because I haven't done any OIDC implementation yet. The code is F/LOSS and available here: https://git.linux-kernel.at/oliver/ivatar - if anyone wants to take a stab at implementing it, I'm open to that.

Once we have implmeneted OIDC (and connected Fedora), does anyone have an idea how we could potentially automate the migration? Again, I've never been in that situation before.

Just to give you some data to better understand the impact.

libravatar=# select count(*) from django_openid_auth_useropenid where claimed_id like '%fedoraproject%';
 count
-------
  6205
(1 row)

libravatar=# select count(*) from django_openid_auth_useropenid;
 count
-------
  6670
(1 row)

Conclusion: 93% of the users using OpenID are in fact Fedora(project) users. :-(

@oliver I see you're already using python-social-auth's Django integration, it shouldn't be too hard to enable the OIDC module, I'll see if I can get it to work.

@abompard This will probably need some database changes as well. As the OpenID username is usually different than the OIDC one.

I'm thinking about trying to do a test deployment of ipsilon on staging with keeping one instance with OIDC support only and redirecting all the OpenID requests to other one with the injected header. What do you think about that? Should I wait till the freeze end with that?

@oliver I see you're already using python-social-auth's Django integration, it shouldn't be too hard to enable the OIDC module, I'll see if I can get it to work.

Yes, social auth is in use and good point about potentially reusing it. Thanks for offering a helping hand!!

Log in to comment on this ticket.

Metadata
Boards 2
dev Status: Backlog
mini-initative Status: Backlog