As communishift is now up and announced we should create an ansible playbook to do the cleaning and update the post release docs, so it's not forgotten.
Not urgent, but sooner the better
@zlopez do you have an example of cleaning the post release doc ? Anyway, I can work on it , but need clarification from someone who worked on it .
@kevin Do we have some docs for post release steps that needs to be done by Fedora Infra or is this part of release engineering guide?
Really the 'cleaning' here should just be 'oc delete project/$i' for all projects (that are not openshift-*/system ones)
Perhaps we could enhance the existing playbook for communishift to remove the projects instead of adding them, then you run that and remove them from the list in git.
We do have https://docs.fedoraproject.org/en-US/infra/release_guide/final_release/ but perhaps we could add a 'post final release' one?
@kevin I agree that adding post final release guide makes sense as this shouldn't be bothering people during release.
post final release
After DevConf.CZ and discussion happening there, I would like to add that part of this should be also some announcements sent to contacts for communishift projects that will let owners know the post release cleaning is coming. This could happen a month before the release and then again 14 days, so owners have time to react.
Metadata Update from @dkirwan: - Issue assigned to dkirwan
I saw this get built by someone at my university some years back, so maybe its not too hard to adapt to the communishift cluster and infra: https://github.com/WillNilges/ShelfLife
looks like it was also built for an openshift cluster and i believe it was also set up to notify people via email when their stuff was about to expire.
As far as scheduling when things get deleted, i think it may be nice to reuse the same date as the EOL-ing of F[N-2] (so 2024-11-12 for F41, see https://fedorapeople.org/groups/schedule/f-41/f-41-key-tasks.html) to give users an extra heads up well in advance (i.e. the user docs could document that the deletion date is around the time of the EOLing of the fedora release).
That said, an arbitrary date like that that may change every cycle may be a moving target thats hard to pin down and automate things against.
Metadata Update from @zlopez: - Issue tagged with: communishift
We will have this as part of Fedora Infra hackfest to discuss when and how to delete the project. The delete with EOL date makes sense, but how much before it we should notify owners of the project. And all of this should be documented as well.
For now we have two requests for not deleting the projects:
As Fedora 39 EOL is coming near, we should move this forward.
@dkirwan Did you have time to work on this?
@zlopez when this is needed can you assign this task of clean the cluster? I'll go through what ever is involved manually, add an ansible task and gather all the moving parts into a SOP.
@dkirwan What task do you mean?
The SOP would be great, we also need to let owners of the projects know about that each cycle of cleaning. So they can react in time.
Looking at adding a playbook to retrieve project admin users emails from each project then send them a notification email about incoming project deletions.
Considering the following template:
to: "{{ item.value.comma_separated_admin_email_list}}" from: "admin@fedoraproject.org" subject: "Fedora Communishift Notification: {{ item.value.name }}" body: "Dear Administrator, This is a reminder that the Communishift project {{ item.value.name }} will be deleted during the Fedora post release process. Please ensure you have a backup of any important configuration or data from your project. To request an exemption for project deletion please open a ticket here: https://pagure.io/fedora-infrastructure/issues"
Might make it more discoverable when the date is?
I guess we decided to do this when EOLing the old fedora release. So, in this case it's 2024-11-26 that F39 goes eol. We could put in the date, or just try and say something like 'at the same time the oldest fedora is retired (1 month after a new fedora release)" or something?
I also wonder, perhaps we should make a ticket in advance and just tell people to add to that one ticket instead of filing a bunch of them? ("To request an exemption for project deletion please add your request to https://pagure.io/fedora-infrastructure/issue/12345678")
Otherwise looks good.
I like the idea of having one ticket created in advance, we can close it once the cleanup is done.
<img alt="Screenshot_from_2024-11-21_16-09-45.png" src="/fedora-infrastructure/issue/raw/files/bbbb7cb2d9a361549622783b40601d1ae073bf23d1bd398636dd2e950f621127-Screenshot_from_2024-11-21_16-09-45.png" />
Updated the communishfit role, and added a new playbook. Can retrieve the communishift projects plus the email addresses of the admins from fasjson. Also have task that can then send the email template in ansible.
Final piece of work to update the To field to point at these admin email addresses.
Awesome. I think at this point we should give them more time (since next week is a holiday in the us and not much notice), but perhaps sometime in mid december?
Did anything happen here?
I guess we should re-notify people again and see what would be cleaned up?
From memory only a single tenant responded in anyway to request the projects be exempted, might need to rethink the auto deleting part entirely. Would feel happier if that was done manually, fairly easy to just oc delete project etc.. especially as it will require extra steps such as disabling/removing FAS groups as not sure disabling them will prevent fasjson returning the data, must test that.
Can do another run of the notifications steps, just let me know when we want to do the actual deleting steps, I can make sure everything is properly documented in an SOP.
The F42 release is already out and F40 EOL, so we can do the delete anytime.
Do I understand it correctly that only one tenant didn't want the project to be deleted?
Yeah only one tenant responded, that might mean the rest marked the reminder email read and never actually read it/saw it.
Perhaps before deleting, shut any services contained within down, and see if anyone complains?
Doing a scream test is not a bad idea. Let's disable the project for a week and see if there will be any response first and then delete it.
We can make that part of the official process, so there is still a time for folks to reach out.
This is a really good idea. Please make this a part of the official process.
Not sure if there is a public list of exceptions somewhere in playbooks and not sure if the one tenant was me. So please, just don't remove these two:
At this point I think re-notifying people might be in order... but yeah, if we have a way to just 'suspend' a project for a while where we could easily just re-enable it if someone comes asking could be a better way to do things.
https://pagure.io/fedora-infra/ansible/blob/main/f/inventory/group_vars/all#_42
@frostyx Yes we mark the fact projects should not be deleted here. I believe both your projects are marked correctly.
Not sure if there is a public list of exceptions somewhere in playbooks and not sure if the one tenant was me. So please, just don't remove these two: https://pagure.io/fedora-infrastructure/issue/11868 https://pagure.io/fedora-infrastructure/issue/11814
Log in to comment on this ticket.