#99 AMI lifetimes (Cloud WG members vote needed)
Closed None Opened 6 years ago by oddshocks.

This ticket is to discuss and vote on appropriate lifetimes for Fedora Cloud AMIs on our official and community cloud accounts, as discussed on the Cloud SIG list this month1 and in this week's meeting.

Everyone seems to agree that after a final release, all TC, RC, Alpha, Beta, and scratch builds should be deleted. This makes sense because there'd be no reason to use any of these AMIs after the final release. Still, we need to definitively decide on:

  1. When should TC, RC, Alpha, Beta, and scratch builds be deleted? (1) After a final release? (2) Progressively, when a newer build of the same type comes out? (3) After a set amount of time, depending on the build type? Note that with (2) or (3), we'd need some way to mark each build with it's type. I don't think that necessarily happens now.

  2. When should final release AMIs be deleted? (1) After a certain number of newer final releases? (2) After a certain amount of time? (3) Never?

Also note that if we choose any age-based deletion policies, we'd need to set up some sort of regular polling of all our AMIs on both accounts.

We want to hold as few AMIs at one time as is reasonable. There are many AMIs created for each image build that happens, so our AWS storage charges will become quite high if we don't keep our accounts clean. Discuss, and then perhaps a vote?

1) let's delete after final release (1)
2) IMO, we should delete some period after support for that release ends. Let's check with FeSCO to make sure that doesn't violate any policies, though?

I think it would be good to get rid of all non-final (testing) AMIs once the next release comes out.
-> Delete F21 alpha/beta etc AMIs once F22 comes out.

Keeping them around for at least some time after final release could be useful. For example, some bug manifests itself in F21 after a month of release. We could use the TCs to determine if it was there in beta/alpha or not and improvie our process to catch a similar bug next time. Maybe this is dumb but it could be useful information at some point.

For final release AMIs I think we can delete them after a period of time.

1) I think pre-release builds should be deleted a few weeks after being created. This lets someone try an earlier build for troubleshooting purposes, but keeps the lifetime of alphas, betas, TCs, etc. to a minimum.

2) AMIs for final releases should be deleted at the time that the particular Fedora release goes EOL. For example, when Fedora 20 goes EOL, the AMI should get deleted at that same time.

Replying to [comment:3 jsmith]:

2) AMIs for final releases should be deleted at the time that the particular Fedora release goes EOL. For example, when Fedora 20 goes EOL, the AMI should get deleted at that same time.

I don't know if I agree. I'd much prefer keeping final release AMIs around for a while (maybe not in all regions, but at least 1) for longer than that. The "cloud" can be quite useful for spinning up and debugging something on an older release to see how the behavior differs from how you remember it behaving and from how it behaves today. I'd compare it to using a live cd to do the same thing.

One example would be that we find bug in kernel Z and want to quickly spin up older versions of Fedora to see how far it goes back. I think the cloud is an interesting use case for being able to test these things out quickly rather than having to find where the images are stored and figure out all the options needed to start it locally.

  1. => (3) I agree with Dusty on that point but just consider keeping it in all region, because his point also works for third-party service providers and software editors who would like to do their testing in a more local region.
  2. => (1) If we want to remain interesting for end-users, then, we shouldn't delete even EOL'ed releases too soon. What's preferrable would be
    a) communicating on lifecycle of images.
    b) having at startup, a message indicating to users that this image is no longer supported and that it will be deleted at some point.
    Let's allow end-users to prepare the transition to a newer image, just remember that our current lifecycle (~13 months) is too short for entreprises.

OK, it's been a while. Let's close the book on this. There are a lot of opinions, so I'll boil it down to a few points that folks can vote on:

  1. Final AMIs should be deleted when that Fedora version reaches EoL.
  2. Final AMIs should never be deleted.
  3. Pre-release AMIs should be deleted after the final release.
  4. Major pre-release AMIs (Alpha, Beta, etc.) should be kept around for one more release cycle after final release, in case they might be helpful for hunting down a bug origins.

As for me, I vote for 1 and 3. My reasoning for this is:

  • If someone needs to spin up an older version of Fedora Cloud on AWS for whatever reason, they can easily create their own AMI for this purpose. Additionally, Fedimg provides a bin script that allows one to manually trigger a Fedimg job by providing it with a URL to a .raw.xz image file. So if someone needs an old Beta or something, all they need to do is give Fedimg the .raw.xz file.
  • It will save space and money.

If these are the only options then I'd vote 2,4.

As for choosing between 1,2, could somewhere in the middle be an option? How about:

Option 1.5 => Final AMIs will be kept for releases > EOL-N, where we decide the number for N. If we choose N=1 then currently we would have F19, F20, F21, F22. We could play with the number as well to say EOL-2 or so on.

Here is the current EOL status: https://fedoraproject.org/wiki/End_of_life?rd=LifeCycle/EOL

Going with the summary in comment #6 my votes are: 2 and 3.

My understanding of AWS and it's users is that AMI's should be there for a long long time. Sans some means to monitor how many times it gets launched, it's probably better to keep them than not to. What's the costs we're looking at?

As for the milestone AMIs, I don't really see a reason to keep them after final. Anything we need to debug we should be able to do in another service or locally.

Yeah, I see why 2 might be the best option.

I propose we roll with 2 (Final AMIs should never be deleted) and 3 (Pre-release AMIs should be deleted after the final release), at least for now. We can always change our decision for the next release cycle if we discover any issues with this policy between now and then.

+1? Ack?

Our new proposal for deleting AMI(s):

  1. Delete the pre-release AMI(s) after the final release.
  2. Delete the nightly build AMI(s) after 5 days of the build.

Can we please vote on this? More details are on the mailing list.

I am +1 on 1 and 2.

I am +1 on 1 and 2

To be clear this is just relating to AMIs right? Not deleting nightly images from koji?

I'm voting +1 to both proposals.

Let's make sure to send a message to the announce list before deleting any of the old AMIs, probably with a week's notice (as nice as it would be to have them cleaned up ASAP given the storage costs).

Can this be closed now? I know we deleted a bunch of stuff. Can we state officially what the new policy is and put that on a wiki page somewhere?

It was decided at last Cloud WG meeting that this can be closed as we are now cleaning up non-release AMIs periodically.

Login to comment on this ticket.