#10388 prune compose OSTree repo
Closed: Fixed 2 years ago by dustymabe. Opened 2 years ago by dustymabe.

  • Describe the issue

I'm working on a fedora-ostree-pruner that will run automatically and prune our repos based on given policy once a week. Currently I'm doing some test runs of the pruner and discussing amongst ourselves some potential issues or features we'd like to implement.

While we are waiting on that there is one thing we can do to get a big win in terms of our actual space used on our shared OSTree netapp volume: prune the compose repo in a oneoff command. In the test run we see:

2021-11-12 02:30:31,354 INFO fedora-ostree-pruner - Running command: ['ostree', 'prune', '--repo', '/mnt/fedora_koji_prod/koji/compose/ostree/repo', '--refs-only', '--keep-younger-than=90 days ago', '--no-prune']
Total objects: 19814886
Would delete: 17081324 objects, freeing 2.0 TB

So we'd get about 2TB back by just pruning the compose repo. Let's go ahead and do that to free up some space while we do finer tuning on the tools. Can we get someone from releng to run this command on a composer machine (maybe compose-x86-01.iad2.fedoraproject.org):

ostree prune --repo=/mnt/koji/compose/ostree/repo/ --refs-only --keep-younger-than="90 days ago"

A couple of things to keep in mind:

  • The dry-run prune command took about 5 hours
  • The dry-run prune command used a lot of memory (we're working on this) so run on a machine with at least 32G memory or more.
  • We can't run any composes during this time period because locking will fail
  • We should trigger a backup/snapshot beforehand in case something bad happens

Seems like maybe a time when we don't anticipate any composes happening would be ideal?

  • When do you need this? (2021/11/25)

Soonish..

  • When is this no longer needed or useful? Never

  • If we cannot complete your request, what is the impact?

We run out of space!


Metadata Update from @humaton:
- Issue tagged with: low-trouble, medium-gain, meeting

2 years ago

Talked with @kevin - we're going to kick this off tomorrow (11/28/2021) at around 14 UTC (9 AM EST). The pungi/bodhi composes don't start again untill 00 UTC so we'll have 8 hours and hopefully it clears in that time. If not we'll disable the composes.

This ran successfully over the course of the day today:

Total objects: 19939933
Deleted 17189180 objects, 2.0 TB freed
real    499m53.558s
user    8m43.842s
sys     14m7.189s

Thanks @kevin!

Metadata Update from @dustymabe:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata