#2398 kojira: bring back parallel repo deletions
Closed: Fixed 3 years ago by tkopecek. Opened 3 years ago by mikem.

in PR#2340, most repo deletions were moved to a separate thread, but the deletions are actually run serially in the thread rather than being queued and run as separate processes.

I believe this could cause problems when maven repos are in use, because of the amount of time that it takes to remove each one.

There are two aspects to the tryDelete function where these changes took place:

  • determining whether the repo should be deleted
  • triggering its deletion

I think it makes sense to keep this in a separate thread. The problem in #2336 was that the delete loop was in updateRepos, which was only called once each pass of the main loop and could only queue up four deletes at a time. For normal repos, those four would be deleted very quickly, but the system would have to wait a long time to queue more.

However, I also think it makes sense to run the actual deletions in parallel separate from the loop that determines that they need to happen. We can have the best of both options here, we just need to be careful about threading.


Some numbers from Brew for reference.

At the moment, there are 485 maven repos on disk, with a lifetime setting of 36 hours. This gives us an estimate for the rate that maven repos are deleted.

If we delete 485 maven repos in the span of 36 hrs, then that's about 4.5 minutes per maven repo. Given that maven repos can take several minutes to delete, this gives me pause. I can see this process backing up easily.

Parallelizing this sort of io in separate processes typically gives very good gains in my experience. I think it is worth bringing this back.

Metadata Update from @mikem:
- Custom field Size adjusted to None

3 years ago

Metadata Update from @jcupova:
- Issue tagged with: testing-ready

3 years ago

Metadata Update from @jcupova:
- Issue tagged with: testing-done

3 years ago

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #2443 Merged 3 years ago
  • #2417 Merged 3 years ago