#1394 Replace maven repo symlinks with some other mechanism
Opened 3 years ago by tkopecek. Modified 3 years ago

Creating symlinks in maven repositories is overwhelming. Such newRepo task can take hours to finish just because of creating hundreds of thousands of symlinks.

My idea is to create some different mechanism which depends on download server configuration. For e.g. apache it would just create config for mod_rewrite in given directory which then can be interpreted by it.

Caveat is obviously here - is download server apache? So, it needs to be configurable, on the other hand probably easily extensible for nginx, etc.

Metadata Update from @tkopecek:
- Issue tagged with: discussion, feature

3 years ago

This sounds fragile. I don't suppose Maven has grown a better way since we wrote this?

I also think some of this is particular to our use netapp. I have always been surprised at how slow symlinks are there.

As long as we're musing, I wonder if we could...

  • recycle an existing, but expired older version of the repo and update the symlinks (very carefully). Lots of symlinks reads, much fewer symlink writes.
  • parallelize the symlinking

More on the recycle idea...

We'd have to modify Kojira behavior. We'd only want to recycle a repo that was ready for deletion. Might be a new repo state.

When recycled, we'd move the old repo dir to the new name and very carefully make the adjustments to it.

We'd add a config option to control this.

Granted, I'm not sure if the recycle idea would actually be faster. Needs some experimentation.

Metadata Update from @yulwang:
- Issue priority set to: Low (was: Normal)
- Issue tagged with: groomed

3 years ago

Login to comment on this ticket.