#1198 RFE: Support zchunk repodata creation with dist-repo
Closed: Fixed 2 years ago by tkopecek. Opened 3 years ago by ngompa.

Starting with Fedora 30, we will provide zchunk variants of the repodata to all delta downloads of repository metadata by package managers.

Koji's dist-repo functionality should be able to tell createrepo_c to create zck repodata and leverage zchunk dictionaries for efficient compression.

Cf. https://fedoraproject.org/wiki/Changes/Zchunk_Metadata

cc: @jdieter

Metadata Update from @mikem:
- Issue tagged with: feature

3 years ago

Metadata Update from @dgregor:
- Custom field Size adjusted to None
- Issue set to the milestone: 1.19

2 years ago

How these dictionaries would be provided? createrepo_c is not run inside the buildroot, so dicts need to be prefetched somehow, or installed on builder permanently.

Easiest hack is to have them downloaded before onto builders in createrepo channel in some special directory. This path can be referenced from tag.extra. But that looks pretty ugly with respect to maintenance and updating of data.

Different dist-repos would need different dicts. Downloading them each time would mean significant slowdown.

The dictionaries are currently provided via an RPM in Fedora and Mageia.

Fedora's is fedora-repo-zdicts, and Mageia's is mageia-repo-zdicts.

So in that case, they probably need to be installed on the builders that run createrepo_c.

Yep, it is a short-term solution. I would like to have something, where every tag can define (somehow) these dicts and even builders with other systems can download/cache them somehow. We're now on these two, but it can multiply.

Maybe it is over-engineering and we can just stay with these two (highly possibly under 10) and really use builder path as a selector.

@mikem ?

Metadata Update from @dgregor:
- Issue set to the milestone: 1.20 (was: 1.19)

2 years ago

I've prepared PR #1743 It has basic support - so you can use --zck option and add --zck-dict-dir which is path to dict on builder. No configuration per tag is present now.

Metadata Update from @jcupova:
- Issue tagged with: testing-ready

2 years ago

Metadata Update from @jcupova:
- Issue tagged with: testing-done

2 years ago

Login to comment on this ticket.

Related Pull Requests
  • #1743 Merged 2 years ago