#9730 enable rsync for ELN composes on the master mirror
Opened a month ago by dcavalca. Modified 12 days ago

Please expose https://dl.fedoraproject.org/odcs/production/ via rsync to make it easier to consume (and potentially mirror) ELN composes done via ODCS. See https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/KGZYEGTJOCXBKPTBQQ5ZRADIYMSCGFIN/ for more details. Thanks!

cc @ngompa who was also interested in this

Metadata Update from @pingou:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-trouble, medium-gain, ops

a month ago

rsync://dl.fedoraproject.org/fedora-eln should be up and running.


Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

18 days ago

Thanks @kevin! The endpoint is working, but some of the files there are symlinks pointing to a path that's not actually exposed:

$ ll production/Fedora-ELN-Rawhide-20210330.6/compose/BaseOS/ppc64le/os/Packages/
lrwxrwxrwx 1 64321 64321 128 Mar 30 11:15 xfsdump-3.1.9-4.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/xfsdump/3.1.9/4.eln109/data/signed/9867c58f/ppc64le/xfsdump-3.1.9-4.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 132 Mar 30 11:15 xfsprogs-5.11.0-1.eln110.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/xfsprogs/5.11.0/1.eln110/data/signed/9867c58f/ppc64le/xfsprogs-5.11.0-1.eln110.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 118 Mar 30 11:15 xz-5.2.5-5.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/xz/5.2.5/5.eln109/data/signed/9867c58f/ppc64le/xz-5.2.5-5.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 123 Mar 30 11:15 xz-libs-5.2.5-5.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/xz/5.2.5/5.eln109/data/signed/9867c58f/ppc64le/xz-libs-5.2.5-5.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 118 Mar 30 11:15 yum-4.6.1-1.eln110.noarch.rpm -> ../../../../../../../../mnt/koji/packages/dnf/4.6.1/1.eln110/data/signed/9867c58f/noarch/yum-4.6.1-1.eln110.noarch.rpm
lrwxrwxrwx 1 64321 64321 131 Mar 30 11:15 zchunk-libs-1.1.9-2.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/zchunk/1.1.9/2.eln109/data/signed/9867c58f/ppc64le/zchunk-libs-1.1.9-2.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 118 Mar 30 11:15 zip-3.0-28.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/zip/3.0/28.eln109/data/signed/9867c58f/ppc64le/zip-3.0-28.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 126 Mar 30 11:15 zlib-1.2.11-25.eln110.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/zlib/1.2.11/25.eln110/data/signed/9867c58f/ppc64le/zlib-1.2.11-25.eln110.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 116 Mar 30 11:15 zsh-5.8-5.eln110.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/zsh/5.8/5.eln110/data/signed/9867c58f/ppc64le/zsh-5.8-5.eln110.ppc64le.rpm

I think this is working fine over HTTP because it follows symlinks by default, but that's not necessarily the case for rsync.

Metadata Update from @dcavalca:
- Issue status updated to: Open (was: Closed)

17 days ago

Does using -L help here?

That fails with

symlink has no referent: "/production/latest-Fedora-ELN/compose/BaseOS/ppc64le/os/Packages/dnf-data-4.6.1-1.eln110.noarch.rpm" (in fedora-eln)

Alas. I am not sure how to fix this aside from changes in ODCS to do 'full' composes. Which of course is going to take up tons of disk space (which is one reason why they did it this way).

@jkaluza @sgallagh any ideas here?

I'm not sure how things are setup, but if the ODCS composes and the symlink targets are on the same filesystem, using hardlinks (or potentially reflinks, if the underlying filesystem supports them) could work, and would avoid taking up extra disk space.

They are seperate nfs volumes. ;)

Ok, let's try looking at this from another angle. Could we setup a batch job to upload ODCS composes to a S3 bucket once they're completed? That would solve the disk space issue, at the cost of taking up a bunch of upload bandwidth after each compose. On the other end, one would just need to sync the bucket down to populate a mirror, which is easy enough. If this is deemed doable, and we can come up with a size estimate, I can get a bucket set up.

Login to comment on this ticket.

Boards 1
ops Status: Backlog