#9730 enable rsync for ELN composes on the master mirror
Closed: Fixed 2 years ago by kevin. Opened 3 years ago by dcavalca.

Please expose https://dl.fedoraproject.org/odcs/production/ via rsync to make it easier to consume (and potentially mirror) ELN composes done via ODCS. See https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/KGZYEGTJOCXBKPTBQQ5ZRADIYMSCGFIN/ for more details. Thanks!


cc @ngompa who was also interested in this

Metadata Update from @pingou:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-trouble, medium-gain, ops

3 years ago

rsync://dl.fedoraproject.org/fedora-eln should be up and running.

Enjoy!

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Thanks @kevin! The endpoint is working, but some of the files there are symlinks pointing to a path that's not actually exposed:

$ ll production/Fedora-ELN-Rawhide-20210330.6/compose/BaseOS/ppc64le/os/Packages/
....
lrwxrwxrwx 1 64321 64321 128 Mar 30 11:15 xfsdump-3.1.9-4.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/xfsdump/3.1.9/4.eln109/data/signed/9867c58f/ppc64le/xfsdump-3.1.9-4.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 132 Mar 30 11:15 xfsprogs-5.11.0-1.eln110.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/xfsprogs/5.11.0/1.eln110/data/signed/9867c58f/ppc64le/xfsprogs-5.11.0-1.eln110.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 118 Mar 30 11:15 xz-5.2.5-5.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/xz/5.2.5/5.eln109/data/signed/9867c58f/ppc64le/xz-5.2.5-5.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 123 Mar 30 11:15 xz-libs-5.2.5-5.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/xz/5.2.5/5.eln109/data/signed/9867c58f/ppc64le/xz-libs-5.2.5-5.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 118 Mar 30 11:15 yum-4.6.1-1.eln110.noarch.rpm -> ../../../../../../../../mnt/koji/packages/dnf/4.6.1/1.eln110/data/signed/9867c58f/noarch/yum-4.6.1-1.eln110.noarch.rpm
lrwxrwxrwx 1 64321 64321 131 Mar 30 11:15 zchunk-libs-1.1.9-2.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/zchunk/1.1.9/2.eln109/data/signed/9867c58f/ppc64le/zchunk-libs-1.1.9-2.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 118 Mar 30 11:15 zip-3.0-28.eln109.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/zip/3.0/28.eln109/data/signed/9867c58f/ppc64le/zip-3.0-28.eln109.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 126 Mar 30 11:15 zlib-1.2.11-25.eln110.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/zlib/1.2.11/25.eln110/data/signed/9867c58f/ppc64le/zlib-1.2.11-25.eln110.ppc64le.rpm
lrwxrwxrwx 1 64321 64321 116 Mar 30 11:15 zsh-5.8-5.eln110.ppc64le.rpm -> ../../../../../../../../mnt/koji/packages/zsh/5.8/5.eln110/data/signed/9867c58f/ppc64le/zsh-5.8-5.eln110.ppc64le.rpm

I think this is working fine over HTTP because it follows symlinks by default, but that's not necessarily the case for rsync.

Metadata Update from @dcavalca:
- Issue status updated to: Open (was: Closed)

2 years ago

Does using -L help here?

That fails with

symlink has no referent: "/production/latest-Fedora-ELN/compose/BaseOS/ppc64le/os/Packages/dnf-data-4.6.1-1.eln110.noarch.rpm" (in fedora-eln)

Alas. I am not sure how to fix this aside from changes in ODCS to do 'full' composes. Which of course is going to take up tons of disk space (which is one reason why they did it this way).

@jkaluza @sgallagh any ideas here?

I'm not sure how things are setup, but if the ODCS composes and the symlink targets are on the same filesystem, using hardlinks (or potentially reflinks, if the underlying filesystem supports them) could work, and would avoid taking up extra disk space.

They are seperate nfs volumes. ;)

Ok, let's try looking at this from another angle. Could we setup a batch job to upload ODCS composes to a S3 bucket once they're completed? That would solve the disk space issue, at the cost of taking up a bunch of upload bandwidth after each compose. On the other end, one would just need to sync the bucket down to populate a mirror, which is easy enough. If this is deemed doable, and we can come up with a size estimate, I can get a bucket set up.

From the ELN meeting today, the work items are being tracked in the ELN issues tracker:

https://github.com/fedora-eln/eln/issues/39 (reduce compose frequency)
https://github.com/fedora-eln/eln/issues/33 (how to consume ELN composes)

I thought of a way we might be able to provide rsync. It will mean that we will break existing consumers of eln from the mirror, but if we can get them to update it could work.

Basically:

move the odcs mount from /srv/odcs to /mnt/odcs and set it's root to be /mnt (so rsync chroots in there) and can then access /mnt/koji/ for the symlinked files.
It means anyone using rsync now will have to put a /odcs/ in their path now.

Can try this next week after release.

Thanks @kevin, happy to help test this out if needed, just let me know.

ok. I got this implemented.

You will need to use rsync with -L:
--copy-links, -L transform symlink into referent file/dir

and it should all otherwise just work. I managed to do it without changing the web side of things, so people copying via https should stay the same.

Let us know if you need anything further or run into problems with it. ;)

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Thanks @kevin, that seems to be working fine. FYI, it looks like a few files are still missing:

symlink has no referent: "production/latest-Fedora-ELN/compose/Everything/aarch64/images/Fedora-Container-Base-ELN-0.1621209601.aarch64.tar.xz" (in fedora-eln)

symlink has no referent: "production/latest-Fedora-ELN/compose/Everything/ppc64le/images/Fedora-Container-Base-ELN-0.1621209601.ppc64le.tar.xz" (in fedora-eln)

symlink has no referent: "production/latest-Fedora-ELN/compose/Everything/x86_64/images/Fedora-Container-Base-ELN-0.1621209601.x86_64.tar.xz" (in fedora-eln)

symlink has no referent: "production/latest-Fedora-ELN/compose/BaseOS/x86_64/images/Fedora-ELN-Guest-9.0-0.1621209601.x86_64.qcow2" (in fedora-eln)

though these shouldn't prevent the compose from being usable.

Thats easy to fix. Done. Those should work now too.

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Done