#3624 compress fullfilelist and update only on-demand
Closed: Can't Fix 2 years ago Opened 9 years ago by mmcgrath.

After each push, we need to run the following command:

rsync -r . > fullfilelist

This should overwrite the fullfilelist that's there and isn't very useful at the moment:

http://download.fedora.redhat.com/pub/fedora/fullfilelist

We can't do this via a cron job, it has to go out after each push so it needs to be added to those scripts.


Also we should compress it or provide a compressed version it compresses 4:1 or 6:1 or so.

This seems to have been "dropped", or at least never picked up.

If I understand things, there are at least 3 scripts that place content on the /pub/fedora/ path. Those are the fedora-updates-push script (for syncing out the updates content), buildrawhide which composes and syncs out rawhide each night, and finally buildbranched which composes and syncs out the branched release (if we have an active one).

Adding a call to rsync -r in each of these scripts, and then hoping none run at the same time seems a bit dangerous. Perhaps a better approach would be to create a script that updates/compresses the file, and does so with a locking mechanism. This way each script or manual process that puts files on the /pub/fedora path can end with a call (to the background) of this script, which will wait (in line?) until any current run is finished then execute.

That's just my suggestion though. Maybe the new releng leader will have time to look at this issue, or some other volunteer :)

There is a plan to use it with mirrormanager. The code from the link above (umdl2) is not yet active, but it is planned to be used (adding pingou for that).

Some mirrors are also using it to check if rsync should actually run over the whole tree or if nothing has changed.

Recently there was some discussion on the mirror-list how to create that file better/faster. Adding smooge as he was involved in the discussions/results.

Replying to [comment:7 adrian]:

There is a plan to use it with mirrormanager. The code from the link above (umdl2) is not yet active, but it is planned to be used (adding pingou for that).

Some mirrors are also using it to check if rsync should actually run over the whole tree or if nothing has changed.

Recently there was some discussion on the mirror-list how to create that file better/faster. Adding smooge as he was involved in the discussions/results.

I was on that thread, the issue is that the way that particular mirror is working iwth it only works in a world where we do not hardlink content. the user complained that we are using a lot more disk. the reson he is seeing that is entirely due to not using hardlinking and how he mirrors the content. in my mind it is a broken use case.

It should be perfectly easy to run hardlink locally as desired.

closing as can't fix. I really do not think there is any changes to be done here.

Metadata Update from @ausil:
- Issue close_status updated to: Can't Fix
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata