@nate-duke Hey, you are listed as admin contact here. Could you take a look?

I mailed nate-duke directly.

He's going to take a look.

Hey guys, sorry. I was having trouble logging in here yesterday and this morning pagure was down all together. I didn't have a chance to look at this yet but it's on my list for Monday morning.

Ok, sorry it took me so long to get to this. Busy week!

I've corrected the rsyncd configuration to provide the correct view for ::fedora-enchilada

❯ rsync --list-only rsync://archive.linux.duke.edu/fedora-enchilada/linux/

archive.linux.duke.edu
 - This rsync server is currently available to any/all people.
 - This is subject to change with little or limited notice.

Modules:


drwxr-xr-x             97 2018/02/21 15:20:22 .
drwxr-xr-x            161 2006/10/17 08:46:37 core
drwxrwxr-x             24 2013/04/25 04:55:49 extras
drwxr-xr-x            722 2024/10/25 15:53:24 releases
drwxrwsr-x            703 2025/02/04 10:27:53 updates

The missing development/ path was due to an effort to save some space in a prior configuration of this system. We've since rearchitected to more sensibly allocate storage resource to this project but I neglected to add extra storage back for fedora and adjust the quick-mirror config. I've corrected this now and we're running a quick-mirror to pull in the new content.

Total on client:        436019 files, 3490 dirs.
Not present on server:  235709 files, 1720 dirs.
Missing on client:      159337 files,  537 dirs.
Size Changed:               58 files.
Timestamps to restore:    2278 files.
Checksum Failed:            72 files.
Filelist changes:       253828 paths.
Total to transfer:      253831 paths.

thanks a lot for the fixes!

So, how are we looking now?

Can we close this? or do we still need to fix anything?

top level looks right now, but the development directory is still missing.

I'll double check it. the development directory was taking a while to sync so i just left it to the persistence of cron to eventually finish.

EDIT: I expect it's the '-T "last day"' argument to our quick-mirror call that's to blame.

Running a fresh sync now without the time specification.

How's it looking now?

Still no development:

[adamw@toolbx openQA (master %)]$ rsync --list-only archive.linux.duke.edu::fedora-enchilada/linux/

archive.linux.duke.edu
 - This rsync server is currently available to any/all people.
 - This is subject to change with little or limited notice.

Modules:


drwxr-xr-x             97 2018/02/21 12:20:22 .
drwxr-xr-x            161 2006/10/17 05:46:37 core
drwxrwxr-x             24 2013/04/25 01:55:49 extras
drwxr-xr-x            722 2025/04/11 04:01:08 releases
drwxrwsr-x            723 2025/02/04 07:27:53 updates

@adamwill Yeah, i've been messing with it every morning for the past week trying to get it to sync that directory. When i run quick-fedora-mirror it makes a bit of progress and then stalls out. Here's the tail of the output. I'm open to any suggestions on alternative ways. I was reading through the docs yesterday and trying to come up with an alternate way to get the development directory and then running the hardlinker. The docs indicated that was a possibility.

The full output of the currrent run is in a snippet here

Total on client: 736623 files, 5449 dirs.
Not present on server: 400240 files, 3147 dirs.
Missing on client: 38414 files, 18 dirs.
Size Changed: 58 files.
Timestamps to restore: 3518 files.
Checksum Failed: 90 files.
Filelist changes: 271653 paths.
Total to transfer: 271656 paths.
>> Log: Counts for fedora-epel: Svr:374794/2320 Loc:736623/5449 Diff:271653 New:269549/2104 Xtra:400240/3147 Miss:38414/18 Size:58 Csum:90 Dtim:3518
>> Log: Processing end: fedora-epel
Finished processing fedora-epel.
Changes in fedora-epel: 271656 files/dirs
============================================================
============================================================
Transferring 1644553 files.
>> Calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l  %n%L -v --files-from=master-transferlist.sorted rsync://dl.fedoraproject.org/fedora-buffet0/ /fedora/pub/
>> Log: calling /usr/bin/rsync --timeout=600 -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i %10l  %n%L -v --files-from=master-transferlist.sorted rsync://dl.fedoraproject.org/fedora-buffet0/ /fedora/pub/

so...it's winding up with a huge transfer, basically. I don't think it's doing anything wrong, but that transfer looks like it's going to take a long time. At various points it logs "Possibly aborted rsync run. Cleaning up.", which indicates that yeah, the previous run just didn't finish.

For fedora-enchilada It looks like you still have the 39 and 40 releases present; manually 'syncing' those (so they just contain the README pointing to archive) might help a bit. It might be a good idea to concentrate on one module at a time; both enchilada and epel seem to want to do a huge amount of syncing, for some reason, so doing them both in one transaction is maybe not a good idea, it might be best to get one synced then work on the other?

I'll give that a shot. Since i posted that and the previous run log i streamlined things a bit and increased verbosity to 7 to maybe get some progress output and it's been chewing on the filelist for a few hours.

I also went back to -T 'last day' in case that was adding to the troubles.

It's not using very much memory and seems single threaded (no surprise there i suppose).
current run output

far as i can tell it hasn't transmitted much data either, just chomping on that file list.

here's some boring graphs of resource usage for the current manual run.

I wonder if it might actually be better to do a non-"quick" sync first to get things back more or less in line? I'm guessing maybe the "quick" sync stuff doesn't work so well if things are wildly out of line to start with.

oh, also, a dumb idea, but: it's not somehow still configured to sync to the same layout you had before, and thus essentially redoing everything from scratch plus throwing away the correct layout, is it?

oh, also, a dumb idea, but: it's not somehow still configured to sync to the same layout you had before, and thus essentially redoing everything from scratch plus throwing away the correct layout, is it?

shouldn't be. This stuff is all running in kubernetes using a specific PVC for each mirror and that hasn't changed. I.e. the backing NFS export, the mount point inside the container nor the parameters for the target directory in the quick-fedora-mirror.conf file.

I wonder if it might actually be better to do a non-"quick" sync first to get things back more or less in line? I'm guessing maybe the "quick" sync stuff doesn't work so well if things are wildly out of line to start with.

I'm open to trying this. i messed around with how to do it a few days ago but the stern warnings coming back from the upstream rsync server to not just use rsync shied me away.

oh, also, a dumb idea, but: it's not somehow still configured to sync to the same layout you had before, and thus essentially redoing everything from scratch plus throwing away the correct layout, is it?

... i've been chewing on this question. Can you clarify the issue this would present?

We are using the same layout. Ultimately the only change that we've made was to change

FILTEREXP='(/releases/test|/development)'

to

FILTEREXP='(/releases/test)'

No, what I meant is how you initially had the top level set as /pub rather than /pub/fedora, as reported in the initial comment. It seemed just barely possible that there was some kind of mismatch there in the sync config so none of the files matched between the ends at all and it would end up wiping all your local files and re-transferring everything. But I don't think that's right, looking at the logs.

It's definitely wanting to transfer a hell of a lot, though. Look at the enchilada summary:

Total on client:       1313976 files, 3535 dirs.
Not present on server:  378826 files,  765 dirs.
Missing on client:      779291 files, 1569 dirs.
Size Changed:               17 files.
Timestamps to restore:    1658 files.
Checksum Failed:            30 files.
Filelist changes:      1374874 paths.
Total to transfer:     1374883 paths.

So it thinks nearly 800,000 files are "missing on client" (i.e. those are files it's going to have to fully copy over from the server) and nearly 400,000 files are "not present on server" (i.e. those are files you have locally but which don't match anything on the server; I was assuming this was just the F39 and F40 files, but...). That seems like a lot of files, though I don't actually know if 800,000 is anyhing like "all the files". I guess it can plausibly just be 'all of development'.

@adamwill, thanks for the explanation. The web front end for this stuff is pretty malleable, that was the issue with the earlier "parenting issues" ;)

I guess we'll just let it run like this and hope for the best. If it hasn't made any progress tomorrow I'll abort it and look into running a bare rsync of just the development tree.

still chugging away today, it has made it up to

 1013500 files...
 1013600 files...
 1013700 files...

but is only using ~400MB of memory (4Gi available) and is basically saturating one core.

I'm going to restart it again and give it more cores to see if that helps is chew on that file list.

ok, i just punted and did a plain rsync of the development tree. I have a "normal" quick-fedora-mirror running now that'll hopefully be able to finish on it's own since all the development content will be present.

🤞🏻

Just as an update. the quick-fedora-mirror process takes a little over 24h to chew through the file list. It's made some good progress overnight and i'm still minding it daily to make sure it hasn't croaked.

I'll drop an update here every couple of days until I see it able to complete in a timely fashion.