#7374 Tier 1 mirror, mirrors.dotsrc.org
Closed: Fixed 5 years ago by smooge. Opened 5 years ago by dotsrc.

Hi!

dotsrc.org would like to contribute a tier 1 mirror. The location of our servers is at Aalborg University in Denmark.

https://admin.fedoraproject.org/mirrormanager/host/2201

We will be pulling from the following IPs:
$ host mirrors.dotsrc.org
mirrors.dotsrc.org has address 130.225.254.116
mirrors.dotsrc.org has IPv6 address 2001:878:346::116


From what I see you are not carrying everything. I just had a look at the requirements for a tier 1 mirror and it indeed does not say that tier 1 mirrors have to mirror everything.

With the existence of quick-fedora-mirror it would be really good if a tier 1 mirror would carry everything so that tier 2 mirrors can use your mirror with quick-fedora-mirror.

I know that this is a lot of data, but could you also mirror everything so that tier2 can use your mirror via quick-fedora-mirror? This means you would also need to offer fedora-buffet via rsync.

No problem - I'll find a nearby mirror to pull everything from.
I have disabled dotsrc in mirror manager while this is going on.
Update: This is going to take some time. I can't find a mirror serving the entire "fedora-buffet" and is capable of more than 100mbit/s. I'll let you know when we are done.

What is the size of fedora-buffet with hard links taken into account?
If I sum up the directories from https://dl.fedoraproject.org/pub/DIRECTORY_SIZES.txt I get around 20TB:
3.7T /pub/alt/
11T /pub/archive/
2.6T /pub/fedora-secondary/
2.7T /pub/fedora/
175G /pub/epel/

If it really is 20TB, I'll need to buy more hard drives for mirrors.dotsrc.org.

I really thought I had replied to this, but I must have forgotten to click the submit button as usual.

In any case, it really is about 20.5TB for a full mirror. However, at least quick-fedora-mirror doesn't require that you actually mirror everything, only that what you do mirror is contained within a master module. So you could exclude archive (to save 11TB) and alt (to save 100GB+/day of churn) and still be useful as an upstream for quick-fedora-mirror as long as you still had the fedora-buffet module that contains fedora, epel, and fedora-secondary. The requirement for a master module allows quick-fedora-mirror to grab, say, fedora and fedora-secondary in one rsync call, allowing hard links to be copied as hard links.

Unfortunately, rather than using that when it can as an optimization, the current quick-fedora-mirror implementation actually requires the master module to work. Fixing that is on my overly long TODO list....

So, whats the status here? Still syncing? I guess once you are fully synced we need to add you to the wiki page.... any other actions here?

Metadata Update from @bowlofeggs:
- Issue priority set to: Waiting on Reporter (was: Needs Review)

5 years ago

Hi!
This is currently on hold, as we are out of free space. I expect new hard disks to arrive within a couple of weeks at which point I'll continue the sync.

Awesome. Let us know when you are synced up and we can add you to the right places and such.

I'll close this for now, but reopen when you are ready.

Metadata Update from @kevin:
- Issue close_status updated to: Upstream
- Issue status updated to: Closed (was: Open)

5 years ago

Hi all!

We now host a copy of the entire buffet at (ftp|rsync|http|https)://mirrors.dotsrc.org/fedora-buffet/ !

The files were pulled from ftp-stud.hs-esslingen.de. Would you like us to pull from a master mirror from now on? And how often should we sync?

Also, please verify this config:
https://admin.fedoraproject.org/mirrormanager/host/2201
I'm not sure what the right URL to use is for "Fedora Archive" and "Fedora Other".

Metadata Update from @dotsrc:
- Issue status updated to: Open (was: Closed)

5 years ago

Metadata Update from @mizdebsk:
- Issue priority set to: Needs Review (was: Waiting on Reporter)

5 years ago

Metadata Update from @smooge:
- Issue assigned to smooge

5 years ago

The site has been added to all download servers. Please test and let us know.

I can confirm that we are able to pull from rsync://dl-tier1.fedoraproject.org/fedora-buffet0/

How often should we sync, and is it still recommended to use quick-fedora-mirror instead of plain rsync?

I would run quick-fedora-mirror doing it at least once a day and a weekly 'huh why didn't I get that before?' sync even as a report to see if quick-fedora-mirror isn't getting everything.

If you are using quick-mirror-fedora you can mirror as often as you like, as it has almost no impact on our side and should return in seconds if everything is in sync.

Metadata Update from @bowlofeggs:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

5 years ago

Alright - I think everything runs as it should on our side now. On http(s)://mirrors.dotsrc.org/ we have the following symlinks:
fedora-epel -> fedora-buffet/epel
fedora(-enchilada) -> fedora-buffet/fedora

I have one last question: The quick-fedora-mirror comes with a feature called 'checkin'. Does that mean we no longer need to run the report_mirror program?

I have one last question: The quick-fedora-mirror comes with a feature called 'checkin'. Does that mean we no longer need to run the report_mirror program?

Yes, quick-fedora-mirror does the check-in for you. No need to run report_mirror.

As you are mirroring everything it would also make sense to update your MirrorManager entry and add Fedora Archive, Fedora Other and Fedora Secondary Architectures.

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 years ago

Login to comment on this ticket.

Metadata