Howdy Team,
Give mirror.fcix.net tier0 access for the fedora-buffet0 rsync module.
23.152.160.16 2620:13b:0:1000::16
John Hawley and I have been operating mirror.fcix.net for the last year, and we have experienced some difficulty getting pre-bitflip access from the existing tier1 mirrors:
So we ended up needing to pull from Hochschule Esslingen since that was seemingly the only functional tier1 mirror hosting the buffet module in the entire northern hemisphere.
Since getting access to pre-bitflip, we have additionally started deploying small "Micro Mirror" appliances, so we now run 22 additional EPEL mirrors and 23 fedora mirrors which are pulling their updates from our main mirror.fcix.net box.
Given this constellation of public mirrors, we would like to be able to pull updates directly from the dl4 and dl5 tier0 mirrors to avoid the additional delay of updating via Hochschule Esslingen
No deadline.
Happy to add you!
Sadly our main datacenter (still) doesn't have ipv6. ;(
We do have 2 other mirrors that pull direct from tier0 that you may want to look at also:
download-ib01.fedoraproject.org download-cc-rdu01.fedoraproject.org
(they do have ipv6)
and one that has a snapmirror of the main content (so it might be behind a tiny bit, but usually is very close) download-rdu01.fedoraproject.org
So, I can add you on all of those and you can see what one best suits your needs. Also, instead of adding ip's we could just add the hostname? That would allow you to move ips without us needing to change anything (ie, its looked up at rsync connection time)
Anyhow, I'll look at submitting a freeze break adding you soon...
That all sounds good. I'm not an expert on how rsync hostname ACLs work, but the mirror.fcix.net A and AAAA forward records will always resolve to the source addresses we'll be using, so just using the hostname should work if that's what it relies on.
mirror.fcix.net
Hi Phil could you post about the other tier-1 mirrors to the mirror mailing list so a general conversation on which ones should be there can be done? Thanks
PR filed: https://pagure.io/fedora-infra/ansible/pull-request/1342 and freeze break requested.
Should be live on all the download servers now. :)
Feel free to try them and decide which is closest/fastest for you.
If you hit any issues, please let us know.
Metadata Update from @kevin: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Unsurprisingly, using bare rsync like I've been doing with Hochschule Esslingen took more than 90 minutes, so I killed that. (typically took about 30 minutes to download the file list from Germany)
Next, I went and got quick-fedora-mirror running, and pointing that at the dl-tier1.fedoraproject.org, it successfully planned the transaction, and then started downloading files at.... <10Mbps.
Trying download-ib01.fedoraproject.org and download-cc-rdu01.fedoraproject.org, quick-fedora-mirror faults out in the beginning that both mirrors were missing a few of the filelist files (archive on both, alt on one of them? I didn't take careful notes)
I would really prefer to hang off the dl-tier1.fedoraproject.org hostname since it seems that there's the fewest moving pieces there, but the extremely slow download speeds surprise me. Is the <10Mbps download speeds expected on the dl4 and dl5 boxes?
The rsync motd is identical on all of the tier0 mirrors, so it wasn't clear which of the two boxes I was landing on for dl-tier1.
That doesn't sound right. I will try and investigate tomorrow...
Metadata Update from @kevin: - Issue status updated to: Open (was: Closed)
Metadata Update from @zlopez: - Issue assigned to kevin - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation
Update: I've been watching downloads from dl-tier1 today, and while the baseline is about 10Mbps, I'm seeing moments of speedup, usually into the 50Mbps range, with best case moments north of 75Mbps.
So it kind of smells like these servers are starved for disk IO from my very limited perspective.
So... I looked around a bit:
21:49:48 up 97 days, 22:28, 1 user, load average: 0.08, 0.26, 0.63
So, I wonder if there's not a network issue between you and us there. Can you do a traceroute/mtr?
download-cc-rdu01 was indeed having problems. It was running out of disk space and failing syncs. I think I have this fixed now, but its syncing a bunch of things now to catch up.
download-ib01 says its all in sync and I can't find anything wrong. ;( Can you try it again when you get a chance and record the error(s)?
download-rdu01 should also be completely in sync. Note that this is different from download-cc-rdu01. Try it?
Starting to feel like it might be an intermediate peering point issue then, is there a way we can get a traceroute back to mirror.fcix.net from either/both dl0[45] and maybe see what a wget -O /dev/null to http://mirror.fcix.net/fedora/linux/releases/37/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-37-1.7.iso gets in terms of rough speed?
There's a couple of options on our end, they just are a bit messy to setup for a quick test and trying to think through a way to test this without causing a huge mess in the process.
For the ib01 issue, this is the error I'm getting with quick mirror:
rsync: link_stat "/archive/fullfiletimelist-archive" (in fedora-buffet0) failed: No such file or directory (2)
download-rdu01 seems to have all the file lists, but now we're hitting the issue that quick-fedora-mirror has filled the root partition on our main box with all its /tmp/ files, so our general reluctance to run all of these per-project custom scripts has reached a middle.
download-rdu01
It being a network path between Cogent and Hurricane Electric... doesn't fully shock me... That might be more tolerable once we catch up, since we use fuzzy, which saves quite a bit of network traffic on the rawhide ISOs. Seems like Hochschule Esslingen hasn't updated the alt/development/rawhide/Labs/x86_64/iso/ folder since 20221022 so we have some catchup to do.
alt/development/rawhide/Labs/x86_64/iso/
fcixadmin@griffin1:~$ traceroute dl-tier1.fedoraproject.org traceroute to dl-tier1.fedoraproject.org (38.145.60.26), 30 hops max, 60 byte packets 1 23.152.160.1 (23.152.160.1) 0.180 ms 0.170 ms 0.142 ms 2 v1762.core3.fmt2.he.net (72.52.94.89) 7.176 ms 7.227 ms v1868.core3.fmt2.he.net (66.220.3.129) 0.289 ms 3 * * * 4 palo-b24-link.telia.net (195.12.255.209) 1.198 ms 1.171 ms 1.146 ms 5 cogent-ic344188-palo-b24.ip.twelve99-cust.net (62.115.174.65) 1.493 ms 1.611 ms 1.724 ms 6 be2379.ccr21.sfo01.atlas.cogentco.com (154.54.42.157) 2.091 ms 2.136 ms 1.954 ms 7 be3110.ccr32.slc01.atlas.cogentco.com (154.54.44.142) 26.332 ms be3109.ccr21.slc01.atlas.cogentco.com (154.54.44.138) 27.207 ms 26.975 ms 8 be3037.ccr21.den01.atlas.cogentco.com (154.54.41.146) 27.174 ms be3038.ccr22.den01.atlas.cogentco.com (154.54.42.98) 26.455 ms be3037.ccr21.den01.atlas.cogentco.com (154.54.41.146) 27.211 ms 9 be3035.ccr21.mci01.atlas.cogentco.com (154.54.5.90) 46.979 ms be3036.ccr22.mci01.atlas.cogentco.com (154.54.31.90) 46.770 ms be3035.ccr21.mci01.atlas.cogentco.com (154.54.5.90) 46.987 ms 10 be2832.ccr42.ord01.atlas.cogentco.com (154.54.44.170) 48.708 ms 48.712 ms be2831.ccr41.ord01.atlas.cogentco.com (154.54.42.166) 48.574 ms 11 be2717.ccr21.cle04.atlas.cogentco.com (154.54.6.222) 61.480 ms be2718.ccr22.cle04.atlas.cogentco.com (154.54.7.130) 132.641 ms be2717.ccr21.cle04.atlas.cogentco.com (154.54.6.222) 61.346 ms 12 be2891.ccr41.dca01.atlas.cogentco.com (154.54.82.250) 72.439 ms be2892.ccr42.dca01.atlas.cogentco.com (154.54.82.254) 72.522 ms be2891.ccr41.dca01.atlas.cogentco.com (154.54.82.250) 72.327 ms 13 be3084.ccr41.iad02.atlas.cogentco.com (154.54.30.66) 71.618 ms 71.723 ms 71.677 ms 14 38.32.106.90 (38.32.106.90) 63.960 ms 63.935 ms 63.910 ms 15 ip-254-185-132-209.redhat.com (209.132.185.254) 66.963 ms 209.132.185.253 (209.132.185.253) 81.557 ms ip-254-185-132-209.redhat.com (209.132.185.254) 66.911 ms 16 * * * 17 * * * 18 * * *
There being a network between the dl-tier1 boxes and the storage would probably explain why their metadata performance is so much slower. I'm going to test between the various boxes with more patience and see which ones make sense. Thanks for the confirmation that the problem isn't expected.
from dl05: (dl04 is in the same datacenter and identical):
--2023-03-08 23:24:55-- http://mirror.fcix.net/fedora/linux/releases/37/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-37-1.7.iso Resolving mirror.fcix.net (mirror.fcix.net)... 23.152.160.16, 2620:13b:0:1000::16 Connecting to mirror.fcix.net (mirror.fcix.net)|23.152.160.16|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 2037372928 (1.9G) [application/octet-stream] Saving to: ‘/dev/null’ /dev/null 100%[============================>] 1.90G 43.9MB/s in 44s 2023-03-08 23:25:39 (44.1 MB/s) - ‘/dev/null’ saved [2037372928/2037372928] FINISHED --2023-03-08 23:25:39-- Total wall clock time: 44s Downloaded: 1 files, 1.9G in 44s (44.1 MB/s) real 0m44.542s user 0m1.104s sys 0m3.767s
traceroute to mirror.fcix.net (23.152.160.16), 30 hops max, 60 byte packets 1 reserved (10.3.163.252) 9.348 ms 9.329 ms 9.310 ms 2 209.132.185.204 (209.132.185.204) 9.310 ms 209.132.185.205 (209.132.185.205) 8.291 ms 8.279 ms 3 209.132.185.249 (209.132.185.249) 0.519 ms 0.433 ms 0.446 ms 4 209.132.185.249 (209.132.185.249) 0.488 ms 0.531 ms 0.474 ms 5 ae4-asbnvajp1.lightower.net (104.207.214.81) 0.727 ms 160.72.43.101.lightower.net (160.72.43.101) 0.498 ms 0.511 ms 6 ae4-asbnvajp1.lightower.net (104.207.214.81) 0.778 ms 144.121.35.1.lightower.net (144.121.35.1) 0.592 ms ae4-asbnvajp1.lightower.net (104.207.214.81) 0.710 ms 7 144.121.35.1.lightower.net (144.121.35.1) 0.560 ms 0.553 ms 0.563 ms 8 et-6-0-0.er1.iad47.us.zip.zayo.com (209.66.120.145) 1.613 ms * 1.578 ms 9 * * * 10 * * * 11 * * * 12 * port-channel1.core2.ash1.he.net (184.105.222.174) 0.800 ms * 13 * 100ge14-2.core3.fmt2.he.net (184.105.222.5) 64.190 ms 64.047 ms 14 100ge14-2.core3.fmt2.he.net (184.105.222.5) 68.108 ms 72.52.94.90 (72.52.94.90) 64.048 ms 66.220.3.130 (66.220.3.130) 64.102 ms 15 66.220.3.130 (66.220.3.130) 64.143 ms 72.52.94.90 (72.52.94.90) 64.181 ms mirror.fcix.net (23.152.160.16) 64.101 ms
Ahhh... yes, apparently we stopping carrying archive on download-ib01. It's 8.5TB and that machine only could barely fit it... but without room for new content. Sorry about that, I should have remembered. So, if you want archive thats not a option.
Yeah, quickmirror likes to eat /tmp... that was in fact the problem on download-cc-rdu01. ;( I told it to use /srv/tmp (which is on a much larger volume).
I understand about running oneoff things. ;( I wish we could figure out a way to upstream what quickmirror does.
The network between dl-tier1 boxes and storage is a 10G local lan to the netapp in the next rack, so I don't think thats the bottleneck, but I could be wrong. rsync is just horrible here because there's so many files and it basically asks the server to stat every one of them and send the info back. :( Thats basically what quick mirror avoids because it generates that once locally and you just need to get the file and check it against local files (or just do nothing if it's not newer than the last time you checked the file).
Anyhow, thanks for testing/feedback... happy to try and get it working better as much as we can. ;)
ok that's starting to look like routing since we dump pretty quickly to telia.net and then cogentco, and you are coming back (and seemingly not seeing a speed issue) via zayo. Ok time to go play with setting up something slightly one off. If we need another domain name temporarily added to the rsync list is that going to be a huge hassle/issue?
Normally no, but right now we are in freeze for beta, so I would need to get a freeze break to do it. Completely doable, just will take longer to file the PR, ask for acks, etc.
Current status:
I'm not really one to tell Fedora or Redhat to go spend several grand on NVMe drives, but if the tier0 mirrors are really NFS mounting a filer in the next rack over, even with 10Gbps Ethernet, no wonder having rsync walk the whole buffet has always been so painful. Modern NVMes or ZFS with metadata L2ARC would really make walking the tree much less painful. SSDs have solved this problem, but only if they're local to the box; NFS tends to throw those IOPS gains away, in my experience. Something to think about, but again, I'm not one to tell anyone here how to run these servers.
Looking at our traceroutes, it looks like our path to the main tier0s are:
It looks like this return path via Crown Castle and Zayo is badly congested. I tried a few different prepend and BGP community options, but it looks like I don't have any way to control how the return path from Redhat routes to us.
Would we be able to run an mtr traceroute for a while to see if the problematic network becomes more clear? Zayo and Crown would be about my last choice of networks I'd want to try and work with to troubleshoot congestion so some kind of evidence of which one of those is being a problem might help.
How good is the relationship with the Redhat NOC? Would it be possible to try dumping the 23.152.160.0/24 route out on Cogent instead of Crown?
At this point, I'm looking at pulling from download-ib01 for everything except /archive/ and then pulling just /archive/ from dl-tier0... I'm currently pulling a LOT of folders in /alt/stage/ which were not on Hochschule Esslingen, which is kind of weird.
/archive/
/alt/stage/
Current status: download-ib01.fedoraproject.org: We're able to pull from this server at about 300Mbps, but this box is not carrying /archive/ in buffet, which is a bummer.
Yeah, we ran low on local space. I put in for a replacement machine with more disk, but... haven't gotten approval for it yet. ;(
download-cc-rdu01.fedoraproject.org: This server still seems to be 5 days behind. I just tried pulling from it and it started sending me the 20230306 dailies.
Yep. I am trying to get it in sync, it has had issues. I am working on it. I can let you know when it's back synced finally.
download-rdu01.fedoraproject.org: The IOPS on this box are painfully slow. It's only able to send us the metadata for a few hundred files per second, which means walking buffet will take most of the day.
Huh, yeah. It's an old machine (I have also requested a replacement for it), but not sure whats going on. I will investigate.
dl-tier1.fedoraproject.org: We're still seeing 10-30Mbps download from there. I'm not really one to tell Fedora or Redhat to go spend several grand on NVMe drives, but if the tier0 mirrors are really NFS mounting a filer in the next rack over, even with 10Gbps Ethernet, no wonder having rsync walk the whole buffet has always been so painful. Modern NVMes or ZFS with metadata L2ARC would really make walking the tree much less painful. SSDs have solved this problem, but only if they're local to the box; NFS tends to throw those IOPS gains away, in my experience. Something to think about, but again, I'm not one to tell anyone here how to run these servers.
Well, I am open to suggestion, but alas, I do not have a magic 'buy a bunch of hardware' wand. ;(
One thing I can do is that I was considering bumping the memory in those download boxes a bunch and see if we could get it so they could cache more of the metadata. Thats doable, but not sure it will happen before the freeze is over (next wed after the beta release on tuesday). But I can do it and see if it helps this case any.
Looking at our traceroutes, it looks like our path to the main tier0s are: mirror.fcix.net → tier0: AS7034 (me) → Hurricane Electric → Telia → Cogent → Redhat tier0 → mirror.fcix.net: Redhat → Crown Castle → Zayo → Hurricane Electric → AS7034 (me) It looks like this return path via Crown Castle and Zayo is badly congested. I tried a few different prepend and BGP community options, but it looks like I don't have any way to control how the return path from Redhat routes to us. Would we be able to run an mtr traceroute for a while to see if the problematic network becomes more clear? Zayo and Crown would be about my last choice of networks I'd want to try and work with to troubleshoot congestion so some kind of evidence of which one of those is being a problem might help.
Sure, you just want a mtr to you from those machines?
I can ask.
Thats likely the beta staged... we stage them there for qa, etc... but not sure why the other mirror doesn't have that too.
Oh geeze download-rdu01 is old. ;( "System x3550 M2"
Thats doable, but not sure it will happen before the freeze is over
I'm in no big hurry here, so post-freeze is fine. If waiting a few weeks makes any of this easier, we should wait a few weeks. Being a mirror is a long term project. :)
One thing I can do is that I was considering bumping the memory in those download boxes a bunch
This is music to our ears. When we were speccing out the box for mirror.fcix.net, John brought enough trauma from his days running mirrors.kernel.org that we ended up with 384GB of RAM on that box, and 2TB of NVMe drives for metadata L2ARC for the ZFS datastore, and let me tell you, we have not a single regret about building the box that big.
I did some research last night to see if there's an easy way to get a mount to cache metadata more aggressively than file contents like in ZFS, but didn't find anything I could suggest here.
Yeah, mtr from one of the two dl-tier1 boxes to mirror.fcix.net with a few hundred or few thousand samples to see if we can narrow down a specific hop that's congested here.
The weird part was that I didn't have a bunch of the 37 staged betas / RCs. I suspect Hochschule Esslingen might be doing some level of filtering on irrelevant staged folders to release disk space. We're sitting on 96TB of disks with 17TB free, so we haven't let go of our completionist attitude yet.
I went and looked at the systems, and I am not sure the issue is memory dependent or NFS related. The NFSstats on the systems is not high on dl04/dl05 (dl-tier1) and local rsync goes a lot faster than the numbers you are seeing outside the site.
When I started looking at the 2 proxy servers and the 3 regular http/dl servers, I found that they are pegged at 800-900 Mbps while the 2 dl-tier were at 100Mbps. While traffic on these virtual machines would be 50% local network, all of those combined would be close to what I remember the uplink to the internet from our side was(*). I think we will need to work with Red Hat IT to see if we are saturating the uplink or if the new firewall from January needs some tuning which we didn't get to because of outage schedules.
(*) my memory on this is fuzzy as it was from 2020ish when I last dealt with this.
[Note while it would be nice for us to have large storage servers like that, we probably do not have either the rack space or the funding for the multiple 100-200 TB systems we would need to replace the current dl system with dedicated hardware. ]
Update: My latest plan of pulling from download-ib01.fedoraproject.org --exclude archive burned me since it looks like that host also has all the centos folders in its fedora-buffet module in addition to lacking /archive/, so I killed that.
I have finally managed to do a complete sync from dl-tier1, so I've currently got mirror.fcix.net configured to just do full pulls from dl-tier1, and it's just a question of the sync being able to complete in less than 24 hours. I am not at all convinced this is going to be able to keep up with real time; the throughput was pretty good over the weekend but is settling back down in the 10Mbps-20Mbps range this week and all the daily ISOs are just a lot of churn.
So we're talking about two performance issues on the tier0 servers:
The metadata walk is a function of how much RAM the rsync servers have and how fast/low latency sequential IOPS are to the disks. Be it that these servers are running off an NFS mount, then it makes sense why walking 25M files has always been so painful and motivated the development and use of quick-fedora-mirror. Given our distaste of quick-fedora-mirror, FCIX is slogging our way through using raw rsync, which takes 90 minutes to do the initial walk, but has the advantage of not filling our root partition and we see a respectable network efficiency improvement on the Fedora module with --fuzzy (which matters due to the second issue...)
--fuzzy
After downloading the filelist and planning the rsync transaction, we're seeing raw throughput issues from dl-tier1→mirror.fcix.net. If this is due to congestion on the Fedora → Redhat or Redhat → Transit links, then that's real easy in that we know where the issue is. Less easy in considering what the solution should be or if there's a solution needed/deserved, but it'd be good to confirm this is the issue or not.
So there may be a philosophical question about if the Fedora uplink is saturated if QOS makes sense to try and accelerate propagation to tier1 mirrors over mirrors using the public dl.fedoraproject.org servers, but we are way way above the paygrade of brand new tier1 volunteers at that point. My perspective on that is biased and severely lacking in relevant details.
I have repaved dl05.fedoraproject.org. It's now RHEL9.1, has 128GB memory, uses nfsv4 instead of v3.
I see your mirror has hit it, will see if it does any better. I'll likely do the rest tomorrow if I can get time...
It may also only help after things are cached in memory more, but that should happen in short order.
I poked around at various other things, but didn't find anything that looks like it would obviously help.
Thanks for the feedback and info here... it's appreciated!
I tried to play with readahead there some and I think I made it really slow for a bit there. ;(
Back at default now and I am going to stop messing with it for now... so, lets take a few runs to see how it looks rather than the current one probibly.
@phirephly @warthog9 dl-tier1.fedoraproject.org is an alias to dl05.fedoraproject.org and dl04.fedoraproject.org Could you run tests against the dl04 and dl05 to see if there is any difference in sync times?
I'm seeing essentially the same metadata performance on both dl04 and dl05 at this point.
Am I correct in understanding that these are regular NFS mounts? Are these shared mounts of one volume, or is each server managing its own dataset? NFS uses its own attribute cache semantics, so you can set mount -o actimeo=600 to tell it to cache attributes for 10 minutes, but this is only safe if the same system is responsible for updating the dataset as well, since the cache could be stale if another client is changing the directory out from under it.
mount -o actimeo=600
Right, but is download performance any different (ie, after it starts transfering?)
For this purpose, it's one netapp volume ("fedora_ftp"... for historical reasons). It's mounted r/o on all the download servers. It's mounted r/w on some compose machines/update machines, so when composes/updates happen, they sync to the volume. So, yeah, too aggressive caching would mean that something would finish and they wouldn't see it, but I'm not sure how big a deal that would be in practice and might be worth fiddling with.
Another datapoint: can you try a sync with '--cc none' (ie, it should not do any checksumming, and just transfer entire files. That should be fine to do as I can't think of anything we update, everything that changes should be new files. It would use potentially more bandwith, since it's not doing any chunks, but for our purposes it should be fine, since only entire files change anyhow.
Ah, ok, so NFS metadata cache doesn't seem like a great plan depending on the exact coherence requirements. Using the fsc flag to enable file system caching might be interesting, but from what I understand that just caches pages, which shouldn't be the issue here since you said there's 10Gbps between the Filer and the VMs.
fsc
I'm not seeing any throughput performance difference between dl04, dl05, or the public dl1-3 servers. I still suspect that the throughput issues are due to congestion in the Fedora → Redhat → Crown Castle → Zayo → HE network path. I would expect more RAM and caching knobs would only improve the metadata walk, which takes >90 minutes atm and should only be a function of dl0N - NFS mount metadata latency.
Well, turns out we already are setting actimeo=600. ;)
fsc I did try a long time ago... it caused a ton of churn on local disks and didn't seem to help too much. I agree it sounds like it's data only anyhow...
So, I will ping our storage folks and see if they have any ideas. I will also see if networking can do anything on that network path.
Note however, that I am traveling all next week, so after tomorrow I probibly won't be able to look into this more until after I am back.
Ok, so tabling the metadata issue and focusing back on the throughput issue, since ultimately that's the issue that's more limiting here.
I realized that we could essentially do equivalent tests from other vantage points just using the public dl.fedoraproject.org rsync servers. Testing from other servers which aren't using Hurricane Electric for transit (i.e. southfront.mm.fcix.net or forksystems.mm.fcix.net) I am seeing profoundly better performance than from dl-tier1→mirror.fcix.net. forksystems.mm.fcix.net was able to pull from the public rsync servers at north of 1Gbps, which I have never seen from dl.fedoraproject.org or dl-tier1.fedoraproject.org from AS7034 in the last year. So I suspect whatever issue we're having here has been persistent across at least the last year that I've been trying to run a Fedora mirror, because 10 months ago I was seeing the same <10Mbps numbers from the public dl servers, but had just assumed that was because they were public and getting hammered.
So I think this shows that the congestion issue isn't between Fedora → Redhat, but there's a major issue in the Redhat → Crown → Zayo → Hurricane Electric path.
I think the most interesting test at this point would be for Redhat to adjust their localpref for our 23.152.160.0/24 prefix from their providers and route it out of not Crown Castle. Enjoy your travels; this can of course wait until next next week.
All the routers were updated and rebooted. The weight for crown should be also lower on the Red Hat side. Could you see if you see any difference in rsync speeds.
I'm still seeing <20Mbps for dl-tier1. If you do a traceroute from that end towards mirror.fcix.net does it show it taking a different path?
# traceroute mirror.fcix.net traceroute to mirror.fcix.net (23.152.160.16), 30 hops max, 60 byte packets 1 reserved (10.3.163.252) 2.169 ms 2.141 ms 2.119 ms 2 209.132.185.205 (209.132.185.205) 5.685 ms 5.692 ms 5.699 ms 3 209.132.185.249 (209.132.185.249) 0.443 ms 0.494 ms 0.487 ms 4 209.132.185.249 (209.132.185.249) 0.474 ms 160.72.43.101.lightower.net (160.72.43.101) 2.515 ms 2.533 ms 5 ae4-asbnvajp1.lightower.net (104.207.214.81) 3.603 ms 3.606 ms 160.72.43.101.lightower.net (160.72.43.101) 1.874 ms 6 144.121.35.1.lightower.net (144.121.35.1) 0.421 ms 0.482 ms 0.493 ms 7 et-6-0-0.er1.iad47.us.zip.zayo.com (209.66.120.145) 0.560 ms 144.121.35.1.lightower.net (144.121.35.1) 0.489 ms et-6-0-0.er1.iad47.us.zip.zayo.com (209.66.120.145) 0.539 ms 8 et-6-0-0.er1.iad47.us.zip.zayo.com (209.66.120.145) 0.601 ms 0.701 ms 0.671 ms 9 * * * 10 * * * 11 * * * 12 * * * 13 100ge14-2.core3.fmt2.he.net (184.105.222.5) 66.343 ms hurricane-svc077499-ic366923.c.telia.net (195.12.255.210) 63.820 ms 100ge14-2.core3.fmt2.he.net (184.105.222.5) 63.202 ms 14 100ge14-2.core3.fmt2.he.net (184.105.222.5) 63.305 ms 64.102 ms 72.52.94.90 (72.52.94.90) 63.011 ms 15 mirror.fcix.net (23.152.160.16) 64.055 ms 72.52.94.90 (72.52.94.90) 62.847 ms mirror.fcix.net (23.152.160.16) 63.215 ms
is lighttower different from crown?
also could you try the following and see if there is any difference?
dl03.fedoraproject.org instead of dl-tier1 ?
lighttower is the rDNS domain Crown Castle uses for who knows why, so it looks like the Redhat → Crown → Zayo → HE → AS7034 path hasn't changed.
I see dl03 oscillating between 20Mbps and maybe 75Mbps, so the same type of congested behavior.
Testing dl03 from a totally different network perspective (southfront.mm.fcix.net) I see 500-700Mbps.
It looks like download-ib01.fedoraproject.org is wedged again. It's stuck back in March 28ish.
Yeah. I am trying to get it to handle archive too (I think it has enough space now).
Sorry for lack of updates here... lots of fires. ;(
download-ib01 should be back and now has also archives if you want to try using it.
Are you still seeing the slower/crown link to iad2? If so we can ping our networking people again...
Sorry for lack of updates here... lots of fires
I hear that.
download-ib01 should be back and now has also archives if you want to try using it
The last time I checked, download-ib01 also had the centos folders in buffet. Do we think that its buffet module matches tier0 now?
Are you still seeing the slower/crown link to iad2?
It's easier for you to answer this question than for me by running a traceroute from dl-tier1 to mirror.fcix.net, but I'm still seeing <10Mbps download on it, yeah.
Sorry for lack of updates here... lots of fires I hear that. download-ib01 should be back and now has also archives if you want to try using it The last time I checked, download-ib01 also had the centos folders in buffet. Do we think that its buffet module matches tier0 now?
I think it should match. rsync seems to think so too. (ie, at times I did just a full rsync of all buffet0 over the local contents to make sure it was in sync)
Are you still seeing the slower/crown link to iad2? It's easier for you to answer this question than for me by running a traceroute from dl-tier1 to mirror.fcix.net, but I'm still seeing <10Mbps download on it, yeah.
ok. :( Will try and get some tracion with network folks... thanks for your patience.
So, sorry this has sat for so long. ;(
I did talk to networking folks and they said they have done what they can to make things for you prefer the other link, but there may well be something else thats deciding that the crown link is 'better'. ;( They don't really have the cycles to do much more on it. They did say that they plan to upgrade the crown link from 2G to 10G soon, perhaps that will help out with the issues. :(
Were you able to get download-ib01 working for your needs?
I am syncing the entire buffet back to a new download-cc-rdu01 now also. That should be in service later this week. Also, I am bringing a new download-ib01 up on new hardware, that should also be live later this week (it should be faster than the current one, and have everything).
So, between those 2 and the master mirrors, do you have a working setup here now? Or should we try and do something more here?
So, both download-ib01 and download-cc-rdu01 are up and fully in sync. They have all content (including archives) and have stayed in sync for a while now.
Please see if either of those work out better to sync from?
I am going to go ahead and close this, but if you want us to try and do anything more, please do open a new ticket or just re-open this one and we will be happy to.
Login to comment on this ticket.