We are a mirror located in Taiwan (mirror.twds.com.tw). We have noticed that when using rsync to synchronize from dl/download-ib01/download-cc-rdu01.fedoraproject.org, the download speed is only around 100Mbps. We suspect that this may be due to a small TCP send buffer limiting the speed (referencing Bandwidth Delay Product). Could you please adjust the TCP buffer to improve the download speed?
Specifically, we suggest adjusting the third value of net.ipv4.tcp_wmem. The default maximum value is only 4MB, which, with a 200ms delay, theoretically caps the speed at around 160Mbps. We recommend increasing this to at least 64MB to accommodate modern bandwidth requirements.
net.ipv4.tcp_wmem
I think we would want to look carefully before changing that... we haven't had any other reports of this that I am aware of.
Do you get the same rate from:
download-ib01.fedoraproject.org
or
download-cc-rdu01.fedoraproject.org
(Those are both mirrors in other datacenters that we sync quickly to).
I suspect it has something to do with network paths/links over tcp settings, but of course I could be wrong.
Yes, both are similar speeds, download-cc-rdu01 is slightly slower.
Since we have tried adjusting the routing and testing with multiple different networks, we are almost certain that the TCP send buffer is too small and is causing the speed limitation.
We may try to adjust one of the hosts first and test whether there is an improvement. You can use this website to calculate the required buffer size: https://www.speedguide.net/bdp.php
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
So, that information is... not correct at least for our setup.
net.ipv4.tcp_wmem = 4096 16384 4194304
The displayed values are in bytes and Red Hat Enterprise Linux uses them in the following way: The first value is the minimum buffer size. New sockets cannot have a smaller size. The second value is the default buffer size. If an application sets no buffer size, this is the default value. The third value is the maximum size of automatically tuned buffers. Using the setsockopt() function with the SO_SNDBUF socket option in an application disables this maximum buffer size. `` also:
If a TCP packet is too large and exceeds the buffer size or packets are sent or received at a too fast rate, the kernel drops any new incoming TCP packet until the data is removed from the buffer. In this case, increasing the socket buffers can prevent packet loss. ```
So, are you seeing packet loss? I don't think adjusting this will do anything to help sync speed.
It might be you will get a better rate from another of the tier1/2 mirrors? https://fedoraproject.org/wiki/Infrastructure/Mirroring/Tiering#Tier_1_Mirrors
The maximum value 4194304Bytes=4MiB, which is definitely not enough in a network environment with a delay of more than 200ms.
We have currently set up a proxy close to upstream and adjusted the TCP settings and synchronized through it, and the speed has been significantly improved.
For reference, here are our TCP settings on the proxy:
net.ipv4.tcp_rmem = 4096 131072 67108864 net.ipv4.tcp_wmem = 4096 16384 67108864 net.ipv4.tcp_congestion_control = bbr
You can refer to Chapter 31. Tuning the network performance on how to adjust it, and GCP's TCP optimization for network performance has a good explanation of how buffer affects speed.
If you want to know more background knowledge, you can search for the keyword "Bandwidth Delay Product".
We have also tried other mirrors, but other mirrors in Asia are significantly behind, which is probably also caused by slow download speeds, and we want to be able to provide fedora-secondary.
$ rsync --no-motd --list-only rsync://download-ib01.fedoraproject.org/fedora-buffet/fedora/fullfilelist -rw-r--r-- 132,238,929 2024/05/20 10:23:44 fullfilelist $ rsync --no-motd --list-only rsync://fedora-rsync.ftp.pub.2iij.net/fedora-enchilada/fullfilelist -rw-r--r-- 132,238,929 2024/05/20 02:12:41 fullfilelist
I don't think we want to switch to bbr, but we can try the buffer sizes I suppose.
ok, I've increased them on dl03.fedoraproject.org.
Please test against that one and either 01/02 (still default)
The test confirmed that dl03 is several times faster than dl01/dl02. bbr is indeed not applicable to all network environments and has fairness issues. Just adjusting the buffer size is enough.
$ rsync --no-motd --stats -h --progress rsync://dl03.fedoraproject.org/fedora-buffet/fedora/linux/releases/40/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-40-1.14.iso . Fedora-Workstation-Live-x86_64-40-1.14.iso 2.30G 100% 37.05MB/s 0:00:59 (xfr#1, to-chk=0/1) Number of files: 1 (reg: 1) Number of created files: 1 (reg: 1) Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 2.30G bytes Total transferred file size: 2.30G bytes Literal data: 2.30G bytes Matched data: 0 bytes File list size: 77 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 43 Total bytes received: 2.30G sent 43 bytes received 2.30G bytes 37.34M bytes/sec total size is 2.30G speedup is 1.00
$ rsync --no-motd --stats -h --progress rsync://dl01.fedoraproject.org/fedora-buffet/fedora/linux/releases/40/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-40-1.14.iso . Fedora-Workstation-Live-x86_64-40-1.14.iso 2.30G 100% 11.09MB/s 0:03:17 (xfr#1, to-chk=0/1) Number of files: 1 (reg: 1) Number of created files: 1 (reg: 1) Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 2.30G bytes Total transferred file size: 2.30G bytes Literal data: 2.30G bytes Matched data: 0 bytes File list size: 77 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 43 Total bytes received: 2.30G sent 43 bytes received 2.30G bytes 11.45M bytes/sec total size is 2.30G speedup is 1.00
$ rsync --no-motd --stats -h --progress rsync://dl02.fedoraproject.org/fedora-buffet/fedora/linux/releases/40/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-40-1.14.iso . Fedora-Workstation-Live-x86_64-40-1.14.iso 2.30G 100% 12.38MB/s 0:02:56 (xfr#1, to-chk=0/1) Number of files: 1 (reg: 1) Number of created files: 1 (reg: 1) Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 2.30G bytes Total transferred file size: 2.30G bytes Literal data: 2.30G bytes Matched data: 0 bytes File list size: 77 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 43 Total bytes received: 2.30G sent 43 bytes received 2.30G bytes 12.79M bytes/sec total size is 2.30G speedup is 1.00
ok. I went ahead and pushed that to all of them. You may want to retry the download-ib01 and download-cc-rdu01 servers now in case one of them is better...
Thanks for the idea and discussion. Let us know if there's anything further to do.
Metadata Update from @kevin: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Metadata Update from @kevin: - Issue assigned to kevin
Log in to comment on this ticket.