My testing setup is, all on the same subnet, ipv4
- Windows Machine with Intel x520
-
- Direct Connect 10Gbps cable
- USW Aggregation switch (10Gbps)
-
- Direct Connect 10Gbps cable
- Synology NAS with Intel x520
-
- SRVIO Connection
- Debian VM
iperf3 Windows To Debian: 6Gbits/sec
.\iperf3.exe -c 192.168.11.57 --get-server-output
Connecting to host 192.168.11.57, port 5201
[ 5] local 192.168.11.132 port 56855 connected to 192.168.11.57 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 817 MBytes 6.79 Gbits/sec
[ 5] 1.01-2.00 sec 806 MBytes 6.82 Gbits/sec
[ 5] 2.00-3.01 sec 822 MBytes 6.85 Gbits/sec
[ 5] 3.01-4.00 sec 805 MBytes 6.81 Gbits/sec
[ 5] 4.00-5.01 sec 818 MBytes 6.81 Gbits/sec
[ 5] 5.01-6.00 sec 806 MBytes 6.82 Gbits/sec
[ 5] 6.00-7.01 sec 821 MBytes 6.83 Gbits/sec
[ 5] 7.01-8.00 sec 805 MBytes 6.80 Gbits/sec
[ 5] 8.00-9.01 sec 820 MBytes 6.82 Gbits/sec
[ 5] 9.01-10.00 sec 809 MBytes 6.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 7.94 GBytes 6.82 Gbits/sec sender
[ 5] 0.00-10.00 sec 7.94 GBytes 6.82 Gbits/sec receiver
Server output:
-----------------------------------------------------------
Server listening on 5201 (test #9)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56854
[ 5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56855
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 806 MBytes 6.76 Gbits/sec
[ 5] 1.00-2.00 sec 812 MBytes 6.82 Gbits/sec
[ 5] 2.00-3.00 sec 816 MBytes 6.85 Gbits/sec
[ 5] 3.00-4.00 sec 812 MBytes 6.81 Gbits/sec
[ 5] 4.00-5.00 sec 812 MBytes 6.81 Gbits/sec
[ 5] 5.00-6.00 sec 812 MBytes 6.81 Gbits/sec
[ 5] 6.00-7.00 sec 815 MBytes 6.84 Gbits/sec
[ 5] 7.00-8.00 sec 811 MBytes 6.80 Gbits/sec
[ 5] 8.00-9.00 sec 814 MBytes 6.82 Gbits/sec
[ 5] 9.00-10.00 sec 815 MBytes 6.84 Gbits/sec
[ 5] 10.00-10.00 sec 1.12 MBytes 4.81 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 7.94 GBytes 6.82 Gbits/sec receiver
iperf Done.
iperf3 debian to windows 9.5Gbits/sec
.\iperf3.exe -c 192.168.11.57 --get-server-output -R
Connecting to host 192.168.11.57, port 5201
Reverse mode, remote host 192.168.11.57 is sending
[ 5] local 192.168.11.132 port 56845 connected to 192.168.11.57 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 1.11 GBytes 9.40 Gbits/sec
[ 5] 1.01-2.00 sec 1.09 GBytes 9.47 Gbits/sec
[ 5] 2.00-3.01 sec 1.11 GBytes 9.47 Gbits/sec
[ 5] 3.01-4.00 sec 1.09 GBytes 9.47 Gbits/sec
[ 5] 4.00-5.01 sec 1.11 GBytes 9.47 Gbits/sec
[ 5] 5.01-6.00 sec 1.09 GBytes 9.47 Gbits/sec
[ 5] 6.00-7.01 sec 1.11 GBytes 9.47 Gbits/sec
[ 5] 7.01-8.00 sec 1.09 GBytes 9.47 Gbits/sec
[ 5] 8.00-9.01 sec 1.11 GBytes 9.47 Gbits/sec
[ 5] 9.01-10.00 sec 1.09 GBytes 9.47 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.0 GBytes 9.46 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.0 GBytes 9.46 Gbits/sec receiver
Server output:
-----------------------------------------------------------
Server listening on 5201 (test #7)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56844
[ 5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56845
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 2.01 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 8.00-9.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 10.00-10.00 sec 2.50 MBytes 7.72 Gbits/sec 0 2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.0 GBytes 9.46 Gbits/sec 0 sender
iperf Done.
I find that rather curious, something in the windows 10 tcp settings that limit the outgoing throughput, window size maybe?
Debian MTU
ip link show ens3 | grep -i "mtu"
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
Windows MTU
netsh interface ipv4 show subinterfaces
MTU MediaSenseState Bytes In Bytes Out Interface
------ --------------- --------- --------- -------------
1500 1 273580016987 64376522487 Ethernet 4
Use the -P flag to set the number of parallel streams. Probably, TCP windows size is limiting you on a single TCP connection but multiple connections would work fine.
That is a great idea to test!
iperf windows to debian -P4 6Gbit/sec
.\iperf3.exe -c 192.168.11.57 --get-server-output --dont-fragmen -P 4 Connecting to host 192.168.11.57, port 5201 [ 5] local 192.168.11.132 port 56910 connected to 192.168.11.57 port 5201 [ 7] local 192.168.11.132 port 56911 connected to 192.168.11.57 port 5201 [ 9] local 192.168.11.132 port 56912 connected to 192.168.11.57 port 5201 [ 11] local 192.168.11.132 port 56913 connected to 192.168.11.57 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 190 MBytes 1.59 Gbits/sec [ 7] 0.00-1.00 sec 192 MBytes 1.60 Gbits/sec [ 9] 0.00-1.00 sec 192 MBytes 1.60 Gbits/sec [ 11] 0.00-1.00 sec 189 MBytes 1.58 Gbits/sec [SUM] 0.00-1.00 sec 764 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.01 sec 190 MBytes 1.58 Gbits/sec [ 7] 1.00-2.01 sec 190 MBytes 1.58 Gbits/sec [ 9] 1.00-2.01 sec 190 MBytes 1.58 Gbits/sec [ 11] 1.00-2.01 sec 190 MBytes 1.58 Gbits/sec [SUM] 1.00-2.01 sec 760 MBytes 6.33 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.01-3.00 sec 186 MBytes 1.58 Gbits/sec [ 7] 2.01-3.00 sec 189 MBytes 1.60 Gbits/sec [ 9] 2.01-3.00 sec 189 MBytes 1.60 Gbits/sec [ 11] 2.01-3.00 sec 189 MBytes 1.60 Gbits/sec [SUM] 2.01-3.00 sec 754 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.01 sec 190 MBytes 1.58 Gbits/sec [ 7] 3.00-4.01 sec 190 MBytes 1.58 Gbits/sec [ 9] 3.00-4.01 sec 190 MBytes 1.58 Gbits/sec [ 11] 3.00-4.01 sec 190 MBytes 1.58 Gbits/sec [SUM] 3.00-4.01 sec 761 MBytes 6.33 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.01-5.00 sec 188 MBytes 1.59 Gbits/sec [ 7] 4.01-5.00 sec 188 MBytes 1.59 Gbits/sec [ 9] 4.01-5.00 sec 188 MBytes 1.59 Gbits/sec [ 11] 4.01-5.00 sec 188 MBytes 1.59 Gbits/sec [SUM] 4.01-5.00 sec 754 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.01 sec 191 MBytes 1.59 Gbits/sec [ 7] 5.00-6.01 sec 191 MBytes 1.59 Gbits/sec [ 9] 5.00-6.01 sec 191 MBytes 1.59 Gbits/sec [ 11] 5.00-6.01 sec 189 MBytes 1.58 Gbits/sec [SUM] 5.00-6.01 sec 762 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.01-7.00 sec 188 MBytes 1.59 Gbits/sec [ 7] 6.01-7.00 sec 189 MBytes 1.59 Gbits/sec [ 9] 6.01-7.00 sec 188 MBytes 1.59 Gbits/sec [ 11] 6.01-7.00 sec 189 MBytes 1.60 Gbits/sec [SUM] 6.01-7.00 sec 754 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.01 sec 192 MBytes 1.59 Gbits/sec [ 7] 7.00-8.01 sec 191 MBytes 1.59 Gbits/sec [ 9] 7.00-8.01 sec 192 MBytes 1.59 Gbits/sec [ 11] 7.00-8.01 sec 191 MBytes 1.59 Gbits/sec [SUM] 7.00-8.01 sec 766 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.01-9.00 sec 188 MBytes 1.59 Gbits/sec [ 7] 8.01-9.00 sec 188 MBytes 1.59 Gbits/sec [ 9] 8.01-9.00 sec 188 MBytes 1.59 Gbits/sec [ 11] 8.01-9.00 sec 188 MBytes 1.59 Gbits/sec [SUM] 8.01-9.00 sec 752 MBytes 6.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.01 sec 191 MBytes 1.59 Gbits/sec [ 7] 9.00-10.01 sec 191 MBytes 1.59 Gbits/sec [ 9] 9.00-10.01 sec 191 MBytes 1.59 Gbits/sec [ 11] 9.00-10.01 sec 191 MBytes 1.59 Gbits/sec [SUM] 9.00-10.01 sec 764 MBytes 6.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec sender [ 5] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 7] 0.00-10.01 sec 1.86 GBytes 1.59 Gbits/sec sender [ 7] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 9] 0.00-10.01 sec 1.86 GBytes 1.59 Gbits/sec sender [ 9] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 11] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec sender [ 11] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [SUM] 0.00-10.01 sec 7.41 GBytes 6.36 Gbits/sec sender [SUM] 0.00-10.01 sec 7.41 GBytes 6.35 Gbits/sec receiver Server output: ----------------------------------------------------------- Server listening on 5201 (test #15) ----------------------------------------------------------- Accepted connection from 192.168.11.132, port 56909 [ 5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56910 [ 8] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56911 [ 10] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56912 [ 12] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56913 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 188 MBytes 1.57 Gbits/sec [ 8] 0.00-1.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 0.00-1.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 0.00-1.00 sec 187 MBytes 1.57 Gbits/sec [SUM] 0.00-1.00 sec 754 MBytes 6.32 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 189 MBytes 1.58 Gbits/sec [ 8] 1.00-2.00 sec 189 MBytes 1.58 Gbits/sec [ 10] 1.00-2.00 sec 189 MBytes 1.58 Gbits/sec [ 12] 1.00-2.00 sec 189 MBytes 1.58 Gbits/sec [SUM] 1.00-2.00 sec 754 MBytes 6.33 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 188 MBytes 1.58 Gbits/sec [ 8] 2.00-3.00 sec 191 MBytes 1.60 Gbits/sec [ 10] 2.00-3.00 sec 191 MBytes 1.60 Gbits/sec [ 12] 2.00-3.00 sec 191 MBytes 1.60 Gbits/sec [SUM] 2.00-3.00 sec 760 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec [ 8] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec [ 10] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec [ 12] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec [SUM] 3.00-4.00 sec 755 MBytes 6.34 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 4.00-5.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 4.00-5.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 4.00-5.00 sec 190 MBytes 1.59 Gbits/sec [SUM] 4.00-5.00 sec 759 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 5.00-6.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 5.00-6.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 5.00-6.00 sec 188 MBytes 1.58 Gbits/sec [SUM] 5.00-6.00 sec 758 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 6.00-7.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 6.00-7.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 6.00-7.00 sec 190 MBytes 1.59 Gbits/sec [SUM] 6.00-7.00 sec 759 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec [SUM] 7.00-8.00 sec 760 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [ 8] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [ 10] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [ 12] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [SUM] 8.00-9.00 sec 758 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 9.00-10.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 9.00-10.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 9.00-10.00 sec 190 MBytes 1.59 Gbits/sec [SUM] 9.00-10.00 sec 759 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 10.00-10.01 sec 2.50 MBytes 1.51 Gbits/sec [ 8] 10.00-10.01 sec 2.50 MBytes 1.51 Gbits/sec [ 10] 10.00-10.01 sec 2.50 MBytes 1.51 Gbits/sec [ 12] 10.00-10.01 sec 2.50 MBytes 1.51 Gbits/sec [SUM] 10.00-10.01 sec 10.0 MBytes 6.03 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 8] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 10] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 12] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [SUM] 0.00-10.01 sec 7.41 GBytes 6.35 Gbits/sec receiver iperf Done.Though curiously i see the same behavior, 9.5Gbit linux->windows, and 5Gbit windows->linux… i’m perplexed
deleted by creator
Same results with -P but without the “don’t fragment”?
Yes, exactly the same results.
I’m thinking it might be on the linux side? i’m double checking my debian network stack now
5 Gbps is the theoretical PCIe 1x limit, but I haven’t seen those in a while, so probably not that.
I’m… half tempted to ask you if you can reverse the cables in the path lol.
Maybe a pair somewhere isn’t as good as the others and affects data rate in that direction.
Far fetched, but…
I like the way you think! This is good engineering
I replaced both direct connect cables with new ones, I used new switch ports.
The asymmetry persists. Linux to windows 9.5GiB/s, Windows to Linux 6.5GiB/s
now we know it probably isn’t the cables!