Bug #264

Performance over the LFN

Added by Dave Täht on Sep 6, 2011. Updated on Apr 21, 2012.
New Normal Jim Gettys

Description

I get rather different results from netperf. This is iperf

From 4 driver buffers, 8 txqueuelen

LFN = root@europa:~# ping 98.229.130.122
PING 98.229.130.122 (98.229.130.122): 56 data bytes
64 bytes from 98.229.130.122: seq=0 ttl=54 time=96.463 ms

root@europa:~# iperf -w220k -c 98.229.130.122 
------------------------------------------------------------
Client connecting to 98.229.130.122, TCP port 5001
TCP window size:  220 KByte
------------------------------------------------------------
[  3] local 149.20.63.19 port 49114 connected with 98.229.130.122 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  4.00 MBytes  3.32 Mbits/sec
root@europa:~# iperf -t 60 -w220k -c 98.229.130.122 
------------------------------------------------------------
Client connecting to 98.229.130.122, TCP port 5001
TCP window size:  220 KByte
------------------------------------------------------------
[  3] local 149.20.63.19 port 49115 connected with 98.229.130.122 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  59.9 MBytes  8.37 Mbits/sec
root@europa:~#

And from a normal desktop, 64 driver buffers, 1000 txqueuelen

^Cd@bob-desktop:~$ iperf -w 220k -\c 98.229.130.122
------------------------------------------------------------
Client connecting to 98.229.130.122, TCP port 5001
TCP window size:  256 KByte (WARNING: requested  220 KByte)
------------------------------------------------------------
[  3] local 149.20.63.20 port 36413 connected with 98.229.130.122 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  10.4 MBytes  8.60 Mbits/sec
d@bob-desktop:~$ iperf -t 60 -w 220k -\c 98.229.130.122
------------------------------------------------------------
Client connecting to 98.229.130.122, TCP port 5001
TCP window size:  256 KByte (WARNING: requested  220 KByte)
------------------------------------------------------------
[  3] local 149.20.63.20 port 36414 connected with 98.229.130.122 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.1 sec  58.4 MBytes  8.15 Mbits/sec

History

Updated by Dave Täht on Sep 6, 2011.
I was going to reboot you into a txqueuelen of 256, but I merely left it setup so if you rebooted the ethernet drivers
would have a txqueuelen of 256.

I enabled qos-scripts, setting it to 2000 bidirectionally, as the max as I could get on a single stream was ~8000

the ‘overhead’ value is for things like ethernet over something. Left it off.

root@europa:~# iperf -t 60 -w220k -c 98.229.130.122 
------------------------------------------------------------
Client connecting to 98.229.130.122, TCP port 5001
TCP window size:  220 KByte
------------------------------------------------------------
[  3] local 149.20.63.19 port 49117 connected with 98.229.130.122 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.7 sec  9.50 MBytes  1.31 Mbits/sec

and as usual red does nothing inside of qos scripts, the outer qdiscs do all the work.

root@gw:/etc/config# tc -s qdisc
qdisc pfifo_fast 0: dev se00 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 2470004177 bytes 1906833 pkt (dropped 39, overlimits 0 requeues 588) 
 backlog 0b 0p requeues 588 
qdisc hfsc 1: dev ge00 root refcnt 2 default 30 
 Sent 671105 bytes 9377 pkt (dropped 0, overlimits 117 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc sfq 100: dev ge00 parent 1:10 limit 127p quantum 1514b divisor 1024 perturb 2sec 
 Sent 344884 bytes 4956 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc sfq 200: dev ge00 parent 1:20 limit 127p quantum 1514b divisor 1024 perturb 2sec 
 Sent 45722 bytes 512 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc red 300: dev ge00 parent 1:30 limit 150Kb min 12800b max 38400b ecn 
 Sent 89683 bytes 372 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
  marked 0 early 0 pdrop 0 other 0
qdisc red 400: dev ge00 parent 1:40 limit 150Kb min 12800b max 38400b ecn 
 Sent 190816 bytes 3537 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
  marked 0 early 0 pdrop 0 other 0
qdisc ingress ffff: dev ge00 parent ffff:fff1 ---------------- 
 Sent 22572047 bytes 16460 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc hfsc 1: dev ifb0 root refcnt 2 default 30 
 Sent 22351988 bytes 15603 pkt (dropped 291, overlimits 29212 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc sfq 100: dev ifb0 parent 1:10 limit 127p quantum 1514b divisor 1024 perturb 2sec 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc sfq 200: dev ifb0 parent 1:20 limit 127p quantum 1514b divisor 1024 perturb 2sec 
 Sent 181753 bytes 490 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc red 300: dev ifb0 parent 1:30 limit 150Kb min 12800b max 38400b ecn 
 Sent 22170235 bytes 15113 pkt (dropped 291, overlimits 408 requeues 0) 
 backlog 0b 0p requeues 0 
  marked 117 early 291 pdrop 0 other 0
qdisc red 400: dev ifb0 parent 1:40 limit 150Kb min 12800b max 38400b ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
  marked 0 early 0 pdrop 0 other 0
qdisc mq 0: dev sw00 root 
 Sent 1980985 bytes 22216 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc mq 0: dev gw01 root 
 Sent 793308 bytes 7866 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc mq 0: dev mon.sw00 root 
 Sent 1387796 bytes 7937 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc mq 0: dev gw00 root 
 Sent 1433275 bytes 9257 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc mq 0: dev sw10 root 
 Sent 11511732 bytes 149188 pkt (dropped 0, overlimits 0 requeues 23) 
 backlog 0b 0p requeues 23 
qdisc mq 0: dev gw11 root 
 Sent 792407 bytes 7852 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc mq 0: dev mon.sw10 root 
 Sent 22944 bytes 142 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
qdisc mq 0: dev gw10 root 
 Sent 1170022 bytes 7800 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
Updated by Dave Täht on Sep 6, 2011.
d@bob-desktop:~\$ iperf -t 60 -w220k -c 98.229.130.122
————————————————————
Client connecting to 98.229.130.122, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 220 KByte)
————————————————————
[ 3] local 149.20.63.20 port 56628 connected with 98.229.130.122 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 13.9 MBytes 1.94 Mbits/sec
Updated by Dave Täht on Sep 6, 2011.
The last result was interesting in context, so I setup the cerowrt router (europa) to have a txqueuelen of 1000 for iperf and repeated the test…

then dropped it back to 8, to minimize the skew inflicted by back to back tests.

root@europa:~# ifconfig ge00 txqueuelen 1000
root@europa:~# iperf -t 60 -w220k -c 98.229.130.122 
------------------------------------------------------------
Client connecting to 98.229.130.122, TCP port 5001
TCP window size:  220 KByte
------------------------------------------------------------
[  3] local 149.20.63.19 port 49118 connected with 98.229.130.122 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  13.6 MBytes  1.90 Mbits/sec
root@europa:~# ifconfig ge00 txqueuelen 8
root@europa:~# iperf -t 60 -w220k -c 98.229.130.122 
------------------------------------------------------------
Client connecting to 98.229.130.122, TCP port 5001
TCP window size:  220 KByte
------------------------------------------------------------
[  3] local 149.20.63.19 port 49119 connected with 98.229.130.122 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.6 sec  13.8 MBytes  1.90 Mbits/sec
Updated by Dave Täht on Sep 17, 2011.
We need to comprehensively nail these problems to the ground for rc7.
Updated by Dave Täht on Apr 21, 2012.
Well, we now have enough infrastructure to look into this harder.

This is a static export of the original bufferbloat.net issue database. As such, no further commenting is possible; the information is solely here for archival purposes.

Cerowrt Project

News items
List of Wiki pages
List of bugs
RSS feed

Recent Updates

Oct 20, 2023 Wiki page
What Can I Do About Bufferbloat?
Dec 3, 2022 Wiki page
Codel Wiki
Jun 11, 2022 Wiki page
More about Bufferbloat
Jun 11, 2022 Wiki page
Tests for Bufferbloat
Dec 7, 2021 Wiki page
Getting SQM Running Right

Find us elsewhere

Bufferbloat Mailing Lists
#bufferbloat on Twitter
Google+ group
Archived Bufferbloat pages from the Wayback Machine

Sponsors

Comcast Research Innovation Fund
Nlnet Foundation
Shuttleworth Foundation
GoFundMe

Bufferbloat Related Projects

OpenWrt Project
Congestion Control Blog
Flent Network Test Suite
Sqm-Scripts
The Cake shaper
AQMs in BSD
IETF AQM WG
CeroWrt (where it all started)

Network Performance Related Resources


Jim Gettys' Blog - The chairman of the Fjord
Toke's Blog - Karlstad University's work on bloat
Voip Users Conference - Weekly Videoconference mostly about voip
Candelatech - A wifi testing company that "gets it".