Bug #402

Tail drop behavior on fq_codel on ack-mostly streams

Added by Dave Täht on Jul 12, 2012. Updated on May 14, 2014.
Closed Urgent Dave Täht

Description

In order to reduce memory requirements I reduced the packet limits for the 24 fq_codel queues to anywhere between 100 and 600 packets depending on type. This, I reasoned, would be enough for most traffic loads.

I missed the case of ack mostly traffic, which as it has a 20x1 ratio vs big packets, requires that the queues be
much larger. On an ACK-heavy load, at high bandwidths, we end up with fq_codel in an “interesting” drop tail state, where it still admits packets but ends up optimizing for “new” flows that aren’t actually “new”.

I do not know the right values. The fq_codel default of 10k packets is sufficient to run the router out of memory in a variety of scenarios, resulting in a crash.

History

Updated by Dave Täht on Jul 12, 2012.
So to add clarity here (see also #401) when I do non-ack mostly streams I run the box out of memory (can’t observe this, not necessarily a valid conclusion) and queue depth, after I went back to pure codel I can still crash…

netperf -Y CS5,CS5 -l 60 -H 172.20.1.1 -t TCP_MAERTS &
netperf -Y CS1,CS1 -l 60 -H 172.20.1.1 -t TCP_MAERTS &
#netperf -Y EF,EF -l 60 -H 172.20.1.1 -t TCP_MAERTS &
netperf -Y EF,EF -l 60 -H 172.20.1.1 -t TCP_MAERTS &
#netperf -Y EF,EF -l 60 -H 172.20.1.1 -t TCP_MAERTS &
#netperf -Y EF,EF -l 60 -H 172.20.1.1 -t TCP_MAERTS &
netperf -Y CS0,CS0 -l 60 -H 172.20.1.1 -t TCP_MAERTS &

I may well have an interaction with netperf running on the box too.

root@testbox:~# tc -s qdisc show dev gw11
qdisc mq 1: root 
 Sent 349384011 bytes 231149 pkt (dropped 491, overlimits 0 requeues 26191) 
 backlog 56210b 40p requeues 26191 
qdisc codel 10: parent 1:1 limit 1000p target 5.0ms interval 100.0ms 
 Sent 344597768 bytes 227688 pkt (dropped 457, overlimits 0 requeues 25833) 
 backlog 7570b 5p requeues 25833 
  count 1 lastcount 1 ldelay 354us drop_next 0us
  maxpacket 1514 ecn_mark 0 drop_overlimit 0
qdisc codel 20: parent 1:2 limit 1000p target 10.0ms interval 100.0ms 
 Sent 4260478 bytes 2827 pkt (dropped 0, overlimits 0 requeues 271) 
 backlog 45420b 30p requeues 271 
  count 0 lastcount 0 ldelay 19.6ms drop_next 0us
  maxpacket 1514 ecn_mark 0 drop_overlimit 0
qdisc codel 30: parent 1:3 limit 1000p target 20.0ms interval 100.0ms 
 Sent 525617 bytes 632 pkt (dropped 34, overlimits 0 requeues 87) 
 backlog 3220b 5p requeues 87 
  count 32 lastcount 23 ldelay 16.7ms drop_next 0us
  maxpacket 1514 ecn_mark 0 drop_overlimit 0
qdisc codel 40: parent 1:4 limit 1000p target 5.0ms interval 100.0ms 
 Sent 148 bytes 2 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
  count 0 lastcount 0 ldelay 0us drop_next 0us
  maxpacket 256 ecn_mark 0 drop_overlimit 0
root@testbox:~# Write failed: Broken pipe
d@ida:~$ ~~~
Updated by Dave Täht on Jul 12, 2012.
partially a heisenbug, when I drive this this hard with netserver, that consumes a ton more ram than usual.

Usually I have a test box and not netperf running on the other side of the router.

Updated by Dave Täht on Jul 12, 2012.
At the observed wireless bandwidths with packet limit 1200 I see fq_codel peak at around 700 packets and not enter drop tail.

Figuring out what those bandwidths actually are and the interrelationship between the device queuing, EDCA, and fq_codel is going to be the subject of a long month or three. Or longer.

Updated by Dave Täht on Aug 10, 2012.
Updated by Dave Täht on May 14, 2014.
1000 packets is enough at most speeds.

This is a static export of the original bufferbloat.net issue database. As such, no further commenting is possible; the information is solely here for archival purposes.

Cerowrt Project

News items
List of Wiki pages
List of bugs
RSS feed

Recent Updates

Oct 20, 2023 Wiki page
What Can I Do About Bufferbloat?
Dec 3, 2022 Wiki page
Codel Wiki
Jun 11, 2022 Wiki page
More about Bufferbloat
Jun 11, 2022 Wiki page
Tests for Bufferbloat
Dec 7, 2021 Wiki page
Getting SQM Running Right

Find us elsewhere

Bufferbloat Mailing Lists
#bufferbloat on Twitter
Google+ group
Archived Bufferbloat pages from the Wayback Machine

Sponsors

Comcast Research Innovation Fund
Nlnet Foundation
Shuttleworth Foundation
GoFundMe

Bufferbloat Related Projects

OpenWrt Project
Congestion Control Blog
Flent Network Test Suite
Sqm-Scripts
The Cake shaper
AQMs in BSD
IETF AQM WG
CeroWrt (where it all started)

Network Performance Related Resources


Jim Gettys' Blog - The chairman of the Fjord
Toke's Blog - Karlstad University's work on bloat
Voip Users Conference - Weekly Videoconference mostly about voip
Candelatech - A wifi testing company that "gets it".