Bufferbloat

What can I say? 3.7.5-2 has been deployed and thoroughly tested at multiple sites, and under high load, and on places like comcast's cable network, and it just. keeps. working.

It's time to retire "sugarland" and get into "Modena". Get the latest firmware or read the Release Notes for details.

In this release:

Vastly improved ipv6 naming and interopability, 3 new forms of codel and fq_codel, much better QoS...

And all the chocolately goodness from OpenWrt Barrier Breaker.

More news to come. Thank you all for your support and donations! We couldn't have done it without you.

[[Best Practices for Benchmarking CoDel and FQ CoDel]] contains lots of useful information for people experimenting with the current (Linux 3.6) CoDel and fq_codel implementations. If you don't take care, you may get bad data.

The Internet Society has issued another in its series on Bandwidth Management, in its Technology Roundtable Series. The executive summary highlights the problems that bufferbloat is causing in today's Internet. The full report is available.

*With apologies to Stuart Cheshire...

I've been working fixing bufferbloat now for a really long time, and of late it's been really difficult to keep the lights on and the servers fed. Last month I'd hit bottom, this month is worse. I put up a donations page, and got a totally wonderful level of response - enough to make rent! Then I tripped over an obscure portion of the Amazon EULA, and ended up refunding everyone's money and cancelling your subscriptions.

So, I've put up a new subscribe/donations page for the cerowrt portion of the project, using paypal at:

http://www.teklibre.com/cerowrt/subscribe.html

I won't jimmy wales you'all on further solicitations like this. (I know, that's what I said last month)

But a little more help in getting over this hump would be very nice to have.

Cerowrt-3.3.8-10 is stable but forward-looking. It has an outline
towards what a more wifi-bloat-free future would look like. Maybe.

For more details, see the announcement on the mailing list

Kamal Mostafa built an Ubuntu 12.4 PPA, and wrote: "The result is very impressive: I see a 30X reduction in ping latency on a fully saturated 10Mbps network, just by switching on the new fq_codel scheduler. For an interactive ssh session over that same saturated 10Mbps network, fq_codel totally eliminates the laggy keyboard response — it feels like there’s no other network traffic at all!"

Steve Gibson, on Steve Gibson's Security Now podcast said:

"They show charts where they show the throughput under these varying conditions, even though they're discarding intelligently, and it's right up between 90 and 100 percent; whereas the traditional algorithms collapse down into the low 10s of percent of overall throughput. It's just - it's fantastic. "

Jim Gettys outlined how The internet is broken, and how to fix it ...

Dave Miller installed CeroWrt

And then there was the truly mind-bending praise

"These extra-ordinary people, today, are silently fixing tomorrow’s internet. And they deserve big props for that. All you technologists who didn’t get a chance to appreciate Nikola Tesla or Dennis Ritchie, here is your chance to do that for some of the real heroes of our time."

I'm not going to let this last bit of exuberance get to me - as I remember how Tesla ended up, after suffering through decades of patent battles. I'm glad Van and Kathie didn't patent codel - and equally glad the reference implementations and sims we did are available under a dual BSD/GPL license, so anyone can use them.

Because we all need a better internet.

And that said, much work remains - analysis and improvements to codel with ecn continues, wifi is still troublesome, 95% of the linux drivers don't have BQL support, and so on.

We're keeping links to useful code and data on http://www.bufferbloat.net/projects/codel/wiki - if you are doing any serious analysis of codel and fq_codel, or have it running on a new OS, please let us know.

I just put out http://huchra.bufferbloat.net/~cero1/3.3/3.3.8-6/ and deployed it as my default gw and ran a bunch of tests that it survived.

This is a version after 5 development releases and I'm hoping it proves out stable enough for more deploy.

I'd prefer to test 24 hours but I'm about to start a trip and can't do that. Hopefully after some more testers leap on it we can declare it stable later this week and move on...

Also the source tree is mostly pushed out but a bit of a mess, I don't know if I'll be able to get Cero independently buildable before late next week.

Features:

+ update to linux 3.3.8
+ Fix for bind9 CVE
+ switch to netifd
+ a complete resync with openwrt - this includes much new stuff,
including wireless-testing - way too many updates to talk about
without pulling in the commit log
+ memory problem with ath9k appears gone
+ ECN dropping instituted under load
+ fq_codel packet limits
+ There is now 6rd support, totally untested and unconfigured;
+ transmission bittorrent is in there, too
+ fq_codel on all interfaces by default, on wireless using all 4 subqueues

-s on this release: I went for "stable" rather than new features after it cost me too much time.

- I had to rip out opkg signing support, and some ipv6/diffserv classification support in transmission that wasn't fully baked.

- re-running simple_qos.sh with new values appears to require a reboot first

-The default gui for AQM doesn't work, the one for "qos" uses hfsc + fq_codel (but lacks ipv6 and diffserv support), and the command line simple_qos.sh has ipv6 and diffserv, but has to be edited and run manually. And perhaps it's use of htb etc can be improved. I get pretty good results on comcast with simple_qos, see speedtest results here:

http://pastebin.com/Fq6G5Q4u

but not quite as good as I hoped for. However, under heavier loads the fq_codel stuff is working great under netperf with various numbers of threads and classifications and users.

I would hope some folk here run some benchmarks against various things but some cautions - for example - chrome's benchmark tends to hit dns
hard, and cero by default is not using your most local forwarder so it can bottleneck on dns - ways to fix that if you have dnssec is to edit
forwarders.conf to point to your local forwarder, and uncomment the forwarders line in named.conf. If your ISP doesn't
do dnssec yet, disable dnssec and point forwarders.conf to their nameservers - but I otherwise am getting good results.

Also: I would really prefer people clearly identify when they are testing over wireless vs ethernet and until you have a fq_codel and
debloat-script enabled kernel on your laptop, too, I am finding most of the time the bloat is coming from the testing box rather than cerowrt
itself!

There are now fq_codel enabled kernels for ubuntu 12.4 and fedora 16 available here:

http://www.bufferbloat.net/projects/codel/wiki

I look forward to analyzing htb vs hfsc and further tuning of qos-scripts and the simple_qos script. I'm too stupid apparently to
come up with a way to run simple_qos out of the aqm gui... (help wanted)

The new version of quagga-babeld is available in the opkg repository and it has been confirmed to work right with ipv4 mesh interfaces. I
am really looking forward to people trying this and the authentication code now in quagga so we can migrate off of the existing babeld.

Have fun. I am traveling the rest of this week. Patches, benchmarks and data gladly accepted (preferably on the cerowrt-devel list)

Jim Gettys writes about the needs for transparency and an action team for handling Internet emergencies as they arise, or preferably, before, in The Next nightmare is coming

A test release of CeroWrt is now available that has support for Kathie Nichols' and Van Jacobson's new AQM, Codel , and Eric Dumazet's new fair queuing implementation on top of that, fq_codel.

fq_codel is enabled on all interfaces by default. It is vastly simpler than what we were using before (sfqred) and draws upon and improves on the same body of ideas (head drop, fq, timestamping) but now tied to Kathie and Van's blinding insights as to a good drop strategy, and Eric's successor ideas as towards head of queue behavior and cache line optimizations.

There is a simple_qos.sh script that can be set to your uplink and downlink speeds, but no uci interface for it as yet, nor gui. (help on finishing aqm-scripts and the luci interface gladly accepted)

To see all the chocolately goodness of what fq_codel can do to wired and wireless latency, it would be good for more to play with it.

Benchmarks have been very good thus far, and more benchmarks and analysis are highly desired.

Caveat:

This release suffers from an unrelated bug ( #379 ) and should NOT be installed as your main router. I would love to beat this bug because it's the only prio 1 remaining but thus far, no luck. Under lighter loads CeroWrt appears to work just fine, but that's for me. YMMV.

Get it here: http://huchra.bufferbloat.net/~cero1/3.3/3.3.6-2/

found a bug or two... fixing... might be tuesday or wednesday at the moment...