Our mail server is suffering from a failed hard disk in the raid array. Replacing now. Hopefully it will come back up.
In the long run, we need to move this box to a virtual somewhere.
CeroWrt News - January 2014¶
The newest build of CeroWrt - 3.10.24-8 - is working quite well for us. There's a Release Note page that gives the current status. Note that, although it has been very stable, this is still an experimental release. It's available at http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.24-8/
The CeroWrt development team has been working to nail down a no-brainer set of instructions for eliminating bufferbloat - the lag/latency that kills voice & video chat, gaming, and overall network responsiveness. The hard part is that optimal configuration of the Smart Queue Management (SQM) link is difficult - there are tons of options an ISP can set. Although CeroWrt can adapt to any of them, it's difficult to find out the exact characteristics of the link you have. Check out the latest version of our instructions at http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_AQM_for_CeroWrt_310
Although the CeroWrt site (http://www.bufferbloat.net/projects/cerowrt/) has been quiet, that doesn't mean that we haven't been working.
The CeroWrt-Devel mailing list (https://lists.bufferbloat.net/listinfo/cerowrt-devel) has been bubbling with lots of energy all summer and fall, and we're getting close to a new release that we can recommend to everyone. Here's what's we've been working on:
- Significant refinement to the CoDel code, to further knock down bufferbloat
- Update to Linux 3.10 kernel that incorporates much of our earlier work on bufferbloat
- All the good effort from the OpenWrt Barrier Breaker development firmware
- And too many other improvements to mention here
You can check out any number of Bufferbloat videos at http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos to learn more about the problem and our solutions.
If you just want good router firmware, purchase a Netgear WNDR3800 and burn the current 3.7.5-2 firmware. You can also follow our activity on the CeroWrt-devel list to see what's happening and learn when experimental builds or solid new releases are available.
What can I say? 3.7.5-2 has been deployed and thoroughly tested at multiple sites, and under high load, and on places like comcast's cable network, and it just. keeps. working.
In this release:
Vastly improved ipv6 naming and interopability, 3 new forms of codel and fq_codel, much better QoS...
And all the chocolately goodness from OpenWrt Barrier Breaker.
More news to come. Thank you all for your support and donations! We couldn't have done it without you.
[[Best Practices for Benchmarking CoDel and FQ CoDel]] contains lots of useful information for people experimenting with the current (Linux 3.6) CoDel and fq_codel implementations. If you don't take care, you may get bad data.
The Internet Society has issued another in its series on Bandwidth Management, in its Technology Roundtable Series. The executive summary highlights the problems that bufferbloat is causing in today's Internet. The full report is available.
*With apologies to Stuart Cheshire...
I've been working fixing bufferbloat now for a really long time, and of late it's been really difficult to keep the lights on and the servers fed. Last month I'd hit bottom, this month is worse. I put up a donations page, and got a totally wonderful level of response - enough to make rent! Then I tripped over an obscure portion of the Amazon EULA, and ended up refunding everyone's money and cancelling your subscriptions.
So, I've put up a new subscribe/donations page for the cerowrt portion of the project, using paypal at:
I won't jimmy wales you'all on further solicitations like this. (I know, that's what I said last month)
But a little more help in getting over this hump would be very nice to have.
Cerowrt-3.3.8-10 is stable but forward-looking. It has an outline
towards what a more wifi-bloat-free future would look like. Maybe.
For more details, see the announcement on the mailing list
Kamal Mostafa built an Ubuntu 12.4 PPA, and wrote: "The result is very impressive: I see a 30X reduction in ping latency on a fully saturated 10Mbps network, just by switching on the new fq_codel scheduler. For an interactive ssh session over that same saturated 10Mbps network, fq_codel totally eliminates the laggy keyboard response — it feels like there’s no other network traffic at all!"
Steve Gibson, on Steve Gibson's Security Now podcast said:
"They show charts where they show the throughput under these varying conditions, even though they're discarding intelligently, and it's right up between 90 and 100 percent; whereas the traditional algorithms collapse down into the low 10s of percent of overall throughput. It's just - it's fantastic. "
Jim Gettys outlined how The internet is broken, and how to fix it ...
Dave Miller installed CeroWrt
"These extra-ordinary people, today, are silently fixing tomorrow’s internet. And they deserve big props for that. All you technologists who didn’t get a chance to appreciate Nikola Tesla or Dennis Ritchie, here is your chance to do that for some of the real heroes of our time."
I'm not going to let this last bit of exuberance get to me - as I remember how Tesla ended up, after suffering through decades of patent battles. I'm glad Van and Kathie didn't patent codel - and equally glad the reference implementations and sims we did are available under a dual BSD/GPL license, so anyone can use them.
Because we all need a better internet.
And that said, much work remains - analysis and improvements to codel with ecn continues, wifi is still troublesome, 95% of the linux drivers don't have BQL support, and so on.
We're keeping links to useful code and data on http://www.bufferbloat.net/projects/codel/wiki - if you are doing any serious analysis of codel and fq_codel, or have it running on a new OS, please let us know.
I just put out http://huchra.bufferbloat.net/~cero1/3.3/3.3.8-6/ and deployed it as my default gw and ran a bunch of tests that it survived.
This is a version after 5 development releases and I'm hoping it proves out stable enough for more deploy.
I'd prefer to test 24 hours but I'm about to start a trip and can't do that. Hopefully after some more testers leap on it we can declare it stable later this week and move on...
Also the source tree is mostly pushed out but a bit of a mess, I don't know if I'll be able to get Cero independently buildable before late next week.
+ update to linux 3.3.8
+ Fix for bind9 CVE
+ switch to netifd
+ a complete resync with openwrt - this includes much new stuff,
including wireless-testing - way too many updates to talk about
without pulling in the commit log
+ memory problem with ath9k appears gone
+ ECN dropping instituted under load
+ fq_codel packet limits
+ There is now 6rd support, totally untested and unconfigured;
+ transmission bittorrent is in there, too
+ fq_codel on all interfaces by default, on wireless using all 4 subqueues
-s on this release: I went for "stable" rather than new features after it cost me too much time.
- I had to rip out opkg signing support, and some ipv6/diffserv classification support in transmission that wasn't fully baked.
- re-running simple_qos.sh with new values appears to require a reboot first
-The default gui for AQM doesn't work, the one for "qos" uses hfsc + fq_codel (but lacks ipv6 and diffserv support), and the command line simple_qos.sh has ipv6 and diffserv, but has to be edited and run manually. And perhaps it's use of htb etc can be improved. I get pretty good results on comcast with simple_qos, see speedtest results here:
but not quite as good as I hoped for. However, under heavier loads the fq_codel stuff is working great under netperf with various numbers of threads and classifications and users.
I would hope some folk here run some benchmarks against various things but some cautions - for example - chrome's benchmark tends to hit dns
hard, and cero by default is not using your most local forwarder so it can bottleneck on dns - ways to fix that if you have dnssec is to edit
forwarders.conf to point to your local forwarder, and uncomment the forwarders line in named.conf. If your ISP doesn't
do dnssec yet, disable dnssec and point forwarders.conf to their nameservers - but I otherwise am getting good results.
Also: I would really prefer people clearly identify when they are testing over wireless vs ethernet and until you have a fq_codel and
debloat-script enabled kernel on your laptop, too, I am finding most of the time the bloat is coming from the testing box rather than cerowrt
There are now fq_codel enabled kernels for ubuntu 12.4 and fedora 16 available here:
I look forward to analyzing htb vs hfsc and further tuning of qos-scripts and the simple_qos script. I'm too stupid apparently to
come up with a way to run simple_qos out of the aqm gui... (help wanted)
The new version of quagga-babeld is available in the opkg repository and it has been confirmed to work right with ipv4 mesh interfaces. I
am really looking forward to people trying this and the authentication code now in quagga so we can migrate off of the existing babeld.
Have fun. I am traveling the rest of this week. Patches, benchmarks and data gladly accepted (preferably on the cerowrt-devel list)
Also available in: Atom