CeroWrt News - January 2014¶
The newest build of CeroWrt - 3.10.24-8 - is working quite well for us. There's a Release Note page that gives the current status. Note that, although it has been very stable, this is still an experimental release. It's available at http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.24-8/
The CeroWrt development team has been working to nail down a no-brainer set of instructions for eliminating bufferbloat - the lag/latency that kills voice & video chat, gaming, and overall network responsiveness. The hard part is that optimal configuration of the Smart Queue Management (SQM) link is difficult - there are tons of options an ISP can set. Although CeroWrt can adapt to any of them, it's difficult to find out the exact characteristics of the link you have. Check out the latest version of our instructions at http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_AQM_for_CeroWrt_310
Although the CeroWrt site (http://www.bufferbloat.net/projects/cerowrt/) has been quiet, that doesn't mean that we haven't been working.
The CeroWrt-Devel mailing list (https://lists.bufferbloat.net/listinfo/cerowrt-devel) has been bubbling with lots of energy all summer and fall, and we're getting close to a new release that we can recommend to everyone. Here's what's we've been working on:
- Significant refinement to the CoDel code, to further knock down bufferbloat
- Update to Linux 3.10 kernel that incorporates much of our earlier work on bufferbloat
- All the good effort from the OpenWrt Barrier Breaker development firmware
- And too many other improvements to mention here
You can check out any number of Bufferbloat videos at http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos to learn more about the problem and our solutions.
If you just want good router firmware, purchase a Netgear WNDR3800 and burn the current 3.7.5-2 firmware. You can also follow our activity on the CeroWrt-devel list to see what's happening and learn when experimental builds or solid new releases are available.
What can I say? 3.7.5-2 has been deployed and thoroughly tested at multiple sites, and under high load, and on places like comcast's cable network, and it just. keeps. working.
In this release:
Vastly improved ipv6 naming and interopability, 3 new forms of codel and fq_codel, much better QoS...
And all the chocolately goodness from OpenWrt Barrier Breaker.
More news to come. Thank you all for your support and donations! We couldn't have done it without you.
I've been working fixing bufferbloat now for a really long time, and of late it's been really difficult to keep the lights on and the servers fed. Last month I'd hit bottom, this month is worse. I put up a donations page, and got a totally wonderful level of response - enough to make rent! Then I tripped over an obscure portion of the Amazon EULA, and ended up refunding everyone's money and cancelling your subscriptions.
So, I've put up a new subscribe/donations page for the cerowrt portion of the project, using paypal at:
I won't jimmy wales you'all on further solicitations like this. (I know, that's what I said last month)
But a little more help in getting over this hump would be very nice to have.
Cerowrt-3.3.8-10 is stable but forward-looking. It has an outline
towards what a more wifi-bloat-free future would look like. Maybe.
For more details, see the announcement on the mailing list
I just put out http://huchra.bufferbloat.net/~cero1/3.3/3.3.8-6/ and deployed it as my default gw and ran a bunch of tests that it survived.
This is a version after 5 development releases and I'm hoping it proves out stable enough for more deploy.
I'd prefer to test 24 hours but I'm about to start a trip and can't do that. Hopefully after some more testers leap on it we can declare it stable later this week and move on...
Also the source tree is mostly pushed out but a bit of a mess, I don't know if I'll be able to get Cero independently buildable before late next week.
+ update to linux 3.3.8
+ Fix for bind9 CVE
+ switch to netifd
+ a complete resync with openwrt - this includes much new stuff,
including wireless-testing - way too many updates to talk about
without pulling in the commit log
+ memory problem with ath9k appears gone
+ ECN dropping instituted under load
+ fq_codel packet limits
+ There is now 6rd support, totally untested and unconfigured;
+ transmission bittorrent is in there, too
+ fq_codel on all interfaces by default, on wireless using all 4 subqueues
-s on this release: I went for "stable" rather than new features after it cost me too much time.
- I had to rip out opkg signing support, and some ipv6/diffserv classification support in transmission that wasn't fully baked.
- re-running simple_qos.sh with new values appears to require a reboot first
-The default gui for AQM doesn't work, the one for "qos" uses hfsc + fq_codel (but lacks ipv6 and diffserv support), and the command line simple_qos.sh has ipv6 and diffserv, but has to be edited and run manually. And perhaps it's use of htb etc can be improved. I get pretty good results on comcast with simple_qos, see speedtest results here:
but not quite as good as I hoped for. However, under heavier loads the fq_codel stuff is working great under netperf with various numbers of threads and classifications and users.
I would hope some folk here run some benchmarks against various things but some cautions - for example - chrome's benchmark tends to hit dns
hard, and cero by default is not using your most local forwarder so it can bottleneck on dns - ways to fix that if you have dnssec is to edit
forwarders.conf to point to your local forwarder, and uncomment the forwarders line in named.conf. If your ISP doesn't
do dnssec yet, disable dnssec and point forwarders.conf to their nameservers - but I otherwise am getting good results.
Also: I would really prefer people clearly identify when they are testing over wireless vs ethernet and until you have a fq_codel and
debloat-script enabled kernel on your laptop, too, I am finding most of the time the bloat is coming from the testing box rather than cerowrt
There are now fq_codel enabled kernels for ubuntu 12.4 and fedora 16 available here:
I look forward to analyzing htb vs hfsc and further tuning of qos-scripts and the simple_qos script. I'm too stupid apparently to
come up with a way to run simple_qos out of the aqm gui... (help wanted)
The new version of quagga-babeld is available in the opkg repository and it has been confirmed to work right with ipv4 mesh interfaces. I
am really looking forward to people trying this and the authentication code now in quagga so we can migrate off of the existing babeld.
Have fun. I am traveling the rest of this week. Patches, benchmarks and data gladly accepted (preferably on the cerowrt-devel list)
A test release of CeroWrt is now available that has support for Kathie Nichols' and Van Jacobson's new AQM, Codel , and Eric Dumazet's new fair queuing implementation on top of that, fq_codel.
fq_codel is enabled on all interfaces by default. It is vastly simpler than what we were using before (sfqred) and draws upon and improves on the same body of ideas (head drop, fq, timestamping) but now tied to Kathie and Van's blinding insights as to a good drop strategy, and Eric's successor ideas as towards head of queue behavior and cache line optimizations.
There is a simple_qos.sh script that can be set to your uplink and downlink speeds, but no uci interface for it as yet, nor gui. (help on finishing aqm-scripts and the luci interface gladly accepted)
To see all the chocolately goodness of what fq_codel can do to wired and wireless latency, it would be good for more to play with it.
Benchmarks have been very good thus far, and more benchmarks and analysis are highly desired.
This release suffers from an unrelated bug ( #379 ) and should NOT be installed as your main router. I would love to beat this bug because it's the only prio 1 remaining but thus far, no luck. Under lighter loads CeroWrt appears to work just fine, but that's for me. YMMV.
Get it here: http://huchra.bufferbloat.net/~cero1/3.3/3.3.6-2/
found a bug or two... fixing... might be tuesday or wednesday at the moment...
The code for Cerowrt 3.3 has been frozen now for about a week. There's some reasons for that, which is why the freeze is "tasty".
(more news on monday)
Multiple routers have stayed up under various workloads for the week. My own criterion before I'm happy is two weeks of uptime under heavy load...
http://huchra.bufferbloat.net/~cero1/3.3/3.3.3-2/ has proven pretty stable, but ipv6 was kind of slow.
http://huchra.bufferbloat.net/~cero1/3.3/dev/3.3.4-3/ is also pretty stable under all but the highest workloads, and has over double the ipv6 performance as all prior versions. However, we continue to battle with bug #379 which is going to take some more (tedious) work to fix.
in the meanwhile I've taken a little time of to work on... well... you'll find out monday.
While builds continue for the linux 3.3.0 series, enough bugs have shown up in it for me to encourage new users to stick with the last known stable(ish) release, which was 3.3rc7-5.
I froze cerowrt on march 14th, got a little distracted by a demo of static deterministic nat and the port control protocol (PCP) as part of the on-going ietf meeting, and fell behind on keeping up on patches to openwrt.
When I got caught up today, patches had landed which touched every major subsystem in cerowrt - the ethernet driver, the wireless driver, the kernel, scripting and the gui. Enough broke in the last two weeks of me not paying attention to the continuous integration process to make going forwards or backwards very difficult. Right now, most noticeable, wifi doesn't come up by default.
I am pleased to say that dnsmasq has made great progress in AAAA dns integration, that iptables and iproute 3.3 have landed, and that I think I've made a dent in bug #113 (dnssec), but there's enough broken thus far to force me to pull all the 3.3.0 related releases until some of the stuff flying in loose formation congeals into one piece. That may not be until 3.3.1 is released.
Also making a bit of progress on dhcp-dp, I think.
Hopefully early next week I'll have the pieces pulled together again to where new builds of cerowrt are useable, rather than merely debuggable. And the sources are as available as possible, on github,if you want to chip in. There's plenty of bugs left to fix...
I look forward to having a more serious code freeze sometime towards the middle of April.
Also available in: Atom