Cerowrt: Cerowrt 3.3.8-10 - fq_codel + wifi latency reductions

Added by Dave Täht about 2 years ago

Cerowrt-3.3.8-10 is stable but forward-looking. It has an outline
towards what a more wifi-bloat-free future would look like. Maybe.

For more details, see the announcement on the mailing list

codel: Codel (coddling) in the news

Added by Dave Täht about 2 years ago

Kamal Mostafa built an Ubuntu 12.4 PPA, and wrote: "The result is very impressive: I see a 30X reduction in ping latency on a fully saturated 10Mbps network, just by switching on the new fq_codel scheduler. For an interactive ssh session over that same saturated 10Mbps network, fq_codel totally eliminates the laggy keyboard response — it feels like there’s no other network traffic at all!"

Steve Gibson, on Steve Gibson's Security Now podcast said:

"They show charts where they show the throughput under these varying conditions, even though they're discarding intelligently, and it's right up between 90 and 100 percent; whereas the traditional algorithms collapse down into the low 10s of percent of overall throughput. It's just - it's fantastic. "

Jim Gettys outlined how The internet is broken, and how to fix it ...

Dave Miller installed CeroWrt

And then there was the truly mind-bending praise

"These extra-ordinary people, today, are silently fixing tomorrow’s internet. And they deserve big props for that. All you technologists who didn’t get a chance to appreciate Nikola Tesla or Dennis Ritchie, here is your chance to do that for some of the real heroes of our time."

I'm not going to let this last bit of exuberance get to me - as I remember how Tesla ended up, after suffering through decades of patent battles. I'm glad Van and Kathie didn't patent codel - and equally glad the reference implementations and sims we did are available under a dual BSD/GPL license, so anyone can use them.

Because we all need a better internet.

And that said, much work remains - analysis and improvements to codel with ecn continues, wifi is still troublesome, 95% of the linux drivers don't have BQL support, and so on.

We're keeping links to useful code and data on - if you are doing any serious analysis of codel and fq_codel, or have it running on a new OS, please let us know.

Cerowrt: 3.3.8-6: A reasonably stable CeroWrt version with fq_codel

Added by Dave Täht about 2 years ago

I just put out and deployed it as my default gw and ran a bunch of tests that it survived.

This is a version after 5 development releases and I'm hoping it proves out stable enough for more deploy.

I'd prefer to test 24 hours but I'm about to start a trip and can't do that. Hopefully after some more testers leap on it we can declare it stable later this week and move on...

Also the source tree is mostly pushed out but a bit of a mess, I don't know if I'll be able to get Cero independently buildable before late next week.


+ update to linux 3.3.8
+ Fix for bind9 CVE
+ switch to netifd
+ a complete resync with openwrt - this includes much new stuff,
including wireless-testing - way too many updates to talk about
without pulling in the commit log
+ memory problem with ath9k appears gone
+ ECN dropping instituted under load
+ fq_codel packet limits
+ There is now 6rd support, totally untested and unconfigured;
+ transmission bittorrent is in there, too
+ fq_codel on all interfaces by default, on wireless using all 4 subqueues

-s on this release: I went for "stable" rather than new features after it cost me too much time.

- I had to rip out opkg signing support, and some ipv6/diffserv classification support in transmission that wasn't fully baked.

- re-running with new values appears to require a reboot first

-The default gui for AQM doesn't work, the one for "qos" uses hfsc + fq_codel (but lacks ipv6 and diffserv support), and the command line has ipv6 and diffserv, but has to be edited and run manually. And perhaps it's use of htb etc can be improved. I get pretty good results on comcast with simple_qos, see speedtest results here:

but not quite as good as I hoped for. However, under heavier loads the fq_codel stuff is working great under netperf with various numbers of threads and classifications and users.

I would hope some folk here run some benchmarks against various things but some cautions - for example - chrome's benchmark tends to hit dns
hard, and cero by default is not using your most local forwarder so it can bottleneck on dns - ways to fix that if you have dnssec is to edit
forwarders.conf to point to your local forwarder, and uncomment the forwarders line in named.conf. If your ISP doesn't
do dnssec yet, disable dnssec and point forwarders.conf to their nameservers - but I otherwise am getting good results.

Also: I would really prefer people clearly identify when they are testing over wireless vs ethernet and until you have a fq_codel and
debloat-script enabled kernel on your laptop, too, I am finding most of the time the bloat is coming from the testing box rather than cerowrt

There are now fq_codel enabled kernels for ubuntu 12.4 and fedora 16 available here:

I look forward to analyzing htb vs hfsc and further tuning of qos-scripts and the simple_qos script. I'm too stupid apparently to
come up with a way to run simple_qos out of the aqm gui... (help wanted)

The new version of quagga-babeld is available in the opkg repository and it has been confirmed to work right with ipv4 mesh interfaces. I
am really looking forward to people trying this and the authentication code now in quagga so we can migrate off of the existing babeld.

Have fun. I am traveling the rest of this week. Patches, benchmarks and data gladly accepted (preferably on the cerowrt-devel list)

Bloat: How do we solve the next big Internet emergency?

Added by Dave Täht about 2 years ago

Jim Gettys writes about the needs for transparency and an action team for handling Internet emergencies as they arise, or preferably, before, in The Next nightmare is coming

Cerowrt: Codel on Cerowrt-3.3.6-2

Added by Dave Täht about 2 years ago

A test release of CeroWrt is now available that has support for Kathie Nichols' and Van Jacobson's new AQM, Codel , and Eric Dumazet's new fair queuing implementation on top of that, fq_codel.

fq_codel is enabled on all interfaces by default. It is vastly simpler than what we were using before (sfqred) and draws upon and improves on the same body of ideas (head drop, fq, timestamping) but now tied to Kathie and Van's blinding insights as to a good drop strategy, and Eric's successor ideas as towards head of queue behavior and cache line optimizations.

There is a script that can be set to your uplink and downlink speeds, but no uci interface for it as yet, nor gui. (help on finishing aqm-scripts and the luci interface gladly accepted)

To see all the chocolately goodness of what fq_codel can do to wired and wireless latency, it would be good for more to play with it.

Benchmarks have been very good thus far, and more benchmarks and analysis are highly desired.


This release suffers from an unrelated bug ( #379 ) and should NOT be installed as your main router. I would love to beat this bug because it's the only prio 1 remaining but thus far, no luck. Under lighter loads CeroWrt appears to work just fine, but that's for me. YMMV.

Get it here:

Cerowrt: Tasty Freeze 2

Added by Dave Täht about 2 years ago

found a bug or two... fixing... might be tuesday or wednesday at the moment...

Cerowrt: Cerowrt 3.3 Tasty freeze

Added by Dave Täht about 2 years ago

The code for Cerowrt 3.3 has been frozen now for about a week. There's some reasons for that, which is why the freeze is "tasty".
(more news on monday)

Multiple routers have stayed up under various workloads for the week. My own criterion before I'm happy is two weeks of uptime under heavy load... has proven pretty stable, but ipv6 was kind of slow. is also pretty stable under all but the highest workloads, and has over double the ipv6 performance as all prior versions. However, we continue to battle with bug #379 which is going to take some more (tedious) work to fix.

in the meanwhile I've taken a little time of to work on... well... you'll find out monday.

Cerowrt: Churn

Added by Dave Täht over 2 years ago

While builds continue for the linux 3.3.0 series, enough bugs have shown up in it for me to encourage new users to stick with the last known stable(ish) release, which was 3.3rc7-5.

I froze cerowrt on march 14th, got a little distracted by a demo of static deterministic nat and the port control protocol (PCP) as part of the on-going ietf meeting, and fell behind on keeping up on patches to openwrt.

When I got caught up today, patches had landed which touched every major subsystem in cerowrt - the ethernet driver, the wireless driver, the kernel, scripting and the gui. Enough broke in the last two weeks of me not paying attention to the continuous integration process to make going forwards or backwards very difficult. Right now, most noticeable, wifi doesn't come up by default.

I am pleased to say that dnsmasq has made great progress in AAAA dns integration, that iptables and iproute 3.3 have landed, and that I think I've made a dent in bug #113 (dnssec), but there's enough broken thus far to force me to pull all the 3.3.0 related releases until some of the stuff flying in loose formation congeals into one piece. That may not be until 3.3.1 is released.

Also making a bit of progress on dhcp-dp, I think.

Hopefully early next week I'll have the pieces pulled together again to where new builds of cerowrt are useable, rather than merely debuggable. And the sources are as available as possible, on github,if you want to chip in. There's plenty of bugs left to fix...

I look forward to having a more serious code freeze sometime towards the middle of April.

Cerowrt: CeroWrt 3.3rc7-5 build available

Added by Rich Brown over 2 years ago

In this CeroWrt-devel message, Dave Täht announced that build 3.3-rc7-5 is available. This build contains the following features/capabilities:

  • Linux 3.3 kernel. Many of the fixes for bufferbloat are being implemented in this 3.3 kernel, so we are tracking these developments carefully.
  • Active Queue Management fixes for bufferbloat including: Byte Queue Limits (BQL - already incorporated into the 3.3 kernel), Stochastic Fair Queueing-Random Early Drop (SFQRED), working ECN, and other queue disciplines
  • IPv6 support. Another major goal of CeroWrt is to make IPv6 networking in the home as simple as IPv4.
  • Babel mesh routing protocol (1.3.1-2 release).
  • DNSSEC and DNSSEC proxying - Secure extensions to the DNS system. Proxying is currently in testing.
  • OpenWrt features. Because we track the OpenWrt code base carefully, we incorporate most of the capabilities of that distribution. We actively push our changes/enhancements back toward the OpenWrt trunk.
  • An attractive web GUI for configuration - LuCI

Read the [[CeroWrt_33_Release_Notes|CeroWrt 3.3 Release Notes]]
Download CeroWrt 3.3 builds:

Cerowrt: CeroWrt 3.3-rc6-2 build available

Added by Rich Brown over 2 years ago

Dave Täht writes the following in the CeroWrt-Devel mailing list:

I put in some time testing the rc6-2 version of cerowrt back in the bloatlab, which, among other nice things, has a dedicated, real /60 ipv6 allocation. I had a chance to fiddle with encrypted wireless (works), the existing mesh network (works), radvd (fiddly), and also setup the new quagga as a 'leaf' babel node, which worked pretty well. Download Read more...

« Previous 1 2 3 4 Next »

Also available in: Atom