CeroWrt Triumphs over Bufferbloat

Added by Rich Brown 7 months ago

The CeroWrt project, and its implementation of fq_codel as seen in the current build of the firmware, eliminates the problem of bufferbloat. These changes have been pushed into Linux kernel and the OpenWrt mainline ("Barrier Breaker" release), and are now widely available.

Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too much data. It has plagued network routers from the early days. The problem was made worse as RAM became cheaper: network engineers worried that dropping packets would make the network slow, so there was an incentive to buffer more and more packets. This had the paradoxical effect of retaining too many packets, which hold up all the traffic behind those buffers.

Many efforts through the 1990s and 2000's attempted to address the problem. Random Early Drop (RED) and its variants showed promise, but didn't monitor the proper variables, and were thus hard to configure properly and would hurt performance if not tuned correctly. Various quality of service (QoS) policies can give priority to certain types of traffic, but they're hard to configure. As traffic types change and evolve, these policies become a maintenance hassle, since they need to be rewritten on a regular basis.

In early 2012, Kathie Nichols took another look at the problem of overbuffered routers and designed the CoDel (pronounced "coddle') algorithm. The major insight was that the best way to avoid "too much buffering" was to monitor a packet's sojourn time - the time elapsed between when it was queued for transmssion and dequeued. If that time exceeds a certain threshold (generally 5 msec), it indicated that the packet had been queued for a long time. CoDel would then drop a percentage of those packet to provide feedback to the sender that it was using more than its share of the available capacity. An elaboration to the CoDel algorithm - fq_codel from Eric Dumazet - placed packets for each source/destination flow in a separate queue, and applied the CoDel algorithm to each queue to extremely good effect.

The resulting fq_codel qdisc was put in to the Linux 3.5 kernel in July 2012.

CeroWrt 3.10.50-1 -- Declaring Victory

Added by Rich Brown 8 months ago

We are extremely pleased to report that CeroWrt has been highly stable since it was built over six months ago. We recently surveyed our user base, and uptimes are terrific - ranging as high as 124 days, with a lion's share of the reports greater than 80 days. Most of the reasons for reboots are due to things like power failures. CeroWrt is meeting its design goals:
  • Reliable, secure, high performance home router
  • Bufferbloat has been controlled with fq_codel and the sqm-scripts
  • IPv6 just works, either from a native provider, such as Comcast, or through a tunnel such as Hurricane Electric
  • DNSSEC just works
  • We've proved the value of routing different interfaces, instead of bridging together the 2.4GHz, 5GHz, and Ethernet interfaces
  • And lots more

The latest CeroWrt 3.10.50-1 was resync'd with the OpenWrt sources on 28 July 2014. Therefore, the CeroWrt builds have ceased to change from that date. You can review the build history from the CeroWrt release notes at:

That said, there are a few important efforts to take into account:

  1. We have aggressively pushed the interesting changes back into the OpenWrt mainline. All these changes are now available through the standard OpenWrt builds.
  2. CeroWrt development is dormant at the moment as we begin to think about the next step - how to "make wi-fi fast". (There are a number of bad behaviors in most wi-fi drivers that lower your wi-fi performance far below what is theoretically possible. We want to fix this.)
  3. OpenWrt has declared victory on their "Barrier Breaker" (BB) firmware evolution based on a Linux 3.10 kernel. They are now pursuing their "Chaos Calmer" (CC) build based on 3.18 (or later) kernel. CC will have all the goodness of BB, plus the new features they're planning.


  • If you already own a Netgear WNDR 3800 (or 3700v2), you certainly won't go wrong with the CeroWrt 3.10.50-1 build. But don't run out and buy one today - they're becoming scarce and expensive.
  • If you're looking for stable, well-supported router firmware for your home, consider the OpenWrt BB build. It's available for a wide variety of routers, and incorporates most of the major capabilities that we put into CeroWrt.
  • If you're willing to put up with a little testing, check out the OpenWrt CC builds. That software is undergoing constant development, tweaks, and enhancements, and contains all the goodness of CeroWrt.
  • If you really want to live on the bleeding edge, join the CeroWrt Developer's list to keep an eye on (or help with!) developments here. In a few months, we're planning to do more work on wi-fi, potentially on new, more available, higher performance hardware.

CeroWrt 3.10.50-1 Released

Added by Rich Brown about 1 year ago

The CeroWrt 3.10.50-1 build has been released. It has several improvements, including resyncing with the OpenWrt head, another fix to wifi that may completely address the problem with bug #442, a GUI for the BCP38 rules, and some fixes to the SQM system.

Get it at:

Update - 21Aug2014: This build has proved very stable, and we strongly recommend that people install it.

CeroWrt latest beta's

Added by Rich Brown about 1 year ago

The most recent beta-test CeroWrt (version 3.10.48-2) is working very well. The incidence of the wifi-related bug #442 has dropped, and it otherwise has been stable for almost two weeks. Get it from

There is a newer version - 3.10.50-1 - that has not been well tested, but it incorporates another wifi bug-fix. If it checks out, this may be close to a release candidate.

Heartbleed and CeroWrt - upgrade please!

Added by Dave Täht over 1 year ago

Heartbleed Update

In response to the heartbleed (CVE-2014-0160) vulnerability, on April
9th 2014 we updated the under-development CeroWrt release to include
the fixed version of openssl. The fix is in CeroWrt 3.10.36-3 and

We have no means of fixing the "stable" (3.7.5) release of CeroWrt,
nor any of the innumerable development releases since then.

Please do a clean, fresh upgrade to CeroWrt 3.10.36-6 or later. [1]
Images are available in:
Reflashing instructions are here:

In the base image, the administration gui of recent CeroWrt versions
depended on openssl (however it is protected by firewall rules to only
be accessible from within your own network), and several optional packages
did also - stunnel - used for "secure" tunneling, and openvpn in particular.

To find out more about the bug go to and/or
see the relevant page on wikipedia:

Heartbleed is one of the most serious bugs that has ever hit the
internet, and in addition to web services, critical network daemons
such as those that manage network printing, logging, monitoring, voip,
chat, tunnels, vpns and email, all can potentially be exploited.

We strongly advise resyncing your source trees with us and distributing
new firmware images containing the updated libraries. All
network facing TLS-using daemons are potentially a risk, as are
any TLS using services exposed behind the firewall.

Once your system is secured again, you should re-issue certs and passwords,
as per:
and check for unverified commits.

Packages maintained in the openwrt core repositories that can be
affected when compiled for openssl2 may include: libevent2,
ustream-ssl, hostapd, openvpn, authsae, luci-ssl, and uhttpd.

Optional network daemons in other repositories such as radsecproxy,
vsftpd, squid, mini_httpd, pure-ftpd, cups, ndyndns, elinks,
libtorrent, monit, nagios, syslog-ng3, boxbackup, rsyncrypto, curl,
cyrus-sasl, openldap, icecast, fetchmail, dovecot, transmission,
stunnel, httptunnel, apache, lighttpd, znc, net-snmp, bitlbee,
asterisk, postfix and openvpn all use TLS level security, are
often linked against openssl, and are thus potentially vulnerable.

Please see the relevant website for each of the products above
for news on their vulnerabilities. Much of the furor over heartbleed
has focused on websites, where notably smtp and imaps and im traffic
has also been shown vulnerable.

Other infrastructure, router and CPE distributions are also affected.

Two examples among many:

Network facing Applications built on top of php4, php5, python, luasec, erlang, ruby
are also potentially affected.

Packages maintained in the ceropackages repository that were potentially
vulnerable are xorp, python-lafs, ccnx, and resiprocate.

Please take this seriously and check your firmware and your products for
usage of the vulnerable openssl versions.

We note also that multiple other serious vulnerabilities have been
fixed in other CeroWrt and OpenWrt packages and in the Linux kernel over
the past years; you should consider fixing those vulnerabilities in
your downstream products and routers while you are at it.

We have long been supportive of adding new features for openwrt to
make it more easily updated in the field, the work could use more
eyeballs and developers, and we need to find resources and funding for
a code audit in the coming months.


[1] Regrettably in the present development branch (3.10.36-4) we are
trying to isolate a wifi bug that crops up after much traffic, we will
announce a fix for that when it arrives. See Bug #442 .

[2] The base as-provided-by OpenWrt base binary installations are not
vulnerable to HeartBleed, as neither the builtin SSH server nor the
optional LuCI SSL support rely on OpenSSL for cryptographic TLS
support. Their Attitude Adjustment release used cyassl as a base,
and the underway Barrier Breaker development series uses PolarSSL
for as many packages as support exists and the GPLv2 license allows.

In other words the OpenSSL library is not installed within the stock
base images available on their download servers, however they too
contain many optional packages that do depend on openssl to function,
and many downstream products may have chosen openssl over those

Check your trees! And if you are having a bad week, perhaps this
will help:

Stay calm and keep on patching!

Insecurity in Home embedded devices 3/19/14 4PM

Added by Dave Täht over 1 year ago

Jim Gettys will be giving a talk at MIT about insecurity
in home devices and what can be done about it.

For more details, see:

CeroWrt News - January 2014

Added by Rich Brown almost 2 years ago

The newest build of CeroWrt - 3.10.24-8 - is working quite well for us. There's a Release Note page that gives the current status. Note that, although it has been very stable, this is still an experimental release. It's available at

The CeroWrt development team has been working to nail down a no-brainer set of instructions for eliminating bufferbloat - the lag/latency that kills voice & video chat, gaming, and overall network responsiveness. The hard part is that optimal configuration of the Smart Queue Management (SQM) link is difficult - there are tons of options an ISP can set. Although CeroWrt can adapt to any of them, it's difficult to find out the exact characteristics of the link you have. Check out the latest version of our instructions at

November 2013: Update on CeroWrt

Added by Rich Brown almost 2 years ago

Although the CeroWrt site ( has been quiet, that doesn't mean that we haven't been working.

The CeroWrt-Devel mailing list ( has been bubbling with lots of energy all summer and fall, and we're getting close to a new release that we can recommend to everyone. Here's what's we've been working on:

  • Significant refinement to the CoDel code, to further knock down bufferbloat
  • Update to Linux 3.10 kernel that incorporates much of our earlier work on bufferbloat
  • All the good effort from the OpenWrt Barrier Breaker development firmware
  • And too many other improvements to mention here

You can check out any number of Bufferbloat videos at to learn more about the problem and our solutions.

If you just want good router firmware, purchase a Netgear WNDR3800 and burn the current 3.7.5-2 firmware. You can also follow our activity on the CeroWrt-devel list to see what's happening and learn when experimental builds or solid new releases are available.

Cerowrt 3.7.5-2 Is the new Stable

Added by Dave Täht over 2 years ago

What can I say? 3.7.5-2 has been deployed and thoroughly tested at multiple sites, and under high load, and on places like comcast's cable network, and it just. keeps. working.

It's time to retire "sugarland" and get into "Modena". Get the latest firmware or read the Release Notes for details.

In this release:

Vastly improved ipv6 naming and interopability, 3 new forms of codel and fq_codel, much better QoS...

And all the chocolately goodness from OpenWrt Barrier Breaker.

More news to come. Thank you all for your support and donations! We couldn't have done it without you.

Paypal for cerowrt subscriptions and donations

Added by Dave Täht about 3 years ago

I've been working fixing bufferbloat now for a really long time, and of late it's been really difficult to keep the lights on and the servers fed. Last month I'd hit bottom, this month is worse. I put up a donations page, and got a totally wonderful level of response - enough to make rent! Then I tripped over an obscure portion of the Amazon EULA, and ended up refunding everyone's money and cancelling your subscriptions.

So, I've put up a new subscribe/donations page for the cerowrt portion of the project, using paypal at:

I won't jimmy wales you'all on further solicitations like this. (I know, that's what I said last month)

But a little more help in getting over this hump would be very nice to have.

1 2 3 Next »

Also available in: Atom