October 14, 2015 06:00 AM Eastern Daylight Time
WASHINGTON--(BUSINESS WIRE)-- In a letter submitted to the Federal Communications Commission (FCC), Dave Täht, co-founder of the Bufferbloat Project, and Dr. Vinton Cerf, co-inventor of the Internet, along with more than 260 other global network and cybersecurity experts, responded to the newly proposed FCC rules laid out in ET Docket No. 15-170 for RF Devices such as Wi-Fi routers by unveiling a new approach to improve the security of these devices and ensure a faster, better, and more secure Internet.
The letter was filed during the agency’s public comment period on this issue.
Dave Farber, former Chief Technologist of the FCC, supports the new approach, stating, “Today there are hundreds of millions of Wi-Fi routers in homes and offices around the globe with severe software flaws that can be easily exploited by criminals. While we agree with the FCC that the rules governing these devices must be updated, we believe the proposed rules laid out by the agency lack critical accountability for the device manufacturers.”
“We can't afford to let any part of the Internet's infrastructure rot in place. We made this proposal because the wireless spectrum must not only be allocated responsibly, but also used responsibly. By requiring a bare minimum of openness in the technology at the edge of the Internet, we'll ensure that any mistakes or cheating are caught early and fixed fast,” said Dr. Vint Cerf, a co-inventor of the Internet and also Senior Vice President and Chief Internet Evangelist at Google.
To improve accountability significantly while keeping the original intent of the regulation, the signatories, who also included Dr. Paul Vixie, Dr. Sascha Meinrath, Dr. Nick Feamster, Jim Gettys, Dr. David P. Reed, Dr. Andreas Petlund, Jeff Osborn, and other well-known industry experts, recommend the FCC mandate the following actions:
- Any vendor of software-defined radio (SDR), wireless, or Wi-Fi radio must make public the full and maintained source code for the device driver and radio firmware in order to maintain FCC compliance. The source code should be in a buildable, change-controlled source code repository on the Internet, available for review and improvement by all.
- The vendor must assure that secure update of firmware be working at time of shipment, and that update streams be under ultimate control of the owner of the equipment. Problems with compliance can then be fixed going forward by the person legally responsible for the router being in compliance.
- The vendor must supply a continuous stream of source and binary updates that must respond to regulatory transgressions and Common Vulnerability and Exposure reports (CVEs) within 45 days of disclosure, for the warranted lifetime of the product, or until five years after the last customer shipment, whichever is longer.
- Failure to comply with these regulations should result in FCC decertification of the existing product and, in severe cases, bar new products from that vendor from being considered for certification.
- Additionally, we ask the FCC to review and rescind any rules for anything that conflicts with open source best practices, produce unmaintainable hardware, or cause vendors to believe they must only ship undocumented “binary blobs” of compiled code or use lockdown mechanisms that forbid user patching. This is an ongoing problem for the Internet community committed to best practice change control and error correction on safety-critical systems.
For the complete letter and the full list of supporters, please visit here.
“Our fight for a free and open Internet began long before the invention and wide use of Wi-Fi home routers, whose manufacturers chose to base on open software. We are at an important inflection point in the history of the Internet. The FCC has an opportunity to take positive action that will increase the security and performance not only of these devices, but also influence how manufacturers develop secure Internet of Things while preserving an open Internet,” said Jim Gettys, Chairman, Bufferbloat Project.
“Networking research and innovation fundamentally depend on the ability to modify firmware on CPE and deploy it in real-world settings in home networks,” said Dr. Nick Feamster, Acting Director of Center for Information Technology Policy at Princeton University.
"The Internet is now effectively a battleground with end-users, our employers, our schools and our vendors on one side, and organized crime and nation-states on the other side. Our home gateways are often repurposed by our adversaries into weapons against us because these small, cheap plastic boxes are unpatchable, abandoned by their makers, and completely opaque. These devices are currently the Internet's public enemy #1. The plan proposed would significantly decontaminate our technology supply chain,” said Dr. Paul Vixie, CEO of Farsight Security, Inc.
“The recommendations in this document would go a long way toward ensuring the existence of a highly performant, secure, and regulation-compliant Internet far into the future,” said Jonathan Corbet, Executive Editor, LWN.net.
“As the recent revelations about the ‘Moon Worm,’ ‘DNSchanger,’ and ‘Misfortune Cookie’ and now the Volkswagen scandal illustrate, secret, locked-down firmware represents a clear and present danger to the security of the Internet,” said Ted Lemon, recent Area Director at the IETF.
“If we raise the bar for firmware code quality, maintenance, and upgrades, we can finish beating bufferbloat, especially on Wi-Fi, deploy IPv6 faster, improve security, and build a vastly better Internet, for everybody,” said Dave Täht, Architect, CeroWrt, co-founder, Bufferbloat Project.
If you care about this important issue and agree with our approach, please contact your local Congressional representative and share our letter with them. For media interview requests or other inquiries, please contact email@example.com.
About the Bufferbloat Project
The Bufferbloat Project is an international coalition of individuals, many who were instrumental in the development of the Internet, and several with Wi-Fi, deeply concerned about the future health, speed, and safety of the edge of the Internet. In operation for 5 years, and working primarily on third-party firmware, it has pioneered new algorithms, boosted safety and security, helped develop new standards, and worked to make as much of this new theory and code available as possible for all to use. For more information, please visit http://www.bufferbloat.net.
Karen Burke, 650-814-3764
Also, read the Supplemental Comments to the original letter at http://www.bufferbloat.net/news/56 These comments couldn't be included in the press release because of length limitations.
“Wi-Fi devices aren’t just radios: they are network devices. The software that governs them impacts the security and reliability of the whole network. If we leave firmware solely in the hands of manufacturers, we close the door to the very independent research that has been advancing the state of Internet technology and security all along.” -- Susan E. Sons, Director, Internet Civil Engineering Institute
"The security implications should be obvious to anyone who's been paying attention to recent headlines. Less obvious is all the research and other work that goes on behind the scenes, by the very people who made the Internet what it is today. Losing the ability to control the software that runs on these devices would bring this kind of research to a halt, and tie the hands of many brilliant minds who have dedicated their lives and careers to the continued improvement of the network we all depend on. If we want to build the fast, secure, reliable Internet of the future, this kind of research must be allowed to continue." -- Jeff Loughlin, Independent Researcher and Open Source Developer
"Leveraging off-the-shelf WiFi hardware plays an important role in academic research in the field. Locked down devices would both independent verification of vendor performance claims and research into improving performance of current and future generation WiFi." -- Toke Høiland-Jørgensen, Bufferbloat researcher at Karlstad University, Sweden
“Dyn provides performance solutions for a significant proportion of the world's most popular web properties, and invests significant resources in maintaining the availability of our global DNS platform in the face of near constant attack. Low-cost (near zero-margin) home and small-business routers, easily compromised, are a significant contributor to attack traffic that we mitigate every day. In the race to the bottom, these devices are effectively unsupported by those who sell them; the ability to harden such devices is essential to the future success of the Internet as a reliable platform for communication and commerce.” -- Joe Abley, Director of Architecture, Dyn
“Software upgradeability and inspection is vitally important, and should not be impaired. It is unimaginable that the FCC would propose anything to impede updates to network elements, and I hope that the words in the petition are considered carefully and used as a template for sane (and light) legislation.” -- John Todd, Senior Technologist at Packet Clearing House
“As the editor of the IETF standards document RFC 7368 “IPv6 Home Networking Architecture Principles”, which describes how future IPv6 home networks can be built, I fully support the Letter submitted to the FCC,. The IETF works on the principle of rough consensus and running code, and the ability to modify and update open source home router images enables innovation and the future advancement of the Internet” -- Dr Tim Chown, Lecturer, University of Southampton (UK)
“While looking at regulation, it would be more useful to consider ensuring vendors adopt IPv6 as they have ignored it for the last 10+ years requiring a complete device replacement. FOSS has enabled many of their devices to be updated rather than dumped in landfill.” -- Brandon Butterworth, Chief Scientist, British Broadcasting Corporation
“The Internet of Things shouldn't be the Internet of Vulnerable Unpatchable Abandoned-by-Manufacturer Things.” -- Will Edwards
"Unlike global climate change, software is a human construct, conceived by engineers. The use and management of it is therefore modifiable to be optimized for future use by qualified people with predictable positive results for security. As such, these issues deserve to have sufficient technical resources and intelligent legal framework to make this happen." -- Randy Resnick, Creator, IP Communications & VoIP Community and VoIP Users Conference
“I was one of the designers of the Minstrel rate control algorithm that is now used for many, many Wi-Fi devices. Minstrel approximately doubled the range and gave a 10x improvement in throughput for Wi-Fi devices of the time, at the same time as decreasing their time-average radiated power. That research would have been prohibited or greatly impeded by the proposed rulemaking. It was done outside of the FCC jurisdiction, in New Zealand, but other countries tend to follow the FCC rule structure, and technical measures to implement the proposed FCC rules would be a serious impediment to doing that sort of work irrespective of the legal aspects. Further, this rule would set a precedent that would inevitably be applied in other areas of technology, and this would cause a great deal of harm to research and innovation. Locked-down firmware on Internet-connected devices stands to become a very serious problem for society, and should not be encouraged in any way. I strongly agree with the proposed alternate path for that reason.” -- Andrew McGregor: Linux WiFi developer (Minstrel rate control), IETF working group chair, AQM developer and SDN researcher
“"Software by its very nature has bugs and will be vulnerable to attack. The only way we can be confident that such a critical piece of our societal infrastructure is as safe as possible is if the source code is made fully available for review and can be fixed by anyone with authority when problems arise. This alternate proposal, which is grounded in a deep understanding of how security issues really play out in this context, should be seriously considered by the FCC." -- Karen M. Sandler, Executive Director, Software Freedom Conservancy
"We are entering an era where an array of interconnected devices will create 'lowest common denominator security' within the most private reaches of our homes. True cybersecurity results from open code, peer review, and user control over their own devices and data; our recommended solutions are fundamentally important to the future growth of the Internet of Things economy.” -- Sascha Meinrath, Palmer Chair in Telecommunications at Penn State University
“With the Internet of Things upon us, regulators can play a vital role in making our hardware secure. By requiring industry-leading engineering practices like we’ve described in this comment, the FCC can ensure Americans have more secure devices which respect the radio spectrum. At the same time, these practices provide makers and inventors with the flexibility necessary for world-leading innovation.” -- Eric Schultz
Washington D.C., October 14th, 2015: In a letter (mirror here) submitted to the Federal Communications Commission (FCC), Dave Täht, co-founder of the Bufferbloat Project, and Dr. Vinton Cerf, co-inventor of the Internet, along with more than 260 other global network and cybersecurity experts, responded to the newly proposed FCC rules laid out in ET Docket No. 15-170 for RF Devices such as Wi-Fi routers by unveiling a new approach to improve the security of these devices and ensure a faster, better, and more secure Internet.
Please read the remainder of the Press Release at: http://www.businesswire.com/news/home/20151014005564/en#.Vh4ojflViko
The CeroWrt project, and its implementation of fq_codel as seen in the current build of the firmware, eliminates the problem of bufferbloat. These changes have been pushed into Linux kernel and the OpenWrt mainline ("Barrier Breaker" release), and are now widely available.
Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too much data. It has plagued network routers from the early days. The problem was made worse as RAM became cheaper: network engineers worried that dropping packets would make the network slow, so there was an incentive to buffer more and more packets. This had the paradoxical effect of retaining too many packets, which hold up all the traffic behind those buffers.
Many efforts through the 1990s and 2000's attempted to address the problem. Random Early Drop (RED) and its variants showed promise, but didn't monitor the proper variables, and were thus hard to configure properly and would hurt performance if not tuned correctly. Various quality of service (QoS) policies can give priority to certain types of traffic, but they're hard to configure. As traffic types change and evolve, these policies become a maintenance hassle, since they need to be rewritten on a regular basis.
In early 2012, Kathie Nichols took another look at the problem of overbuffered routers and designed the CoDel (pronounced "coddle') algorithm. The major insight was that the best way to avoid "too much buffering" was to monitor a packet's sojourn time - the time elapsed between when it was queued for transmssion and dequeued. If that time exceeds a certain threshold (generally 5 msec), it indicated that the packet had been queued for a long time. CoDel would then drop a percentage of those packet to provide feedback to the sender that it was using more than its share of the available capacity. An elaboration to the CoDel algorithm - fq_codel from Eric Dumazet - placed packets for each source/destination flow in a separate queue, and applied the CoDel algorithm to each queue to extremely good effect.
The resulting fq_codel qdisc was put in to the Linux 3.5 kernel in July 2012.
- Reliable, secure, high performance home router
- Bufferbloat has been controlled with fq_codel and the sqm-scripts
- IPv6 just works, either from a native provider, such as Comcast, or through a tunnel such as Hurricane Electric
- DNSSEC just works
- We've proved the value of routing different interfaces, instead of bridging together the 2.4GHz, 5GHz, and Ethernet interfaces
- And lots more
The latest CeroWrt 3.10.50-1 was resync'd with the OpenWrt sources on 28 July 2014. Therefore, the CeroWrt builds have ceased to change from that date. You can review the build history from the CeroWrt release notes at: http://www.bufferbloat.net/projects/cerowrt/wiki/CeroWrt_310_Release_Notes
That said, there are a few important efforts to take into account:
- We have aggressively pushed the interesting changes back into the OpenWrt mainline. All these changes are now available through the standard OpenWrt builds.
- CeroWrt development is dormant at the moment as we begin to think about the next step - how to "make wi-fi fast". (There are a number of bad behaviors in most wi-fi drivers that lower your wi-fi performance far below what is theoretically possible. We want to fix this.)
- OpenWrt has declared victory on their "Barrier Breaker" (BB) firmware evolution based on a Linux 3.10 kernel. They are now pursuing their "Chaos Calmer" (CC) build based on 3.18 (or later) kernel. CC will have all the goodness of BB, plus the new features they're planning.
- If you already own a Netgear WNDR 3800 (or 3700v2), you certainly won't go wrong with the CeroWrt 3.10.50-1 build. But don't run out and buy one today - they're becoming scarce and expensive.
- If you're looking for stable, well-supported router firmware for your home, consider the OpenWrt BB build. It's available for a wide variety of routers, and incorporates most of the major capabilities that we put into CeroWrt.
- If you're willing to put up with a little testing, check out the OpenWrt CC builds. That software is undergoing constant development, tweaks, and enhancements, and contains all the goodness of CeroWrt.
- If you really want to live on the bleeding edge, join the CeroWrt Developer's list https://lists.bufferbloat.net/listinfo/cerowrt-devel to keep an eye on (or help with!) developments here. In a few months, we're planning to do more work on wi-fi, potentially on new, more available, higher performance hardware.
The CeroWrt 3.10.50-1 build has been released. It has several improvements, including resyncing with the OpenWrt head, another fix to wifi that may completely address the problem with bug #442, a GUI for the BCP38 rules, and some fixes to the SQM system.
Update - 21Aug2014: This build has proved very stable, and we strongly recommend that people install it.
The most recent beta-test CeroWrt (version 3.10.48-2) is working very well. The incidence of the wifi-related bug #442 has dropped, and it otherwise has been stable for almost two weeks. Get it from http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.48-2/
There is a newer version - 3.10.50-1 - that has not been well tested, but it incorporates another wifi bug-fix. If it checks out, this may be close to a release candidate.
In response to the heartbleed (CVE-2014-0160) vulnerability, on April
9th 2014 we updated the under-development CeroWrt release to include
the fixed version of openssl. The fix is in CeroWrt 3.10.36-3 and
We have no means of fixing the "stable" (3.7.5) release of CeroWrt,
nor any of the innumerable development releases since then.
Please do a clean, fresh upgrade to CeroWrt 3.10.36-6 or later. 
Images are available in: http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/
Reflashing instructions are here:
In the base image, the administration gui of recent CeroWrt versions
depended on openssl (however it is protected by firewall rules to only
be accessible from within your own network), and several optional packages
did also - stunnel - used for "secure" tunneling, and openvpn in particular.
Heartbleed is one of the most serious bugs that has ever hit the
internet, and in addition to web services, critical network daemons
such as those that manage network printing, logging, monitoring, voip,
chat, tunnels, vpns and email, all can potentially be exploited.
We strongly advise resyncing your source trees with us and distributing
new firmware images containing the updated libraries. All
network facing TLS-using daemons are potentially a risk, as are
any TLS using services exposed behind the firewall.
Once your system is secured again, you should re-issue certs and passwords,
as per: https://www.eff.org/deeplinks/2014/04/bleeding-hearts-club-heartbleed-recovery-system-administrators
and check for unverified commits.
Packages maintained in the openwrt core repositories that can be
affected when compiled for openssl2 may include: libevent2,
ustream-ssl, hostapd, openvpn, authsae, luci-ssl, and uhttpd.
Optional network daemons in other repositories such as radsecproxy,
vsftpd, squid, mini_httpd, pure-ftpd, cups, ndyndns, elinks,
libtorrent, monit, nagios, syslog-ng3, boxbackup, rsyncrypto, curl,
cyrus-sasl, openldap, icecast, fetchmail, dovecot, transmission,
stunnel, httptunnel, apache, lighttpd, znc, net-snmp, bitlbee,
asterisk, postfix and openvpn all use TLS level security, are
often linked against openssl, and are thus potentially vulnerable.
Please see the relevant website for each of the products above
for news on their vulnerabilities. Much of the furor over heartbleed
has focused on websites, where notably smtp and imaps and im traffic
has also been shown vulnerable.
Other infrastructure, router and CPE distributions are also affected.
Two examples among many:
Network facing Applications built on top of php4, php5, python, luasec, erlang, ruby
are also potentially affected.
Packages maintained in the ceropackages repository that were potentially
vulnerable are xorp, python-lafs, ccnx, and resiprocate.
Please take this seriously and check your firmware and your products for
usage of the vulnerable openssl versions.
We note also that multiple other serious vulnerabilities have been
fixed in other CeroWrt and OpenWrt packages and in the Linux kernel over
the past years; you should consider fixing those vulnerabilities in
your downstream products and routers while you are at it.
We have long been supportive of adding new features for openwrt to
make it more easily updated in the field, the work could use more
eyeballs and developers, and we need to find resources and funding for
a code audit in the coming months.
 Regrettably in the present development branch (3.10.36-4) we are
trying to isolate a wifi bug that crops up after much traffic, we will
announce a fix for that when it arrives. See Bug #442 .
 The base as-provided-by OpenWrt base binary installations are not
vulnerable to HeartBleed, as neither the builtin SSH server nor the
optional LuCI SSL support rely on OpenSSL for cryptographic TLS
support. Their Attitude Adjustment release used cyassl as a base,
and the underway Barrier Breaker development series uses PolarSSL
for as many packages as support exists and the GPLv2 license allows.
In other words the OpenSSL library is not installed within the stock
base images available on their download servers, however they too
contain many optional packages that do depend on openssl to function,
and many downstream products may have chosen openssl over those
Check your trees! And if you are having a bad week, perhaps this
will help: http://www.taht.net/~mtaht/uncle_bills_helicopter.html
Stay calm and keep on patching!
Jim Gettys will be giving a talk at MIT about insecurity
in home devices and what can be done about it.
For more details, see:
The newest build of CeroWrt - 3.10.24-8 - is working quite well for us. There's a Release Note page that gives the current status. Note that, although it has been very stable, this is still an experimental release. It's available at http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.24-8/
The CeroWrt development team has been working to nail down a no-brainer set of instructions for eliminating bufferbloat - the lag/latency that kills voice & video chat, gaming, and overall network responsiveness. The hard part is that optimal configuration of the Smart Queue Management (SQM) link is difficult - there are tons of options an ISP can set. Although CeroWrt can adapt to any of them, it's difficult to find out the exact characteristics of the link you have. Check out the latest version of our instructions at http://www.bufferbloat.net/projects/cerowrt/wiki/Setting_up_AQM_for_CeroWrt_310
Also available in: Atom