Bloat: CACM Bufferbloat paper and case study is out. (108 comments)
CACM Bufferbloat paper is out. It's the same as the ACM Queue version, but killing trees... You too can read it in the comfort of your bathroom.
The case study conversation led by Vint Cerf is in the February issue, but that hasn't arrived in at least my mailbox yet.
FWIW: they should be fixing the access on the ACM digital library site to be public, once I deal with copyright stuff. But the ACM queue links are just fine for now.
Two articles have just been posted on the ACM Queue Web site as part of a Bufferbloat case study. These are an article by Jim Gettys and Kathleen Nichols, and Vint Cerf interviewing Van Jacobson, Jim Gettys, and Nick Weaver.
These will hit your snail mailbox in the January 2012 CACM (Communications of the ACM).The publications are:
You can find it online here.
It is copyright IEEE; it is used with their permission on this web site.
Bloat: Server maintenence today - Monday, April 25 (1 comment)
Bufferbloat.net has grown by leaps and bounds and we are in need of replacing the email server and migrating a few services over to our new, beefier box, "huchra".
Email, web, and build services will be disrupted periodically.
I'd intended to write up summaries of bufferbloat related activity once a month, but am running a bit behind. Both JG and I have been travelling heavily.
There's been a lot going on under the covers!
Probably the biggest news is that we are working with Georgia Tech on their bismark project. They are out to diagnoise the Internet and we are out to fix it. The two goals seemed compatible. In particular: we are trying to de-"heisenbug" the test routers so they can accurately test the upstream services.
We've also taken the wraps off the "uberwrt" project2, which is an attempt to get the debloating work TESTED in realistic situations at the edge and also into openwrt. (Some work from this also flows into bismark)
I was going to write formal joint press releases on these but have been too busy traveling, talking and hacking. (if anyone wants to step up to handle PR?)
Although traffic on the bloat mailing list has been slow of late, the bismark-devel list has been hopping. Feel free to join bismark/uberwrt projects and/or the mailing list3, especially if you are interested in embedded hardware.
Moving on to other topics...
Based on the early difficulties in getting debloat-testing to be a useful base for the eBDP and A* algorithms, we started looking around for ONE driver to work with and have settled on ath9k hardware (for now) as a base for routers and wireless cards.  We need to do a little testing of the laptop cards, but things are looking good. the WNDR3700v2 is AWESOME, actually. 16MB of flash. LUXURY.
Dan Siemon's pfifo_fast fix for ECN has been backported into 2.6.37.X for openwrt's git head as of Saturday. It's also now part of 2.6.39 and 2.6.38 stable.
SFB is in mainline Linux 2.6.39-RCX and woefully undertested in its current incarnation.
Felix Feitkeu has some patches more fully instrumenting the ath9k driver (when mildly more complete, these should get slammed into debloat-testing as well) 
Dan Siemon has improved both his TC shaper test scripts and ping-exp 
Media: There were a couple articles on bufferbloat that went by this month, I think they were all covered on this list...
There are 236 members of the bloat list now.
We are moving a ton of work to a new build server and also moving the lists machine to that. Regrettably as I write, "huchra" is down due to finger-foo. It should be back up again Monday.
Multiple other servers in other locations are in the queue. I hope to get that sorted out with isc while I am in California.
JG will be in California April 25-30. I will also be in California April 25-30 (in at least one of the same places as JG), and am available for additional talks/coding/consulting/etc along the western seaboard in early May if anyone wants me and can cover my expenses. (Sort of scheduled: Byte and Atheros U) I'll also be visiting Seattle at some point in May, too.
Travel last month:
I spent a week in florida gathering strength for my world tour. Then I spent a week with Georgia Tech helping get their Bismark project off the ground and hammering out workflow issues.
I was tickled pink when I gave an introductory talk on bufferbloat to a class there, only to discover when Q&A rolled around that everyone participating was already up to speed on bufferbloat and queueing disciplines, and peppered me with questions on SFB, RED, eBDP and other algorithms we are playing with. 3 months ago I would have been met with blank looks, now it's a struggle to keep up!
I then spent a week with esr getting one of the first near-complete builds of the wndr3700 working well, working on gpsd (wanted accurate time on openwrt) and rsnapshot and split dns and a host of semi-bufferbloat.net-and-uberwrt issues... And we also got a revised version of the intro to bufferbloat document up on the wiki .
I'm very happy to see thyrsus.com go ipv6 enabled.
The bufferbloat wiki is still in dire need of love, see the Todo list for more details  -
And that's all the news I can remember this late Sunday evening. It's my hope that SFB will make it into bismark/uberwrt this week so we can test SFB a little more while it is still a RC in 2.6.39. I'd VERY MUCH like to make sure SFB works when it is released to millions of users worldwide. That will be in 4 weeks or so... I'm feeling a little schedule pressure here... See dan siemon's scripts... 
 http://www.bufferbloat.net/projects/bismark/wiki Georgia Tech's project
 https://lists.bufferbloat.net/listinfo/ (bismark, bismark-devel)
 http://git.coverfire.com/ PLEASE PLAY WITH TC, SFB, and PING-EXP!!!!
The bandwidth you save may be your own.
 http://battlemesh.org/ has summaries and videos from the battlemesh
 The original of the bufferbloat introductory piece was extensively discussed on this mailing list. This versions incorporates most of those changes. If you don't like this version... It's a wiki document now! Please feel free to fix, extend, and add links! http://www.bufferbloat.net/projects/bloat/wiki/Introduction
 LOTS of writing left http://www.bufferbloat.net/projects/bloat/wiki/ToDo
 After evaluating multiple routers, http://www.bufferbloat.net/projects/uberwrt/wiki/Hardware_evaluation
seemed like the best choice
Bloat-announce mailing list
Jim Gettys will be giving two talks at the Battlemesh conference March 16-19, in Sant Bartomeu del Grau (SBG for short), near Barcelona, Catalunia, Spain.
One will be a condensed version of the existing Bell Labs bufferbloat talk, the other will be about the OLPC meltdowns.
There are also interesting talks scheduled about babel-z, the mesh potato, and other topics of interest to the mesh networking community.
Not to mention the battle of the mesh technologies! May the best debloated router win!
I've put a fairly large (and yet still incomplete) summary of the voip conference call with the freeswitch folk up at:
The call went on for over 2 hours as we discussed all aspects of bufferbloat as it applies to telephony. The call was recorded, and here is a link to the recording:
If you have a listen, and hear something interesting that I did not document, please update the wiki page!
Some interesting tidbits:
- Some freeswitch users are using TCP Vegas for their servers due to the
latency sensitivity of that form of TCP and their need to prioritize
- getting time right on various virtual server technologies is doable, but hard
- freeswitch handled 32+ people in the conference using a variety of
dial-in technologies really well.
- Their wiki was very wrong on recomendations for voip servers (corrected)
I'm thinking that doing a weekly conference call among ourselves here might be useful. Thoughts?
Again, way more detail on the call is at:
Bloat: VOIP concall with the Freeswitch folk 1PM EST Mar 9 (2 comments)
The members of the Bufferbloat project are invited to sit in on Freeswitch's regular VOIP conference call at 1PM Eastern Time, Wednesday, March 9th. Please dial into:
There will also be chatter on the appropriate irc channels as we attempt a melding of the minds. More details to follow.
FreeSWITCH is a scalable, open source, cross-platform soft-switch that allows for consolidating various forms of communication media. It is used primarily for telephony applications - VoIP and traditional PSTN - but also supports video, chat, and other forms of communication.
FreeSWITCH was created in 2006 in response to the need for an open source, freely available telephony platform that could fill the void left by proprietary commercial offerings. Anthony Minessale is the chief architect and lead programmer on the FreeSWITCH project. He is joined by two veteran developers - Brian K West and Michael Jerris. All three of these experienced engineers spent a number of years developing for the Asterisk PBX project before starting FreeSWITCH.
FreeSWITCH was designed with the goals of modularity, stability, and scalability. It is now used in thousands of servers to power VoIP communications for business and organizations around the world, and runs on nearly every operating system.
About the Bufferbloat project¶
If you are not already aware of what bufferbloat is, the fastest way to get up to speed is to peruse the slides and listen to 25 minutes of audio before the conference call, at:
It would be great to have an informed audience so we can only touch lightly on the preliminaries and then dive deeper into bufferbloat and VOIP issues. See also Jim Gettys' blog postings2, discussions on lwn.net3, and slashdot4, and elsewhere5, as well as two very busy mailing lists6
In short, the bufferbloat problem is that there are really big, bloated network buffers in many (especially new) routers, home (mostly wireless) gateways, hosts, and ADSL/FIOS/cable modems that can dramatically affect VOIP performance.
The bufferbloat project7 is attempting to identify equipment and software where truly bloated buffers exist, and mitigate or fix the issues with new software algorithms and heightened awareness. Recently we've released a debloat-testing Linux kernel8 that may help in some cases. Bufferbloat is not a Linux-specific problem, it exists in all OSes, and may become more acute as Windows 7 gets rolled out.
The Bufferbloat project news summary, February 17 - March 3, 2011¶
I'm going to try to put out summary news like this once a month in
the future. This is just a summary of the past two weeks. For a summary
of the first two weeks, see here .
In March, I hope we start to get some patches tested by lots more people,
start getting the qdisc folk to talk to the driver folk (and vice versa)
and continue to unify several separate lines of development. I hope also
we improve the wiki, and continue to recruit more people with different
layers of perspectives and understandings of the problem. Late this month
a new (8 core) server will come online at isc.org for use for building
both debloat-testing and openwrt based kernels for various devices. More
servers and locations will follow.
Extremely high on the list is getting more testing tools modified not only
to detect bufferbloat, but test the new kernel features to see how well they
The Debloat-testing Linux
kernel tree has been updated to 2.6.38-rc7-db. John Linville made an abortive
attempt to move it into the qdisc layer. There are other issues...
41 people have registered on the bufferbloat project web site .
I seem to be the only one making edits. Is there something broken?
35 people (and one robot) regularly in the #bufferbloat irc channel on chat.freenode.net.
References to "bufferbloat" on google doubled from roughly 30,000 to 61,200 results.
(Bufferbloat is now .00053% as popular as sex)
Web hits on the presentations and site itself are not yet available.
We survived slashdot!¶
The thoroughly debloated servers bufferbloat.net is running on survived both a
discussion on lwn.net and a slo-mo slashdotting on the
Got (bufferbloat) Bloat thread.
Slashdot unfortunately got the link to the debloat-testing tree wrong, and never corrected it.
Blog posts & media coverage related to Bufferbloat (most recent first)¶
|Richard Pitt||Usage Based Billing - it's all about perceived congestion|
|Jim Gettys||ECN study results from MIT|
|Jim Gettys||Benchmarking Broadband|
|Brian Proffitt||Latency under Load|
|Richard Pitt||Bufferbloat and your ISP's problem|
|Joe Brockmeier||The fight against Bufferbloat|
|Jim Gettys, Dave Täht||Bufferbloat and VOIP podcast|
|Jim Gettys||A call for help with Bufferbloat animations|
While bufferbloat is not a Linux-specific problem, we're the first community out the gate with
solutions and patches. We are actively looking for iphone, ipad, and android users to test
some basic assumptions regarding [[ECN]].
(if you are on an iphone, and having trouble
accessing this site or this mp3 , please let us know!)
And there are tons of [[Papers|research papers]], tools and [[Linux Tips]] linked to off of our [[wiki]] that need sorting out.
Principal threads of conversation so far this month:¶
... (Still ongoing from last month)
Shifting the Market - How to get the word out? (#33)
SFB - Discussion of the features of Stochastic fair blue queuing, with Juliusz Chroboczek , the author of the SFB patch for the Linux kernel
The dangers of AQM - a cautionary note about Active Queue Management, by Kathie Nichols and Van Jacobson
Bufferbloat and You - a draft contributed by Eric Raymond, Much discussion as to good analogies for how the Internet really works ensued
About LEDBAT, µTP and BitTorrent - an exploration of the issues and advantages of bittorrent
TCP Vegas vs Cubic - Some [[Experiment_-_TCP_cubic_vs_TCP_vegas|puzzling data about TCP vegas]] with current hardware
The wireless problem in a nutshell The unique problems 802.11 introduces for TCP/ip (related: Wireless multiqueue behavior )
Progress on the wiki and infrastructure¶
The wiki needs some serious love. Help merely on defining some terms would be good.
Also available in: Atom