Two articles have just been posted on the ACM Queue Web site as part of a Bufferbloat case study. These are an article by Jim Gettys and Kathleen Nichols, and Vint Cerf interviewing Van Jacobson, Jim Gettys, and Nick Weaver.

These will hit your snail mailbox in the January 2012 CACM (Communications of the ACM).

• BufferBloat: What's Wrong with the Internet?
  A discussion with Vint Cerf, Van Jacobson, Nick Weaver, and Jim Gettys
• Bufferbloat: Dark Buffers in the Internet by Jim Gettys and Kathleen Nichols

You can find it online here.

It is copyright IEEE; it is used with their permission on this web site.

Bufferbloat.net has grown by leaps and bounds and we are in need of replacing the email server and migrating a few services over to our new, beefier box, "huchra".

Email, web, and build services will be disrupted periodically.

I'd intended to write up summaries of bufferbloat related activity once a month, but am running a bit behind. Both JG and I have been travelling heavily.

There's been a lot going on under the covers!

Probably the biggest news is that we are working with Georgia Tech on their bismark project.[1] They are out to diagnoise the Internet and we are out to fix it. The two goals seemed compatible. In particular: we are trying to de-"heisenbug" the test routers so they can accurately test the upstream services.

We've also taken the wraps off the "uberwrt" project², which is an attempt to get the debloating work TESTED in realistic situations at the edge and also into openwrt. (Some work from this also flows into bismark)

I was going to write formal joint press releases on these but have been too busy traveling, talking and hacking. (if anyone wants to step up to handle PR?)

Although traffic on the bloat mailing list has been slow of late, the bismark-devel list has been hopping. Feel free to join bismark/uberwrt projects and/or the mailing list³, especially if you are interested in embedded hardware.

Moving on to other topics...

Based on the early difficulties in getting debloat-testing to be a useful base for the eBDP and A* algorithms, we started looking around for ONE driver to work with and have settled on ath9k hardware (for now) as a base for routers and wireless cards. [10] We need to do a little testing of the laptop cards, but things are looking good. the WNDR3700v2 is AWESOME, actually. 16MB of flash. LUXURY.

Other Patches:

Dan Siemon's pfifo_fast fix for ECN has been backported into 2.6.37.X for openwrt's git head as of Saturday. It's also now part of 2.6.39 and 2.6.38 stable.

SFB is in mainline Linux 2.6.39-RCX and woefully undertested in its current incarnation.

Felix Feitkeu has some patches more fully instrumenting the ath9k driver (when mildly more complete, these should get slammed into debloat-testing as well) [4]

Dan Siemon has improved both his TC shaper test scripts and ping-exp [5]

Media: There were a couple articles on bufferbloat that went by this month, I think they were all covered on this list...

There are 236 members of the bloat list now.

Infrastructure:

We are moving a ton of work to a new build server and also moving the lists machine to that. Regrettably as I write, "huchra" is down due to finger-foo. It should be back up again Monday.

Multiple other servers in other locations are in the queue. I hope to get that sorted out with isc while I am in California.

Upcoming Travel:

JG will be in California April 25-30. I will also be in California April 25-30 (in at least one of the same places as JG), and am available for additional talks/coding/consulting/etc along the western seaboard in early May if anyone wants me and can cover my expenses. (Sort of scheduled: Byte and Atheros U) I'll also be visiting Seattle at some point in May, too.

Travel last month:

JG spent spent several weeks in Europe, first attending the Wireless BattleMesh conference⁶, then the IETF, giving a shorter version of his bufferbloat talk⁷.

I spent a week in florida gathering strength for my world tour. Then I spent a week with Georgia Tech helping get their Bismark project off the ground and hammering out workflow issues.

I was tickled pink when I gave an introductory talk on bufferbloat to a class there, only to discover when Q&A rolled around that everyone participating was already up to speed on bufferbloat and queueing disciplines, and peppered me with questions on SFB, RED, eBDP and other algorithms we are playing with. 3 months ago I would have been met with blank looks, now it's a struggle to keep up!

I then spent a week with esr getting one of the first near-complete builds of the wndr3700 working well, working on gpsd (wanted accurate time on openwrt) and rsnapshot and split dns and a host of semi-bufferbloat.net-and-uberwrt issues... And we also got a revised version of the intro to bufferbloat document up on the wiki [8].

I'm very happy to see thyrsus.com go ipv6 enabled.

The bufferbloat wiki is still in dire need of love, see the Todo list for more details [9] -

Conclusion:

And that's all the news I can remember this late Sunday evening. It's my hope that SFB will make it into bismark/uberwrt this week so we can test SFB a little more while it is still a RC in 2.6.39. I'd VERY MUCH like to make sure SFB works when it is released to millions of users worldwide. That will be in 4 weeks or so... I'm feeling a little schedule pressure here... See dan siemon's scripts... [5]

[1] http://www.bufferbloat.net/projects/bismark/wiki Georgia Tech's project
[2] http://www.bufferbloat.net/projects/uberwrt/wiki
[3] https://lists.bufferbloat.net/listinfo/ (bismark, bismark-devel)
[4] http://www.bufferbloat.net/projects/uberwrt/wiki/Experimental_patches
[5] http://git.coverfire.com/ PLEASE PLAY WITH TC, SFB, and PING-EXP!!!!
The bandwidth you save may be your own.
[6] http://battlemesh.org/ has summaries and videos from the battlemesh
[7] http://mirrors.bufferbloat.net/Talks/PragueIETF/
[8] The original of the bufferbloat introductory piece was extensively discussed on this mailing list. This versions incorporates most of those changes. If you don't like this version... It's a wiki document now! Please feel free to fix, extend, and add links! http://www.bufferbloat.net/projects/bloat/wiki/Introduction
[9] LOTS of writing left http://www.bufferbloat.net/projects/bloat/wiki/ToDo

[10] After evaluating multiple routers, http://www.bufferbloat.net/projects/uberwrt/wiki/Hardware_evaluation

the http://www.bufferbloat.net/projects/bismark/wiki/Wndr3700v2
seemed like the best choice

_______________________________________
Bloat-announce mailing list
Bloat-announce@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat-announce

Slides: http://mirrors.bufferbloat.net/Talks/PragueIETF/

Audio:

http://ietf80streaming.dnsalias.net/ietf80/ietf80-ch4-wed-am.mp3

Jim Gettys will be giving two talks at the Battlemesh conference March 16-19, in Sant Bartomeu del Grau (SBG for short), near Barcelona, Catalunia, Spain.

One will be a condensed version of the existing Bell Labs bufferbloat talk, the other will be about the OLPC meltdowns.

There are also interesting talks scheduled about babel-z, the mesh potato, and other topics of interest to the mesh networking community.

Not to mention the battle of the mesh technologies! May the best debloated router win!

I've put a fairly large (and yet still incomplete) summary of the voip conference call with the freeswitch folk up at:

http://www.bufferbloat.net/projects/bloat/wiki/Bufferbloat_and_Freeswitch_Conference_Call_March_9

The call went on for over 2 hours as we discussed all aspects of bufferbloat as it applies to telephony. The call was recorded, and here is a link to the recording:

http://wiki.freeswitch.org/files/conf_call_2011-03-09.mp3

If you have a listen, and hear something interesting that I did not document, please update the wiki page!

Some interesting tidbits:

• Some freeswitch users are using TCP Vegas for their servers due to the
  latency sensitivity of that form of TCP and their need to prioritize
  RTP traffic
• getting time right on various virtual server technologies is doable, but hard
• freeswitch handled 32+ people in the conference using a variety of
  dial-in technologies really well.
• Their wiki was very wrong on recomendations for voip servers (corrected)

I'm thinking that doing a weekly conference call among ourselves here might be useful. Thoughts?

Again, way more detail on the call is at:

http://www.bufferbloat.net/projects/bloat/wiki/Bufferbloat_and_Freeswitch_Conference_Call_March_9

The members of the Bufferbloat project are invited to sit in on Freeswitch's regular VOIP conference call at 1PM Eastern Time, Wednesday, March 9th. Please dial into:

PSTN:       USA +1-919-386-9900 or
               +44-3300-100-295
SIP:         sip:888@conference.freeswitch.org
GOOGLETALK: conf+888@conference.freeswitch.org

There will also be chatter on the appropriate irc channels as we attempt a melding of the minds. More details to follow.

About Freeswitch¹¶

FreeSWITCH is a scalable, open source, cross-platform soft-switch that allows for consolidating various forms of communication media. It is used primarily for telephony applications - VoIP and traditional PSTN - but also supports video, chat, and other forms of communication.

FreeSWITCH was created in 2006 in response to the need for an open source, freely available telephony platform that could fill the void left by proprietary commercial offerings. Anthony Minessale is the chief architect and lead programmer on the FreeSWITCH project. He is joined by two veteran developers - Brian K West and Michael Jerris. All three of these experienced engineers spent a number of years developing for the Asterisk PBX project before starting FreeSWITCH. 

FreeSWITCH was designed with the goals of modularity, stability, and scalability. It is now used in thousands of servers to power VoIP communications for business and organizations around the world, and runs on nearly every operating system.

About the Bufferbloat project¶

If you are not already aware of what bufferbloat is, the fastest way to get up to speed is to peruse the slides and listen to 25 minutes of audio before the conference call, at:

http://mirrors.bufferbloat.net/Talks/BellLabs01192011/

It would be great to have an informed audience so we can only touch lightly on the preliminaries and then dive deeper into bufferbloat and VOIP issues. See also Jim Gettys' blog postings², discussions on lwn.net³, and slashdot⁴, and elsewhere⁵, as well as two very busy mailing lists⁶

In short, the bufferbloat problem is that there are really big, bloated network buffers in many (especially new) routers, home (mostly wireless) gateways, hosts, and ADSL/FIOS/cable modems that can dramatically affect VOIP performance.

The bufferbloat project⁷ is attempting to identify equipment and software where truly bloated buffers exist, and mitigate or fix the issues with new software algorithms and heightened awareness. Recently we've released a debloat-testing Linux kernel⁸ that may help in some cases. Bufferbloat is not a Linux-specific problem, it exists in all OSes, and may become more acute as Windows 7 gets rolled out.

[1] http://www.freeswitch.org/
[2] http://en.wordpress.com/tag/bufferbloat/
[3] http://lwn.net/Articles/429931/
[4] http://linux.slashdot.org/story/11/02/26/038249/Got-Buffer-Bloat
[5] http://www.bufferbloat.net/projects/bloat/news
[6] https://lists.bufferbloat.net/listinfo/
[7] http://www.bufferbloat.net/projects/bloat/wiki
[8] http://lwn.net/Articles/429943/

The Bufferbloat project news summary, February 17 - March 3, 2011¶

I'm going to try to put out summary news like this once a month in
the future. This is just a summary of the past two weeks. For a summary
of the first two weeks, see here .

For those of you new to bufferbloat, the fastest way to get up to speed
is to review Jim Gettys' presentation: Bufferbloat - Dark buffers on the Internet
and the audio recording

In March, I hope we start to get some patches tested by lots more people,
start getting the qdisc folk to talk to the driver folk (and vice versa)
and continue to unify several separate lines of development. I hope also
we improve the wiki, and continue to recruit more people with different
layers of perspectives and understandings of the problem. Late this month
a new (8 core) server will come online at isc.org for use for building
both debloat-testing and openwrt based kernels for various devices. More
servers and locations will follow.

Extremely high on the list is getting more testing tools modified not only
to detect bufferbloat, but test the new kernel features to see how well they
are working.

Development¶

The Debloat-testing Linux
kernel tree has been updated to 2.6.38-rc7-db. John Linville made an abortive
attempt to move it into the qdisc layer. There are other issues...

Statistics¶

There are now 181 people registered in the general bufferbloat email list
and 42 in bufferbloat development list

41 people have registered on the bufferbloat project web site .
I seem to be the only one making edits. Is there something broken?

35 people (and one robot) regularly in the #bufferbloat irc channel on chat.freenode.net.

References to "bufferbloat" on google doubled from roughly 30,000 to 61,200 results.

(Bufferbloat is now .00053% as popular as sex)

Web hits on the presentations and site itself are not yet available.

We survived slashdot!¶

The thoroughly debloated servers bufferbloat.net is running on survived both a
discussion on lwn.net and a slo-mo slashdotting on the
Got (bufferbloat) Bloat thread.
Slashdot unfortunately got the link to the debloat-testing tree wrong, and never corrected it.

Blog posts & media coverage related to Bufferbloat (most recent first)¶

While bufferbloat is not a Linux-specific problem, we're the first community out the gate with
solutions and patches. We are actively looking for iphone, ipad, and android users to test
some basic assumptions regarding [[ECN]].
(if you are on an iphone, and having trouble
accessing this site or this mp3 , please let us know!)

And there are tons of [[Papers|research papers]], tools and [[Linux Tips]] linked to off of our [[wiki]] that need sorting out.

Principal threads of conversation so far this month:¶

Getting more organized
GSO: https://lists.bufferbloat.net/pipermail/bloat-devel/2011-March/000077.html
Usage based billing

... (Still ongoing from last month)

Shifting the Market - How to get the word out? (#33)
SFB - Discussion of the features of Stochastic fair blue queuing, with Juliusz Chroboczek , the author of the SFB patch for the Linux kernel
The dangers of AQM - a cautionary note about Active Queue Management, by Kathie Nichols and Van Jacobson
Bufferbloat and You - a draft contributed by Eric Raymond, Much discussion as to good analogies for how the Internet really works ensued
About LEDBAT, µTP and BitTorrent - an exploration of the issues and advantages of bittorrent
TCP Vegas vs Cubic - Some [[Experiment_-_TCP_cubic_vs_TCP_vegas|puzzling data about TCP vegas]] with current hardware
The wireless problem in a nutshell The unique problems 802.11 introduces for TCP/ip (related: Wireless multiqueue behavior )

Progress on the wiki and infrastructure¶

The wiki needs some serious love. Help merely on defining some terms would be good.

WHAT A MONTH!
(and it isn't over yet)¶

Since John Carmack was just kind enough to tweet about us, I thought I'd put up the latest news as fast as I could:

What is bufferbloat, anyway?

For indepth background on bufferbloat, see and hear these slides and brief audio

Bufferbloat is a major problem in all the new kit coming out this year - cable modems and 802.11n routers especially. Latencies get so high as to make fragging frickin impossible.

There is some gamer specific discussion in the above link, as well as elsewhere on Jim Gettys' blog

For the current status of our increasingly world wide efforts - see the mailing list archives:

general bufferbloat discussion

bufferbloat development discussion

Multiple new algorithms are being both discussed as well as tested. A Linux kernel repository just went up at:

http://git.infradead.org/debloat-testing.git

there are now 131 people on the mailing list, and 19 people and one robot regularly in the #bufferbloat channel on irc.

While bufferbloat is not a Linux-specific problem, we're the first community out the gate with solutions and patches. We are actively looking for iphone, ipad, and android users to test some basic assumptions regarding [[ECN]]. (if you are on an iphone, and having trouble accessing this site or this mp3 , please let us know!)

Brian Proffitt just wrote a good intro to the latency under load problem

And there are tons of [[Papers|research papers]], tools and [[Linux Tips]] linked to off of our [[wiki]].

the mailing list and site have been up for less than a month.

Please join us and help beat the bloat!¶