Bug #305

Cerowrt needs a good default set of queueing disciplines enabled (PFIFO_FAST MUST DIE)

Added by Dave Täht on Nov 24, 2011. Updated on May 17, 2012.
Closed Urgent Dave Täht


Probably the biggest new thing in rc8 is going to be all the interfaces will be running some non-default qdisc by default, or so I hope.

To start off with wireless - there is no qdisc that is really appropo at this time. Work is in progress towards abstracting out enough stuff to make an aggregatable qdisc but I expect that work to take 6 months or more longer to even show a plausible result, much less

Of the choices available… SFQ has properties that are pretty compatible with packet aggregation in wireless-n, and does not do too much harm to wireless-g (or so I think), and I have been using that for several months now with no other ill effects noted. The usually less-than-desirable property of SFQ is that it can bunch up to 16 packets up and not be entirely ‘fair’ that way. This is basically what we want, actually, when we aggregate packets.

I am explicitly filtering out multicast macs and making sure they end up in the VO queue so they don’t confuse SFQ as much.(presently multicast in MAC80211 gets figured out after it hits the driver, rather than before it hits the qdiscs)

I also have mods in there to do better DSCP level classification (since rc5), and if I can finish it, I have an ‘ANT’ classifier as well that I’ll stick in there. The net effect of these two classifiers is to toss more real-time requiring (but short) stuff into the VI queue. It’s my hope that I can push both the dscp and ant things upstream after rc8 is done, also. So what I think is going to happen on the gw00 gw10 sw00 sw10 interfaces is 3 SFQ qdiscs on VI, BE, and BK - and I might either do pfifo_head_drop on VO or QFQ + head drop on VO. gw01 and g11 I haven’t decided on as presently packets can’t be aggregated there and I’m in love with QFQ….

As for ethernet, I’ve proven that I can run QFQ and QFQ + RED at rates that seem thus far indistinguishable from pfifo_fast from a cpu perspective.

so either they aren’t working… or they are pretty fast… more to come on that.

So the internal ethernet interface is going to be QFQ for sure, but I don’t see much point in red there at present. The external interface… well… let me do that in a different bug…..


Updated by David Taht on Dec 8, 2011.
This is the 6th in a series of mails doing a post-mortem and rethink
of what is in cerowrt.

Basically I think the 2.4ghz spectrum is beyond hope. Everywhere I’ve
been there are dozens of radios all on all the channels available, all
competing, all interfering, and the difference in performance results
I get minute to minute, hour to hour varies so wildly that in order to
get a good picture of how basic 2.4ghz performance worked I had to go
to remote valley in france where there was no competing signals to
deal with.

There I got pretty equivalent results between 2.4ghz and 5. The rest
of the time, not so much.

As for wireless-g vs n, the current architecture of the mac80211 stack
buries packet aggregation so deeply within it that the two
technologies are very incompatible. It also bothers me that management
frames are buried so deeply, too.

The only places I’ve been this year that had a functional wireless
network, rigorously divided up the SSIDs into g, n, ipv4 and ipv6
specific parts, and were using Cisco hardware.

I don’t see the need to have separate SSIDs for 4 and 6, but I think
g+QFQ, and n+SFQ might do better in the general case on an AP.

QFQ would work well on client machines, particularly with n, to break
up packet bursts more sanely into multiple streams. SFQ, less so. I’d
rather pursue QFQ on the clients as it better ‘shreds’ competing
bursts. I’d like others to play with it too.

and in all cases getting to where something BQL-like + Time in Queue
on top of those two techniques will help most of all,
but better feedback loops and means of determining bandwidth
differences between destinations are kind of required.

And doing MUCH better classification into the hardware queues would be
good, too.

So in the next release of cerowrt I’m planning on having a separate
n-only or g-only SSID available on the 5ghz channel, and
to experiment with the above qdiscs. I sort of have some major ANT and
DSCP specific classifier code that I plan to polish up.

as for BQL and Time in Queue, that work is only just beginning and I
hope to track it closely. As to how to make it apply to wireless,
I look forward to the engineering debate(s). It’s a ton of work.

Updated by David Taht on Dec 8, 2011.
note, that I do try on occasion to capture stuff into the bug tracker
When you see something like [#305] in the subject
or cerowrt ccd, it goes there…

That said, I have to not surprise people with that ‘feature’.

On Thu, Dec 8, 2011 at 1:46 PM, Dave Taht dave.taht@gmail.com wrote:
> On Thu, Dec 8, 2011 at 1:25 PM,  david@lang.hm wrote:
>> On Thu, 8 Dec 2011, Dave Taht wrote:
>>> On Thu, Dec 8, 2011 at 12:51 PM,  david@lang.hm wrote:
>>>> On Thu, 8 Dec 2011, Dave Taht wrote:
>>>> this puzzles me.
>>>> splitting 2.4G and 5G into different different networks (broadcast
>>>> domains)
>>>> is a huge win. cince I can’t find any open implementation fo band
>>>> steering,
>>>> this requires putting the two bands on different SSIDs.
>>> Oh, god no, I’m not dropping that. Having those split AND off the wired
>>> network is staying in…
>>>> but I don’t understand why there is a big problem with G and N sharing
>>>> the
>>>> same SSID.
>>> Because you can fully FQ G, and if you do that to N, it messes up
>>> aggregation.
>> I don’t recognize the term “FQ”.
> Fair Queue
> http://info.iet.unipi.it/~luigi/qfq/ in this case.
>> when you say it messes up aggregation, do you mean combining two channels
>> togeather for higher throughput or something else?
> No, the way the driver is structured it swallows as many packets as are
> aggregatable for a given destination, then ships them. If you instead
> try to do the right thing - which is to break up packet bursts into
> as tiny pieces as possible, aggregation goes to heck. What you want
> to do is aggregate fair queued packets for a given destination, at a
> size that will fit (up to 64 packets or 64kbytes) at the rate the wireless
> interface is running at.
> As a result nobody does FQ, nor AQM, on wireless n, where it is
> so desparately needed.
>> I was assuming that if you are running a mized network you only use a single
>> channel for N. If you are using multiple channels for N they should be a
>> separate SSID, just from the fact that you are using two channels for N but
>> only one for G (which one would be the question)
> One channel for both N and G in this case. Only one radio for 5ghz.
>>>> there is some
>>> Some?
>> some, but it’s an unavoidable feature of wireless communication. You can
>> consider turning off some modulation types, but since the clients
>> automatically fall back to slower modulation types when there is a problem,
>> the result will be failed connections.
> To give you an idea, at 5ghz I’m capable with cerowrt at achieving 150Mbits
> with TCP - in the clean air here.
> At 2.4, it’s rare I can get more than 20, and fairly often much less than that.
> Any given test I run regarding wireless simply is not repeatable if I do it
> on 2.4ghz.
> I can usually ‘hear’ more than 30 access points at my apt, as another
> example.
>> now, this may still be the right thing to do, because the failback to a
>> slower modulation type works well for weak signal situations, but in a high
>> density situation (which is basically every 2.4G deployment in the real
>> world nowdays), taking longer to send the same data just means that you are
>> more vunerable to another transmitter clobbering you, so it actually
>> decreases reliability.
> yes, minstrel rocks.
>> David Lang
>>>> grief with having different speeds on the same channel, but
>>>> only in that the same amount of data will take longer to transmit
>>>> (causing
>>>> problems with predicting how long the queue is in terms of time as it
>>>> will
>>>> vary on the destintation), but even if you stick with G for example, it
>>>> can
>>>> transmit at 54, 48, 36, 24, 18, 12, 6, 1 Mb/s. adding N just adds some
>>>> higher speeds to this. If the devices are configured sanely, they should
>>>> be
>>>> transmitting the header for a G frame to reserve the air time and then
>>>> sending the N frame inside of that. this has a slight overhead compared
>>>> to a
>>>> pure N network, but it doesn’t matter if the G network is on the same
>>>> SSID
>>>> or on a different one, the problem is sharing the airtime on the channel.
>>> It’s a packet scheduler test more than anything else.
>>>> David Lang

Updated by Dave Täht on Apr 20, 2012.
sfqred runs on EVERYTHING.
Updated by Dave Täht on Apr 21, 2012.
Updated by Dave Täht on May 16, 2012.
I’m not in a position to build kernels for anything but ubuntu and cerowrt at the moment, although fq_codel has landed in operwrt, (doing builds for tons of arches)

The last important patch for fq_codel landed in net-next a few minutes ago.

So if anyone out there is running any other linux derivative that is linux 3.3 based, I’d love it if they could get codel and fq_codel running on it and get a build out there for others to try.

aqm is not just for routers anymore

Updated by Jonathan Morton on May 17, 2012.
On 16 May, 2012, at 10:50 pm, cerowrt@lists.bufferbloat.net wrote:

So if anyone out there is running any other linux derivative that is linux 3.3 based, I’d love it if they could get codel and fq_codel running on it and get a build out there for others to try.

Unfortunately I don’t think Gentoo counts. :-)

However, my tests using my Gentoo-flavoured PowerBook firewall are very promising indeed. With the bandwidth restricted to 4M/1M and several torrents running, ordinary HTTP traffic (including DNS etc.) still got through reliably and smoothly with fq_codel, and less so with plain codel or plain sfq.

With the torrent client set up to restrict the bandwidth of TCP connections explicitly, but let uTP run freely, there was an interesting quirk. With no AQM/FQ specified, a single HTTP download was able to outcompete dozens of uTP streams and receive a varying but relatively large share of the bandwidth - a clear sign that uTP’s efforts to fade into the background are in fact effective. With fq_codel, the HTTP download received roughly equal bandwidth to one of the faster members of the uTP set. With plain codel, the results were much less convincing, as the HTTP download often stalled - it’s not immediately clear to me why that might be.

Most striking is the behaviour of fq_codel when downloading a typical web page with many constituent files, particularly images. These files download neatly and smoothly in parallel, just like you would expect them to if you knew nothing about networking. It’s beautiful. SFQ has the same basic effect, but with fq_codel it seems to be more robust.

I have yet to run any successful tests with QFQ. With the vanilla kernel I got zero traffic as soon as I turned it on. With net-next I might get better results.

  • Jonathan Morton

This is a static export of the original bufferbloat.net issue database. As such, no further commenting is possible; the information is solely here for archival purposes.
RSS feed

Recent Updates

Oct 20, 2023 Wiki page
What Can I Do About Bufferbloat?
Dec 3, 2022 Wiki page
Codel Wiki
Jun 11, 2022 Wiki page
More about Bufferbloat
Jun 11, 2022 Wiki page
Tests for Bufferbloat
Dec 7, 2021 Wiki page
Getting SQM Running Right

Find us elsewhere

Bufferbloat Mailing Lists
#bufferbloat on Twitter
Google+ group
Archived Bufferbloat pages from the Wayback Machine


Comcast Research Innovation Fund
Nlnet Foundation
Shuttleworth Foundation

Bufferbloat Related Projects

OpenWrt Project
Congestion Control Blog
Flent Network Test Suite
The Cake shaper
CeroWrt (where it all started)

Network Performance Related Resources

Jim Gettys' Blog - The chairman of the Fjord
Toke's Blog - Karlstad University's work on bloat
Voip Users Conference - Weekly Videoconference mostly about voip
Candelatech - A wifi testing company that "gets it".