Feature #305

Cerowrt needs a good default set of queueing disciplines enabled (PFIFO_FAST MUST DIE)

Added by Dave Täht about 3 years ago. Updated over 2 years ago.

Status:Closed Start date:11/24/2011
Priority:Urgent Due date:
Assignee:Dave Täht % Done:

10%

Category:Networking Spent time: -
Target version:1st Usable CeroWrt release Estimated time:120.00 hours

Description

Probably the biggest new thing in rc8 is going to be all the interfaces will be running some non-default qdisc by default, or so I hope.

To start off with wireless - there is no qdisc that is really appropo at this time. Work is in progress towards abstracting out enough stuff to make an aggregatable qdisc but I expect that work to take 6 months or more longer to even show a plausible result, much less

Of the choices available... SFQ has properties that are pretty compatible with packet aggregation in wireless-n, and does not do too much harm to wireless-g (or so I think), and I have been using that for several months now with no other ill effects noted. The usually less-than-desirable property of SFQ is that it can bunch up to 16 packets up and not be entirely 'fair' that way. This is basically what we want, actually, when we aggregate packets.

I am explicitly filtering out multicast macs and making sure they end up in the VO queue so they don't confuse SFQ as much.(presently multicast in MAC80211 gets figured out after it hits the driver, rather than before it hits the qdiscs)

I also have mods in there to do better DSCP level classification (since rc5), and if I can finish it, I have an 'ANT' classifier as well that I'll stick in there. The net effect of these two classifiers is to toss more real-time requiring (but short) stuff into the VI queue. It's my hope that I can push both the dscp and ant things upstream after rc8 is done, also. So what I think is going to happen on the gw00 gw10 sw00 sw10 interfaces is 3 SFQ qdiscs on VI, BE, and BK - and I might either do pfifo_head_drop on VO or QFQ + head drop on VO. gw01 and g11 I haven't decided on as presently packets can't be aggregated there and I'm in love with QFQ....

As for ethernet, I've proven that I can run QFQ and QFQ + RED at rates that seem thus far indistinguishable from pfifo_fast from a cpu perspective.

so either they aren't working... or they are pretty fast... more to come on that.

So the internal ethernet interface is going to be QFQ for sure, but I don't see much point in red there at present. The external interface... well... let me do that in a different bug.....


Subtasks

Feature #306: Better QoS for the external interfaceNewDave Täht


Related issues

related to Cerowrt - Feature #301: Need to arrange for AQM testing and bakeoff.... New 11/18/2011
related to Cerowrt - Bug #262: poor wireless performance in smoketest rc6 In Progress 09/04/2011
related to Cerowrt - Feature #321: ar71xx architecture needs to move into the kernel mainline New 12/08/2011

History

Updated by David Taht almost 3 years ago

This is the 6th in a series of mails doing a post-mortem and rethink
of what is in cerowrt.

Basically I think the 2.4ghz spectrum is beyond hope. Everywhere I've
been there are dozens of radios all on all the channels available, all
competing, all interfering, and the difference in performance results
I get minute to minute, hour to hour varies so wildly that in order to
get a good picture of how basic 2.4ghz performance worked I had to go
to remote valley in france where there was no competing signals to
deal with.

There I got pretty equivalent results between 2.4ghz and 5. The rest
of the time, not so much.

As for wireless-g vs n, the current architecture of the mac80211 stack
buries packet aggregation so deeply within it that the two
technologies are very incompatible. It also bothers me that management
frames are buried so deeply, too.

The only places I've been this year that had a functional wireless
network, rigorously divided up the SSIDs into g, n, ipv4 and ipv6
specific parts, and were using Cisco hardware.

I don't see the need to have separate SSIDs for 4 and 6, but I think
g+QFQ, and n+SFQ might do better in the general case on an AP.

QFQ would work well on client machines, particularly with n, to break
up packet bursts more sanely into multiple streams. SFQ, less so. I'd
rather pursue QFQ on the clients as it better 'shreds' competing
bursts. I'd like others to play with it too.

and in all cases getting to where something BQL-like + Time in Queue
on top of those two techniques will help most of all,
but better feedback loops and means of determining bandwidth
differences between destinations are kind of required.

And doing MUCH better classification into the hardware queues would be
good, too.

So in the next release of cerowrt I'm planning on having a separate
n-only or g-only SSID available on the 5ghz channel, and
to experiment with the above qdiscs. I sort of have some major ANT and
DSCP specific classifier code that I plan to polish up.

as for BQL and Time in Queue, that work is only just beginning and I
hope to track it closely. As to how to make it apply to wireless,
I look forward to the engineering debate(s). It's a ton of work.

Updated by David Taht almost 3 years ago

note, that I do try on occasion to capture stuff into the bug tracker
When you see something like [#305] in the subject
or cerowrt ccd, it goes there...

That said, I have to not surprise people with that 'feature'.

On Thu, Dec 8, 2011 at 1:46 PM, Dave Taht <> wrote:

On Thu, Dec 8, 2011 at 1:25 PM,  <> wrote:

On Thu, 8 Dec 2011, Dave Taht wrote:

On Thu, Dec 8, 2011 at 12:51 PM,  <> wrote:

On Thu, 8 Dec 2011, Dave Taht wrote:

this puzzles me.

splitting 2.4G and 5G into different different networks (broadcast domains) is a huge win. cince I can't find any open implementation fo band steering, this requires putting the two bands on different SSIDs.

Oh, god no, I'm not dropping that. Having those split AND off the wired network is staying in...

but I don't understand why there is a big problem with G and N sharing the same SSID.

Because you can fully FQ G, and if you do that to N, it messes up aggregation.

I don't recognize the term "FQ".

Fair Queue

http://info.iet.unipi.it/~luigi/qfq/ in this case.

when you say it messes up aggregation, do you mean combining two channels togeather for higher throughput or something else?

No, the way the driver is structured it swallows as many packets as are aggregatable for a given destination, then ships them. If you instead try to do the right thing - which is to break up packet bursts into as tiny pieces as possible, aggregation goes to heck. What you want to do is aggregate fair queued packets for a given destination, at a size that will fit (up to 64 packets or 64kbytes) at the rate the wireless interface is running at.

As a result nobody does FQ, nor AQM, on wireless n, where it is so desparately needed.

I was assuming that if you are running a mized network you only use a single channel for N. If you are using multiple channels for N they should be a separate SSID, just from the fact that you are using two channels for N but only one for G (which one would be the question)

One channel for both N and G in this case. Only one radio for 5ghz.

there is some

Some?

some, but it's an unavoidable feature of wireless communication. You can consider turning off some modulation types, but since the clients automatically fall back to slower modulation types when there is a problem, the result will be failed connections.

To give you an idea, at 5ghz I'm capable with cerowrt at achieving 150Mbits with TCP - in the clean air here.

At 2.4, it's rare I can get more than 20, and fairly often much less than that.

Any given test I run regarding wireless simply is not repeatable if I do it on 2.4ghz.

I can usually 'hear' more than 30 access points at my apt, as another example.

now, this may still be the right thing to do, because the failback to a slower modulation type works well for weak signal situations, but in a high density situation (which is basically every 2.4G deployment in the real world nowdays), taking longer to send the same data just means that you are more vunerable to another transmitter clobbering you, so it actually decreases reliability.

yes, minstrel rocks.

David Lang

grief with having different speeds on the same channel, but only in that the same amount of data will take longer to transmit (causing problems with predicting how long the queue is in terms of time as it will vary on the destintation), but even if you stick with G for example, it can transmit at 54, 48, 36, 24, 18, 12, 6, 1 Mb/s. adding N just adds some higher speeds to this. If the devices are configured sanely, they should be transmitting the header for a G frame to reserve the air time and then sending the N frame inside of that. this has a slight overhead compared to a pure N network, but it doesn't matter if the G network is on the same SSID or on a different one, the problem is sharing the airtime on the channel.

It's a packet scheduler test more than anything else.

David Lang

Updated by Dave Täht over 2 years ago

  • Status changed from New to Closed

sfqred runs on EVERYTHING.

Updated by Dave Täht over 2 years ago

  • Target version changed from Cerowrt-1.0-rc8 to 1st Usable CeroWrt release

Updated by Dave Täht over 2 years ago

I'm not in a position to build kernels for anything but ubuntu and cerowrt at the moment, although fq_codel has landed in operwrt, (doing builds for tons of arches)

The last important patch for fq_codel landed in net-next a few minutes ago.

So if anyone out there is running any other linux derivative that is linux 3.3 based, I'd love it if they could get codel and fq_codel running on it and get a build out there for others to try.

aqm is not just for routers anymore

Updated by Jonathan Morton over 2 years ago

On 16 May, 2012, at 10:50 pm, wrote:

So if anyone out there is running any other linux derivative that is linux 3.3 based, I'd love it if they could get codel and fq_codel running on it and get a build out there for others to try.

Unfortunately I don't think Gentoo counts. :-)

However, my tests using my Gentoo-flavoured PowerBook firewall are very promising indeed. With the bandwidth restricted to 4M/1M and several torrents running, ordinary HTTP traffic (including DNS etc.) still got through reliably and smoothly with fq_codel, and less so with plain codel or plain sfq.

With the torrent client set up to restrict the bandwidth of TCP connections explicitly, but let uTP run freely, there was an interesting quirk. With no AQM/FQ specified, a single HTTP download was able to outcompete dozens of uTP streams and receive a varying but relatively large share of the bandwidth - a clear sign that uTP's efforts to fade into the background are in fact effective. With fq_codel, the HTTP download received roughly equal bandwidth to one of the faster members of the uTP set. With plain codel, the results were much less convincing, as the HTTP download often stalled - it's not immediately clear to me why that might be.

Most striking is the behaviour of fq_codel when downloading a typical web page with many constituent files, particularly images. These files download neatly and smoothly in parallel, just like you would expect them to if you knew nothing about networking. It's beautiful. SFQ has the same basic effect, but with fq_codel it seems to be more robust.

I have yet to run any successful tests with QFQ. With the vanilla kernel I got zero traffic as soon as I turned it on. With net-next I might get better results.

- Jonathan Morton

Also available in: Atom PDF