Enable ECN on multiple operating systems¶

Solaris¶

OSX¶

Linux¶

FreeBSD¶

notes to sort out:

[Tue Jun 21 2011]
<bmc> What was the name of the bufferbloat-related sysctl parameter on Linux?
ECN? I'm drawing a blank. [07:43]
<dtaht> ?
<dtaht> Got your plug?
<bmc> Up and running.
<dtaht> heh
<bmc> net.ipv4.tcp_ecn [07:44]
<bmc> That's it.
<dtaht> http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle
<bmc> I have it set to 2 which, as I recall, means "try ECN, but
fallback if it's not there"
<dtaht> no, it's worse than that
<dtaht> secondly this only applies to connections initiated from or to the
router [07:45]
<dtaht> dsack, and sack are good too
<bmc> But it should help with NAT'ed connections, no?
<dtaht> I just got a guruplug version of openwrt built, but debian is WAY
easier to deal with.
<dtaht> no
<dtaht> proxied via something like polipo, yes [07:46]
<bmc> Hmm. But the original problem you had here was that my crappy
Linksys didn't grok ECN at all, and you had ECN = 1 on your Linux
box. Right?
<dtaht> right [07:47]
<dtaht> your router was not passing through the ECN bits - dying terribly.
<dtaht> Now you should be able to turn it on on the rest of your internal gear
<bmc> So, if my plug router groks ECN, and I enable ECN on it, I gain …
what?
<bmc> It's a flow-control thing, isn't it?
<dtaht> and have all the relative theoretical chocolaty goodness
<dtaht> yes
<dtaht> marking rather than dropping packets is a theoretical goodness [07:48]
<bmc> Right. So, bottom line, I want to enable it on the plug, and
enable it (set to 2 or 1) on the various internal machines.
<dtaht> yes.
<bmc> Interesting. Out of the box, the plug's Debian has: [07:49]
<bmc> net.ipv4.tcp_sack = 1 net.ipv4.tcp_ecn =
2 net.ipv4.tcp_dsack = 1 
<dtaht> that are the best defaults for yesterday's internet. The new hotness
is ecn
<dtaht> :)
<bmc> The internal Ubuntu servers are similarly configured.
<dtaht> yep
<dtaht> ECN breakage was a real problem
<bmc> Yes, but, IIRC, ecn=2 is essentially ecn=1, with a fallback to
oldness. [07:50]
<dtaht> in fact, we've fixed ECN, TOS, and Diffserv problems all over the
Linux stack in the last month.
<dtaht> ummmm
<bmc> Or did I misunderstand your original explanation?
<dtaht> Or I was drunk...
<dtaht> ecn = 0 entirely disabled
<dtaht> ecn = 1 enabled
<bmc> Right. I got that.
<bmc> What is ecn=2?
<dtaht> ecn = 2...
<dtaht> I think means accept it if presented but don't initiate it... Wait one
[07:51]
<dtaht> yea
<bmc> Ah. So, ecn=2 on the router is fine, but I want ecn=1 on the
internal client machines.
<dtaht> so ecn=1 is good, and a fallback, like ecn=3 - if it existed, which
would have the desired fallback behavior would be good [07:52]
<dtaht> there's a patch for that coming, I think
<bmc> Sounds like I should leave the router at 2, in case there are
internal machines (e.g., windows breakage) that don't do ECN.
<dtaht> heh
<dtaht> I'm not aware of ecn=1 breaking on ANYTHING except your old
router. You were the first, the last, and the worst.
<dtaht> but whatever
<bmc> I'm just trying to grok this stuff. [07:53]
<bmc> I'll try 1 on the router, see what happens.
<dtaht> it helps to have qos on the router that applies it to streams that are
killing your life, rather than drops it... But that's not a huge issue
for you as you have bandwidth to burn.
<bmc> Man, this dream plug is sweet. Tiny, small footprint (physical and
electrical), quiet as all fuck, and real *nix, to boot.
<dtaht> thx for tryin it. How does the 'dreamplug' feel? Is it a better piece
of gear?
<dtaht> hahha
<dtaht> answered my question in advance [07:54]
<bmc> Way better. I resurrected the GuruPlug, via the JTAG. Trying to
decide whether I have a use for it.
<bmc> Or whether I should give it to someone (e.g., Costine) and spread
the love.
<dtaht> yea, I loved the openrd, but the interviening releases of sheevaplug
and guruplug left me cold. I'm also not happy with the wireless chip,
but that's relatively minor.
<dtaht> spread the love, man....
<bmc> That's what I'm thinking. … Okay, time for a conf call. By then,
my kid should be awake and ready for waffles. [07:55]
<bmc> BTW, the wireless is working just fine here.
<bmc> I use rc.local to adjust the params via uapctl.
<dtaht> yea, well, start moving to the edge of the range and then watch your
ping times....
<bmc> Not an issue here.

dtaht plans to get one of these puppies too, after bmc is happy for a few
weeks.
<bmc> I have four WAPs in this interference-laden place. :-)
dtaht is using diversity mesh routing now with babel [07:56]
<dtaht> babel + ahcpd is sweet
<dtaht> sit down, plug in the laptop
<dtaht> it figures out it's on wired
<bmc> Something to look into, when I have time. If I ever have time
again.
<dtaht> unplug, it figures out you are on wireless
<bmc> Niiiice.
<dtaht> streams, and connections, STAY UP
<bmc> Very sweet.
<bmc> Okay, time to make that call. Later.
<dtaht> yea, it's amazing how freeing it is to be able to plug in again.
<dtaht> later [07:57]
<dtaht> thx
<dtaht> also the mesh routing 'diversity' means that I have several nodes that
route over 5.x ghz and are aps over 2.4, and vice versa... Connect
your neighbors
<dtaht> whenever you get time I wanted to know how that meeting turned out.
[07:58]
<bmc> What meeting? [08:00]
<dtaht> some users group meeting of all the users groups [08:07]
<bmc> Right. I went to a lot of meetings last week. :-) That went
well. We're moving forward, trying to get common venues, common
calendars, etc. [08:10]
<bmc> FYI, on Mac OS X: [08:12]
<bmc> $ sysctl -a | egrep 'ecn|sack' net.inet.tcp.ecn_initiate_out:
0 net.inet.tcp.ecn_negotiate_in: 0 net.inet.tcp.sack:
1 net.inet.tcp.sack_maxholes:
128 net.inet.tcp.sack_globalmaxholes:
65536 net.inet.tcp.sack_globalholes: 0 net.inet.ipsec.ecn:
0 net.inet6.ipsec6.ecn: 0 
<dtaht> heh. THANK YOU [08:22]
dtaht wanted a mesh network in philly to implement.
<bmc> Another data point: (dancer:bmc) ~ $ uname -s -r FreeBSD
8.2-RELEASE (dancer:bmc) ~ $ sysctl -a | egrep
'sack|ecn' vfs.bufreusecnt: 932 net.inet.tcp.ecn.maxretries:
1 net.inet.tcp.ecn.enable: 0 net.inet.tcp.sack.globalholes:
0 net.inet.tcp.sack.globalmaxholes:
65536 net.inet.tcp.sack.maxholes: 128 net.inet.tcp.sack.enable:
1 net.inet.sctp.enable_sack_immediately: [08:48]
<bmc> 0 net.inet.sctp.nr_sack_on_off: 0 net.inet.sctp.sack_freq:
2 net.inet.sctp.delayed_sack_time: 200 net.inet.sctp.strict_sacks:
1 net.inet.sctp.ecn_nonce: 0 net.inet.sctp.ecn_enable:
1 
<dtaht> sight [08:50]
<dtaht> sigh [08:51]
<bmc> They're all different.
<bmc> I have OpenSolaris here, too. Want that data point?
<dtaht> sure
<bmc> Booting...
dtaht just had 3 new olpcs arrive and is mildly distracted
<bmc> Nice. [08:52]
<bmc> They make good nightlights, I'm told.
<dtaht> the 1.5s are better nightlights
<dtaht> the 1.75s (I'm on the list) are cool
<dtaht> I'm trying to convince them to add 5.x ghz support
<bmc> Every child should have that.
<dtaht> hahahaha
<bmc> Hmm… SunOS 5.11. sysctl not found... [08:53]
<bmc> $ ndd /dev/tcp \?|egrep -i 'ecn|sack' tcp_sack_permitted
(read and write) tcp_ecn_permitted (read and
write) [08:58]
<bmc> (sunball:bmc) /etc $ ndd -get /dev/tcp
tcp_sack_permitted 2 (sunball:bmc) /etc $ ndd -get /dev/tcp
tcp_ecn_permitted 1 [08:59]
<bmc>
http://download.oracle.com/docs/cd/E19963-01/html/821-1450/chapter4-31.html
<bmc> (Oracle Solaris Tunable Parameters Reference Manual)
<bmc> ecn: 0 (disabled), 1 (passive enabled), or 2 (active enabled)
 [09:00]
<bmc> Same values for SACK
<bmc> Defaults are as shown above. [09:01]
<dtaht> excellent. [09:09]
<bmc> I just tweeted this: When you've spent the last decade mostly
using Linux and BSD, the infrequent foray into Solaris feels like going
to Mars. #unix
<dtaht> So if I get you to turn it on, that will only leave about 2 billion
computers left to fix and 10s of thousands or routers left to junk
[09:10]
<dtaht> heheheh
<dtaht> solaris is like 1998
<bmc> I don't even want to think about how one accomplishes this in
HP/UX.
<bmc> Assuming it's even possible.
<dtaht> thx for the data I'm going to update the wiki in a bit
<bmc> Or, for that matter, AIX, which makes Mars seem familiar.
<dtaht> hahahah
<dtaht> do these things have sysctl.conf?
<bmc> I didn't see a reference to an ndd.conf [09:11]
<bmc> http://www.sean.de/Solaris/soltune.html#ndd
<bmc> Worth skimming. [09:12]
<bmc> Of course, Solaris no longer has an /etc/rc.local, either.
<bmc> Instead, you're supposed to create something in /etc/init.d
(rc.local is fine), then symlink it to an "S" file in the appropriate
runlevel directory (e.g., rc3.d).
<bmc> *sigh*
<bmc> Wait... [09:13]
<bmc> I just ran across this:
<bmc> To set parameters so they remain in effect after you reboot the
system, add the parameter values to /etc/system when you want to
configure parameters for all devices in the system.
<bmc> A startup script can also be used to set a ndd parameters across
system reboots. Include the appropriate ndd command in a system startup
script, such as the /etc/init.d/inetinit file or a customized script in
/etc/rc2.d or /etc/rc3.d. Be sure to make a copy of any files before
adding the ndd commands. [09:14]
ERC>

Bloat

Wiki

Enable ECN on multiple operating systems¶

Solaris¶

OSX¶

Linux¶

FreeBSD¶