Bloat - Bufferbloat

1

h1. Enable ECN on multiple operating systems

2

1

3

1

h2. Solaris

4

1

5

1

h2. OSX

6

1

7

1

h2. Linux

8

1

9

1

h2. FreeBSD

10

1

11

1

notes to sort out:

12

13

14

15

[Tue Jun 21 2011]

16

1

<bmc> What was the name of the bufferbloat-related sysctl parameter on Linux?

17

1

      ECN? I'm drawing a blank.  [07:43]

18

1

<dtaht> ?

19

1

<dtaht> Got your plug?

20

1

<bmc> <FONT>Up and running.</FONT>

21

1

<dtaht> heh

22

1

<bmc> <FONT>net.ipv4.tcp_ecn</FONT>  [07:44]

23

1

<bmc> <FONT>That's it.</FONT>

24

1

<dtaht> http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle

25

1

<bmc> <FONT>I have it set to 2 which, as I recall, means "try ECN, but

26

1

      fallback if it's not there"</FONT>

27

1

<dtaht> no, it's worse than that

28

1

<dtaht> secondly this only applies to connections initiated from or to the

29

1

	router  [07:45]

30

1

<dtaht> dsack, and sack are good too

31

1

<bmc> <FONT>But it should help with NAT'ed connections, no?</FONT>

32

1

<dtaht> I just got a guruplug version of openwrt built, but debian is WAY

33

1

	easier to deal with.

34

1

<dtaht> no

35

1

<dtaht> proxied via something like polipo, yes  [07:46]

36

1

<bmc> <FONT>Hmm. But the original problem you had here was that my crappy

37

1

      Linksys didn't grok ECN at all, and you had ECN = 1 on your Linux

38

1

      box. Right?</FONT>

39

1

<dtaht> right  [07:47]

40

1

<dtaht> your router was not passing through the ECN bits - dying terribly.

41

1

<dtaht> Now you should be able to turn it on on the rest of your internal gear

42

1

<bmc> <FONT>So, if my plug router groks ECN, and I enable ECN on it, I gain …

43

1

      what?</FONT>

44

1

<bmc> <FONT>It's a flow-control thing, isn't it?</FONT>

45

1

<dtaht> and have all the relative theoretical chocolaty goodness

46

1

<dtaht> yes

47

1

<dtaht> marking rather than dropping packets is a theoretical goodness  [07:48]

48

1

<bmc> <FONT>Right. So, bottom line, I want to enable it on the plug, and

49

1

      enable it (set to 2 or 1) on the various internal machines.</FONT>

50

1

<dtaht> yes.

51

1

<bmc> <FONT>Interesting. Out of the box, the plug's Debian has:</FONT>  [07:49]

52

1

<bmc> <FONT>net.ipv4.tcp_sack = 1<BR>net.ipv4.tcp_ecn =

53

1

      2<BR>net.ipv4.tcp_dsack = 1<BR></FONT>

54

1

<dtaht> that are the best defaults for yesterday's internet. The new hotness

55

1

	is ecn

56

1

<dtaht> :)

57

1

<bmc> <FONT>The internal Ubuntu servers are similarly configured.</FONT>

58

1

<dtaht> yep

59

1

<dtaht> ECN breakage was a real problem

60

1

<bmc> <FONT>Yes, but, IIRC, ecn=2 is essentially ecn=1, with a fallback to

61

1

      oldness.</FONT>  [07:50]

62

1

<dtaht> in fact, we've fixed ECN, TOS, and Diffserv problems all over the

63

1

	Linux stack in the last month.

64

1

<dtaht> ummmm

65

1

<bmc> <FONT>Or did I misunderstand your original explanation?</FONT>

66

1

<dtaht> Or I was drunk...

67

1

<dtaht> ecn = 0 entirely disabled

68

1

<dtaht> ecn = 1 enabled

69

1

<bmc> <FONT>Right. I got that.</FONT>

70

1

<bmc> <FONT>What is ecn=2?</FONT>

71

1

<dtaht> ecn = 2...

72

1

<dtaht> I think means accept it if presented but don't initiate it... Wait one

73

1

								        [07:51]

74

1

<dtaht> yea

75

1

<bmc> <FONT>Ah. So, ecn=2 on the router is fine, but I want ecn=1 on the

76

1

      internal client machines.</FONT>

77

1

<dtaht> so ecn=1 is good, and a fallback, like ecn=3 - if it existed, which

78

1

	would have the desired fallback behavior would be good  [07:52]

79

1

<dtaht> there's a patch for that coming, I think

80

1

<bmc> <FONT>Sounds like I should leave the router at 2, in case there are

81

1

      internal machines (e.g., windows breakage) that don't do ECN.</FONT>

82

1

<dtaht> heh

83

1

<dtaht> I'm not aware of ecn=1 breaking on ANYTHING except your old

84

1

	router. You were the first, the last, and the worst.

85

1

<dtaht> but whatever

86

1

<bmc> <FONT>I'm just trying to grok this stuff.</FONT>  [07:53]

87

1

<bmc> <FONT>I'll try 1 on the router, see what happens.</FONT>

88

1

<dtaht> it helps to have qos on the router that applies it to streams that are

89

1

	killing your life, rather than drops it... But that's not a huge issue

90

1

	for you as you have bandwidth to burn.

91

1

<bmc> <FONT>Man, this dream plug is sweet. Tiny, small footprint (physical and

92

1

      electrical), quiet as all fuck, and real *nix, to boot.</FONT>

93

1

<dtaht> thx for tryin it. How does the 'dreamplug' feel? Is it a better piece

94

1

	of gear?

95

1

<dtaht> hahha

96

1

<dtaht> answered my question in advance  [07:54]

97

1

<bmc> <FONT>Way better. I resurrected the GuruPlug, via the JTAG. Trying to

98

1

      decide whether I have a use for it.</FONT>

99

1

<bmc> <FONT>Or whether I should give it to someone (e.g., Costine) and spread

100

1

      the love.</FONT>

101

1

<dtaht> yea, I loved the openrd, but the interviening releases of sheevaplug

102

1

	and guruplug left me cold. I'm also not happy with the wireless chip,

103

1

	but that's relatively minor.

104

1

<dtaht> spread the love, man....

105

1

<bmc> <FONT>That's what I'm thinking. … Okay, time for a conf call. By then,

106

1

      my kid should be awake and ready for waffles.</FONT>  [07:55]

107

1

<bmc> <FONT>BTW, the wireless is working just fine here.</FONT>

108

1

<bmc> <FONT>I use rc.local to adjust the params via uapctl.</FONT>

109

1

<dtaht> yea, well, start moving to the edge of the range and then watch your

110

1

	ping times....

111

1

<bmc> <FONT>Not an issue here.</FONT>

112

1

* dtaht plans to get one of these puppies too, after bmc is happy for a few

113

1

  weeks.

114

1

<bmc> <FONT>I have four WAPs in this interference-laden place. :-)</FONT>

115

1

* dtaht is using diversity mesh routing now with babel  [07:56]

116

1

<dtaht> babel + ahcpd is sweet

117

1

<dtaht> sit down, plug in the laptop

118

1

<dtaht> it figures out it's on wired

119

1

<bmc> <FONT>Something to look into, when I have time. If I ever have time

120

1

      again.</FONT>

121

1

<dtaht> unplug, it figures out you are on wireless

122

1

<bmc> <FONT>Niiiice.</FONT>

123

1

<dtaht> streams, and connections, STAY UP

124

1

<bmc> <FONT>Very sweet.</FONT>

125

1

<bmc> <FONT>Okay, time to make that call. Later.</FONT>

126

1

<dtaht> yea, it's amazing how freeing it is to be able to plug in again.

127

1

<dtaht> later  [07:57]

128

1

<dtaht> thx

129

1

<dtaht> also the mesh routing 'diversity' means that I have several nodes that

130

1

	route over 5.x ghz and are aps over 2.4, and vice versa... Connect

131

1

	your neighbors

132

1

<dtaht> whenever you get time I wanted to know how that meeting turned out.

133

1

								        [07:58]

134

1

<bmc> <FONT>What meeting?</FONT>  [08:00]

135

1

<dtaht> some users group meeting of all the users groups  [08:07]

136

1

<bmc> <FONT>Right. I went to a lot of meetings last week. :-) That went

137

1

      well. We're moving forward, trying to get common venues, common

138

1

      calendars, etc.</FONT>  [08:10]

139

1

<bmc> <FONT>FYI, on Mac OS X:</FONT>  [08:12]

140

1

<bmc> <FONT>$ sysctl -a | egrep 'ecn|sack'<BR>net.inet.tcp.ecn_initiate_out:

141

1

      0<BR>net.inet.tcp.ecn_negotiate_in: 0<BR>net.inet.tcp.sack:

142

1

      1<BR>net.inet.tcp.sack_maxholes:

143

1

      128<BR>net.inet.tcp.sack_globalmaxholes:

144

1

      65536<BR>net.inet.tcp.sack_globalholes: 0<BR>net.inet.ipsec.ecn:

145

1

      0<BR>net.inet6.ipsec6.ecn: 0<BR></FONT>

146

1

<dtaht> heh. THANK YOU  [08:22]

147

1

* dtaht wanted a mesh network in philly to implement.

148

1

<bmc> <FONT>Another data point: (dancer:bmc) ~ $ uname -s -r  <BR>FreeBSD

149

1

      8.2-RELEASE<BR>(dancer:bmc) ~ $ sysctl -a | egrep

150

1

      'sack|ecn'<BR>vfs.bufreusecnt: 932<BR>net.inet.tcp.ecn.maxretries:

151

1

      1<BR>net.inet.tcp.ecn.enable: 0<BR>net.inet.tcp.sack.globalholes:

152

1

      0<BR>net.inet.tcp.sack.globalmaxholes:

153

1

      65536<BR>net.inet.tcp.sack.maxholes: 128<BR>net.inet.tcp.sack.enable:

154

1

      1<BR>net.inet.sctp.enable_sack_immediately:  [08:48]

155

1

<bmc> 0<BR>net.inet.sctp.nr_sack_on_off: 0<BR>net.inet.sctp.sack_freq:

156

1

      2<BR>net.inet.sctp.delayed_sack_time: 200<BR>net.inet.sctp.strict_sacks:

157

1

      1<BR>net.inet.sctp.ecn_nonce: 0<BR>net.inet.sctp.ecn_enable:

158

1

      1<BR></FONT>

159

1

<dtaht> sight  [08:50]

160

1

<dtaht> sigh  [08:51]

161

1

<bmc> <FONT>They're all different.</FONT>

162

1

<bmc> <FONT>I have OpenSolaris here, too. Want that data point?</FONT>

163

1

<dtaht> sure

164

1

<bmc> <FONT>Booting...</FONT>

165

1

* dtaht just had 3 new olpcs arrive and is mildly distracted

166

1

<bmc> <FONT>Nice.</FONT>  [08:52]

167

1

<bmc> <FONT>They make good nightlights, I'm told.</FONT>

168

1

<dtaht> the 1.5s are better nightlights

169

1

<dtaht> the 1.75s (I'm on the list) are cool

170

1

<dtaht> I'm trying to convince them to add 5.x ghz support

171

1

<bmc> <FONT>Every child should have that.</FONT>

172

1

<dtaht> hahahaha

173

1

<bmc> <FONT>Hmm… SunOS 5.11. sysctl not found...</FONT>  [08:53]

174

1

<bmc> <FONT>$ ndd /dev/tcp \?|egrep -i 'ecn|sack'<BR>tcp_sack_permitted

175

1

      (read and write)<BR>tcp_ecn_permitted              (read and

176

1

      write)<BR></FONT>  [08:58]

177

1

<bmc> <FONT>(sunball:bmc) /etc $ ndd -get /dev/tcp

178

1

      tcp_sack_permitted<BR>2<BR>(sunball:bmc) /etc $ ndd -get /dev/tcp

179

1

      tcp_ecn_permitted<BR>1<BR></FONT>  [08:59]

180

1

<bmc>

181

1

      <FONT>http://download.oracle.com/docs/cd/E19963-01/html/821-1450/chapter4-31.html</FONT>

182

1

<bmc> <FONT>(Oracle Solaris Tunable Parameters Reference Manual)</FONT>

183

1

<bmc> <FONT>ecn: 0 (disabled), 1 (passive enabled), or 2 (active enabled)

184

1

      </FONT>  [09:00]

185

1

<bmc> <FONT>Same values for SACK</FONT>

186

1

<bmc> <FONT>Defaults are as shown above.</FONT>  [09:01]

187

1

<dtaht> excellent.  [09:09]

188

1

<bmc> <FONT>I just tweeted this: When you've spent the last decade mostly

189

1

      using Linux and BSD, the infrequent foray into Solaris feels like going

190

1

      to Mars. #unix</FONT>

191

1

<dtaht> So if I get you to turn it on, that will only leave about 2 billion

192

1

	computers left to fix and 10s of thousands or routers left to junk

193

1

								        [09:10]

194

1

<dtaht> heheheh

195

1

<dtaht> solaris is like 1998

196

1

<bmc> <FONT>I don't even want to think about how one accomplishes this in

197

1

      HP/UX.</FONT>

198

1

<bmc> <FONT>Assuming it's even possible.</FONT>

199

1

<dtaht> thx for the data I'm going to update the wiki in a bit

200

1

<bmc> <FONT>Or, for that matter, AIX, which makes Mars seem familiar.</FONT>

201

1

<dtaht> hahahah

202

1

<dtaht> do these things have sysctl.conf?

203

1

<bmc> <FONT>I didn't see a reference to an ndd.conf</FONT>  [09:11]

204

1

<bmc> <FONT>http://www.sean.de/Solaris/soltune.html#ndd</FONT>

205

1

<bmc> <FONT>Worth skimming.</FONT>  [09:12]

206

1

<bmc> <FONT>Of course, Solaris no longer has an /etc/rc.local, either.</FONT>

207

1

<bmc> <FONT>Instead, you're supposed to create something in /etc/init.d

208

1

      (rc.local is fine), then symlink it to an "S" file in the appropriate

209

1

      runlevel directory (e.g., rc3.d).</FONT>

210

1

<bmc> <FONT>*sigh*</FONT>

211

1

<bmc> <FONT>Wait...</FONT>  [09:13]

212

1

<bmc> <FONT>I just ran across this:</FONT>

213

1

<bmc> <FONT>To set parameters so they remain in effect after you reboot the

214

1

      system, add the parameter values to /etc/system when you want to

215

1

      configure parameters for all devices in the system.</FONT>

216

1

<bmc> <FONT>A startup script can also be used to set a ndd parameters across

217

1

      system reboots. Include the appropriate ndd command in a system startup

218

1

      script, such as the /etc/init.d/inetinit file or a customized script in

219

1

      /etc/rc2.d or /etc/rc3.d. Be sure to make a copy of any files before

220

1

      adding the ndd commands.</FONT>  [09:14]

221

1

ERC>