Bug #385
more traps
| Status: | New | Start date: | 05/09/2012 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assignee: | - | % Done: | 0% |
|
| Category: | - | Spent time: | - | |
| Target version: | 1st Public Cerowrt release |
Description
some notes while fiddling with codel in the light of #379
Related issues
| related to Cerowrt - Bug #379: 3.3.4-3 router crashes under heavy load | New | |||
| related to Cerowrt - Bug #360: Patchwork [OpenWrt-Devel] ar71xx TCP/IPsec unaligned inst... | Closed | 04/11/2012 | ||
| related to Cerowrt - Bug #371: route table updates perturbed by alignment errors | New | 04/20/2012 | ||
| related to Cerowrt - Bug #386: QFQ locks up | New | 05/09/2012 |
History
Updated by Dave Täht about 1 year ago
This is under heavy load. On the plus side, overall delays stay at about 5ms.
The ntp restart problem is a different bug.
And I only end up with 24 traps after a few minutes
root@codel:/sys/kernel/debug/mips# May 10 01:45:52 codel daemon.err ntpd4128: i/o error on routing socket No buffer space available - disabling
May 10 01:45:52 codel kern.warn kernel: [ 860.550781] Cpu 0
May 10 01:45:52 codel kern.warn kernel: [ 860.554687] $ 0 : 00000000 00000000 00000000 00000000
May 10 01:45:52 codel kern.warn kernel: [ 860.558593] $ 4 : 00502478 7fec7870 00000001 00000000
May 10 01:45:52 codel kern.warn kernel: [ 860.566406] $ 8 : 000007ff 001a0041 83b68480 00000001
May 10 01:45:52 codel kern.warn kernel: [ 860.570312] $12 : 00000019 00190041 00000001 00480000
May 10 01:45:52 codel kern.warn kernel: [ 860.574218] $16 : 72670001 00480000 00000001 00000002
May 10 01:45:52 codel kern.warn kernel: [ 860.582031] $20 : 00000001 004878d4 00487954 00480000
May 10 01:45:52 codel kern.warn kernel: [ 860.585937] $24 : 004840ec 76f2c100
May 10 01:45:52 codel kern.warn kernel: [ 860.593750] $28 : 76f48030 7fec7850 00480000 0040f478
May 10 01:45:52 codel kern.warn kernel: [ 860.597656] Hi : 00000000
May 10 01:45:52 codel kern.warn kernel: [ 860.601562] Lo : 00000001
May 10 01:45:52 codel kern.warn kernel: [ 860.605468] epc : 0040f444 0x40f444
May 10 01:45:52 codel kern.warn kernel: [ 860.609375] Tainted: G O
May 10 01:45:52 codel kern.warn kernel: [ 860.613281] ra : 0040f478 0x40f478
May 10 01:45:52 codel kern.warn kernel: [ 860.617187] Status: 0000fc13 USER EXL IE
May 10 01:45:52 codel kern.warn kernel: [ 860.621093] Cause : 00800010
May 10 01:45:52 codel kern.warn kernel: [ 860.625000] BadVA : 72670005
May 10 01:45:52 codel kern.warn kernel: [ 860.625000] PrId : 00019374 (MIPS 24Kc)
May 10 01:45:52 codel kern.warn kernel: [truncated] [ 860.628906] Modules linked in: sch_codel cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip
May 10 01:45:52 codel kern.warn kernel: [ 860.781250] Process ntpd (pid: 4128, threadinfo=82b42000, task=83f05d18, tls=77171750)
May 10 01:45:52 codel kern.warn kernel: [ 860.789062] Stack : 20f1f3b0 76f05cfc 76f1f3b0 0000000b 7fec788c 7fec792d 00000000 76ee1668
May 10 01:45:52 codel kern.warn kernel: [ 860.796875] 00000001 770bbc50 00487954 770e717c 00000010 d3559c50 41eb9350 00000000
May 10 01:45:52 codel kern.warn kernel: [ 860.804687] 00000000 00000000 00000400 00000000 00000000 00000000 00000000 00000000
May 10 01:45:52 codel kern.warn kernel: [ 860.816406] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May 10 01:45:52 codel kern.warn kernel: [ 860.824218] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May 10 01:45:52 codel kern.warn kernel: [ 860.832031] ...
May 10 01:45:52 codel kern.warn kernel: [ 860.835937] Call Trace:
May 10 01:45:52 codel kern.warn kernel: [ 860.835937] [<80138521>] jffs2_do_read_inode_internal+0xb69/0x1654
May 10 01:45:52 codel kern.warn kernel: [ 860.843750]
May 10 01:45:52 codel kern.warn kernel: [ 860.843750]
May 10 01:45:52 codel kern.warn kernel: [ 860.843750] Code: 8e100000 12000012 3c110048 <8e020004> 8fa504b0 00021942 00031880 00a31821 8c630024
May 10 01:45:56 codel daemon.info named3820: success resolving 'ns1.lab.bufferbloat.net/AAAA' (in 'lab.bufferbloat.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
May 10 01:45:56 codel daemon.info named3820: success resolving 'ns1.lab.bufferbloat.net/A' (in 'lab.bufferbloat.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Updated by Dave Täht about 1 year ago
ecn_mark 0 drop_overlimit 0 states 2407396 : 234189 95 170 root@codel:/sys/kernel/debug/mips# tc -s qdisc show dev se00 qdisc codel 8002: root refcnt 2 limit 1000p target 5.0ms interval 100.0ms Sent 3881154957 bytes 5110579 pkt (dropped 271, overlimits 0 requeues 112483) backlog 206172b 138p requeues 112483 maxpacket 1494 count 3 lastcount 2 ldelay 7.0ms ecn_mark 0 drop_overlimit 0 states 2433428 : 240043 97 174 root@codel:/sys/kernel/debug/mips# tc -s qdisc show dev se00 qdisc codel 8002: root refcnt 2 limit 1000p target 5.0ms interval 100.0ms Sent 3917649177 bytes 5135009 pkt (dropped 278, overlimits 0 requeues 113623) backlog 178Kb 122p requeues 113623 maxpacket 1494 count 2 lastcount 1 ldelay 6.8ms ecn_mark 0 drop_overlimit 0 states 2457852 : 245960 98 180 root@codel:/sys/kernel/debug/mips# tc -s qdisc show dev se00 qdisc codel 8002: root refcnt 2 limit 1000p target 5.0ms interval 100.0ms Sent 3959334937 bytes 5162913 pkt (dropped 287, overlimits 0 requeues 115022) backlog 177786b 120p requeues 115022 maxpacket 1494 count 3 lastcount 1 ldelay 6.4ms ecn_mark 0 drop_overlimit 0 states 2485756 : 252529 100 187 root@codel:/sys/kernel/debug/mips# tc -s qdisc show dev se00 May 10 01:52:13 codel kern.warn kernel: [ 1231.812500] Cpu 0 May 10 01:52:13 codel kern.warn kernel: [ 1231.816406] $ 0 : 00000000 00000001 00000000 00000000 May 10 01:52:13 codel kern.warn kernel: [ 1231.820312] $ 4 : 00000000 00000500 00000000 00f45d2c May 10 01:52:13 codel kern.warn kernel: [ 1231.824218] $ 8 : 00000000 00000000 00000000 00000000 May 10 01:52:13 codel kern.warn kernel: [ 1231.832031] $12 : 00060000 00000bac 00000000 00000000 May 10 01:52:13 codel kern.warn kernel: [ 1231.835937] $16 : 00000028 81f01b40 8332e600 000004d8 May 10 01:52:13 codel kern.warn kernel: [ 1231.839843] $20 : 833ef896 0000003b 833ef86e 82a20000 May 10 01:52:13 codel kern.warn kernel: [ 1231.847656] $24 : 00000000 831c318c May 10 01:52:13 codel kern.warn kernel: [ 1231.851562] $28 : 8326e000 8326f838 00000001 82a262bc May 10 01:52:13 codel kern.warn kernel: [truncated] [ 1231.855468] Hi : 000e0029eader ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack May 10 01:52:13 codel kern.warn kernel: [ 1232.945312] Process named (pid: 3820, threadinfo=8326e000, task=82bd8000, tls=77ce2440) May 10 01:52:13 codel kern.warn kernel: [ 1232.953125] Stack : 00000000 831c5a5c 00000001 8020cc14 8332e300 81f01b40 00000000 83d44000 May 10 01:52:13 codel kern.warn kernel: [ 1232.964843] 00000000 8326f7f0 8332e300 83ec3600 831f5ad0 0000002c fffffffe 00000006 May 10 01:52:13 codel kern.warn kernel: [ 1232.972656] 831fa6f4 30000000 80360000 831c5cb8 83d44000 00000000 fffffe71 831c63c0 May 10 01:52:13 codel kern.warn kernel: [ 1232.980468] 8332e300 81f01b40 803013fc 831c5a10 00000000 00000000 83d44000 00000000 May 10 01:52:13 codel kern.warn kernel: [ 1232.988281] fffffe71 82a26bc8 1b5f435e 00000001 833ed076 833ed086 00000000 831c5a10 May 10 01:52:13 codel kern.warn kernel: [ 1232.996093] ... May 10 01:52:13 codel kern.warn kernel: [ 1233.000000] Call Trace: May 10 01:52:13 codel kern.warn kernel: [ 1233.000000] [<831e5880>] ip6_frag_init+0x270/0xe38 [ipv6] May 10 01:52:13 codel kern.warn kernel: [ 1233.007812] May 10 01:52:13 codel kern.warn kernel: [ 1233.007812] May 10 01:52:13 codel kern.warn kernel: [ 1233.007812] Code: ac640088 ac62008c 26f0b194 <8e640004> 8e070100 26250008 26260018 afa40010 afa50018 May 10 01:52:13 codel kern.warn kernel: [ 1233.164062] Cpu 0 May 10 01:52:13 codel kern.warn kernel: [ 1233.167968] $ 0 : 00000000 00000001 00000000 00000000 May 10 01:52:13 codel kern.warn kernel: [ 1233.171875] $ 4 : 00000000 00000500 00000000 00f47783 May 10 01:52:13 codel kern.warn kernel: [ 1233.175781] $ 8 : 00000000 00000000 00000000 00000000 May 10 01:52:13 codel kern.warn kernel: [ 1233.183593] $12 : 00060000 a4000027 a4000042 00000000 May 10 01:52:13 codel kern.warn kernel: [ 1233.187500] $16 : 00000028 82aa9180 831409c0 000004d8 May 10 01:52:13 codel kern.warn kernel: [ 1233.191406] $20 : 82a83096 0000003b 82a8306e 82a20000 May 10 01:52:13 codel kern.warn kernel: [ 1233.199218] $24 : 00000000 831c318c May 10 01:52:13 codel kern.warn kernel: [ 1233.203125] $28 : 82bbc000 82bbdca8 00000001 82a262bc May 10 01:52:13 codel kern.warn kernel: [ 1233.207031] Hi : 000cf4e8 May 10 01:52:13 codel kern.warn kernel: [ 1233.210937] Lo : 91a09954 May 10 01:52:13 codel kern.warn kernel: [ 1233.214843] epc : 82a2632c nf_ct_frag6_gather+0x180/0x948 [nf_defrag_ipv6] May 10 01:52:13 codel kern.warn kernel: [ 1233.222656] Tainted: G O May 10 01:52:13 codel kern.warn kernel: [ 1233.226562] ra : 82a262bc nf_ct_frag6_gather+0x110/0x948 [nf_defrag_ipv6] May 10 01:52:13 codel kern.warn kernel: [ 1233.230468] Status: 1000fc03 KERNEL EXL IE May 10 01:52:13 codel kern.warn kernel: [ 1233.234375] Cause : 00800010 May 10 01:52:13 codel kern.warn kernel: [ 1233.238281] BadVA : 82a8309a May 10 01:52:13 codel kern.warn kernel: [ 1233.242187] PrId : 00019374 (MIPS 24Kc) May 10 01:52:13 codel kern.warn kernel: [truncated] [ 1233.246093] Modules linked in: sch_codel cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip May 10 01:52:13 codel kern.warn kernel: [ 1233.390625] Process hostapd (pid: 2166, threadinfo=82bbc000, task=82bd9d18, tls=772ee440) May 10 01:52:13 codel kern.warn kernel: [ 1233.398437] Stack : 0a00c800 831f3878 83f85f80 00000000 06000001 83f852b0 82bbdd48 82bbdd48 May 10 01:52:13 codel kern.warn kernel: [ 1233.406250] 00000006 83d44000 8030141c 00000004 00000000 831409c0 83d44000 00000000 May 10 01:52:13 codel kern.warn kernel: [ 1233.414062] 00000000 83d44000 00000001 00000004 00000000 82a2607c 80350000 8009766c May 10 01:52:14 codel kern.warn kernel: [ 1233.421875] 00000004 00000000 0000000a 8030141c 00000000 82bbdd90 82a26ee0 831409c0 May 10 01:52:14 codel kern.warn kernel: [ 1233.429687] 803013fc 8020cb24 83c3f000 83d34100 00000000 80300000 831c5a10 80300b38 May 10 01:52:14 codel kern.warn kernel: [ 1233.441406] ... May 10 01:52:14 codel kern.warn kernel: [ 1233.441406] Call Trace: May 10 01:52:14 codel kern.warn kernel: [ 1233.445312] [<82a2632c>] nf_ct_frag6_gather+0x180/0x948 [nf_defrag_ipv6] May 10 01:52:14 codel kern.warn kernel: [ 1233.453125] [<82a2607c>] nf_defrag_ipv6_enable+0x7c/0x1ac [nf_defrag_ipv6] May 10 01:52:14 codel kern.warn kernel: [ 1233.457031] May 10 01:52:14 codel kern.warn kernel: [ 1233.460937] May 10 01:52:14 codel kern.warn kernel: [ 1233.460937] Code: ae340090 0ca89834 00000000 <8e930004> afbe0014 3c1e82a2 26d50008 27d0712c 26d60018 May 10 01:52:14 codel kern.warn kernel: [ 1233.472656] Cpu 0 May 10 01:52:14 codel kern.warn kernel: [ 1233.472656] $ 0 : 00000000 00000001 00000000 00000948 May 10 01:52:14 codel kern.warn kernel: [ 1233.480468] $ 4 : 00000000 0000019c 00000000 00f47784 May 10 01:52:14 codel kern.warn kernel: [ 1233.484375] $ 8 : 00000000 00000000 00000000 00000000 May 10 01:52:14 codel kern.warn kernel: [ 1233.488281] $12 : 00060000 a4000027 a4000042 00000000 May 10 01:52:14 codel kern.warn kernel: [ 1233.496093] $16 : 00000028 82aa9900 83140e40 00000174 qdisc codel 8002: root refcnt 2 limit 1000p target 5.0ms interval 100.0ms Sent 4001191037 bytes 5190931 pkt (dropped 293, overlimits 0 requeues 116341) backlog 219618b 147p requeues 116341 maxpacket 1494 count 1 lastcount 1 ldelay 7.3ms ecn_mark 0 drop_overlimit 0 states 2513767 : 256573 102 191 root@codel:/sys/kernel/debug/mips# May 10 01:52:14 codel kern.warn kernel: [ 1233.500000] $20 : 82a80896 0000003b 82a8086e 82a20000 May 10 01:52:14 codel kern.warn kernel: [ 1233.503906] $24 : 00000000 831c318c May 10 01:52:14 codel kern.warn kernel: [ 1233.511718] $28 : 82bbc000 82bbdca8 00000001 82a262bc May 10 01:52:14 codel kern.warn kernel: [ 1233.515625] Hi : 0008287b May 10 01:52:14 codel kern.warn kernel: [ 1233.519531] Lo : 66dd43c4 May 10 01:52:14 codel kern.warn kernel: [ 1233.519531] epc : 82a2632c nf_ct_frag6_gather+0x180/0x948 [nf_defrag_ipv6] May 10 01:52:14 codel kern.warn kernel: [ 1233.527343] Tainted: G O May 10 01:52:14 codel kern.warn kernel: [ 1233.531250] ra : 82a262bc nf_ct_frag6_gather+0x110/0x948 [nf_defrag_ipv6] May 10 01:52:14 codel kern.warn kernel: [ 1233.539062] Status: 1000fc03 KERNEL EXL IE May 10 01:52:14 codel kern.warn kernel: [ 1233.542968] Cause : 00800010 May 10 01:52:14 codel kern.warn kernel: [ 1233.546875] BadVA : 82a8089a May 10 01:52:14 codel kern.warn kernel: [ 1233.550781] PrId : 00019374 (MIPS 24Kc) May 10 01:52:14 codel kern.warn kernel: [truncated] [ 1233.554687] Modules linked in: sch_codel cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip May 10 01:52:14 codel kern.warn kernel: [ 1233.695312] Process hostapd (pid: 2166, threadinfo=82bbc000, task=82bd9d18, tls=772ee440) May 10 01:52:14 codel kern.warn kernel: [ 1233.707031] Stack : 0a00c800 831f3878 83f85f80 00000000 a79a55ba 00000001 82a83076 82a83086 May 10 01:52:14 codel kern.warn kernel: [ 1233.714843] 00000006 83d44000 8030141c 00000004 00000000 83140e40 83d44000 00000000 May 10 01:52:14 codel kern.warn kernel: [ 1233.722656] 00000000 83d44000 00000001 00000004 00000000 82a2607c 80350000 8009766c May 10 01:52:14 codel kern.warn kernel: [ 1233.730468] 00000004 00000000 0000000a 8030141c 00000000 82bbdd90 82a26ee0 83140e40 May 10 01:52:14 codel kern.warn kernel: [ 1233.738281] 803013fc 8020cb24 83c3f000 83d34100 00000000 80300000 831c5a10 80300b38 May 10 01:52:14 codel kern.warn kernel: [ 1233.746093] ... May 10 01:52:14 codel kern.warn kernel: [ 1233.750000] Call Trace: May 10 01:52:14 codel kern.warn kernel: [ 1233.753906] [<82a2632c>] nf_ct_frag6_gather+0x180/0x948 [nf_defrag_ipv6] May 10 01:52:14 codel kern.warn kernel: [ 1233.757812] [<82a2607c>] nf_defrag_ipv6_enable+0x7c/0x1ac [nf_defrag_ipv6] May 10 01:52:14 codel kern.warn kernel: [ 1233.765625] May 10 01:52:14 codel kern.warn kernel: [ 1233.765625] May 10 01:52:14 codel kern.warn kernel: [ 1233.765625] Code: ae340090 0ca89834 00000000 <8e930004> afbe0014 3c1e82a2 26d50008 27d0712c 26d60018 May 10 01:52:14 codel kern.warn kernel: [ 1233.777343] Cpu 0 May 10 01:52:14 codel kern.warn kernel: [ 1233.781250] $ 0 : 00000000 00000001 00000000 00000000 May 10 01:52:14 codel kern.warn kernel: [ 1233.785156] $ 4 : 00000000 000004d8 00000000 00000000 May 10 01:52:14 codel kern.warn kernel: [ 1233.792968] $ 8 : 00000000 00000000 00000000 00000030 May 10 01:52:14 codel kern.warn kernel: [ 1233.796875] $12 : 00000000 00000000 00000000 00000000 May 10 01:52:14 codel kern.warn kernel: [ 1233.800781] $16 : 831fb194 82a8306e 831409c0 82a83096 May 10 01:52:14 codel kern.warn kernel: [ 1233.808593] $20 : 80360000 80360000 831fa6f4 83200000 May 10 01:52:14 codel kern.warn kernel: [ 1233.812500] $24 : 00000000 831c6374 May 10 01:52:14 codel kern.warn kernel: [ 1233.816406] $28 : 82bbc000 82bbdc30 80360000 831c5cb8 May 10 01:52:14 codel kern.warn kernel: [ 1233.824218] Hi : 00000000 May 10 01:52:14 codel kern.warn kernel: [ 1233.824218] Lo : 00000001 May 10 01:52:14 codel kern.warn kernel: [ 1233.828125] epc : 831e5880 ip6_frag_init+0x270/0xe38 [ipv6] May 10 01:52:14 codel kern.warn kernel: [ 1233.835937] Tainted: G O May 10 01:52:15 codel kern.warn kernel: [ 1233.839843] ra : 831c5cb8 ip6_rcv_finish+0x2a8/0x4a8 [ipv6] May 10 01:52:15 codel kern.warn kernel: [ 1233.843750] Status: 1000fc03 KERNEL EXL IE May 10 01:52:15 codel kern.warn kernel: [ 1233.847656] Cause : 00800010 May 10 01:52:15 codel kern.warn kernel: [ 1233.851562] BadVA : 82a8309a May 10 01:52:15 codel kern.warn kernel: [ 1233.855468] PrId : 00019374 (MIPS 24Kc) May 10 01:52:15 codel kern.warn kernel: [truncated] [ 1233.859375] Modules linked in: sch_codel cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip May 10 01:52:15 codel kern.warn kernel: [ 1234.003906] Process hostapd (pid: 2166, threadinfo=82bbc000, task=82bd9d18, tls=772ee440) May 10 01:52:15 codel kern.warn kernel: [ 1234.011718] Stack : 00000000 831c5a5c 00000001 8020cc14 831409c0 82aa9180 00000000 83d44000 May 10 01:52:15 codel kern.warn kernel: [ 1234.019531] 00000000 82bbdc60 831409c0 83ec3600 831f5ad0 0000002c fffffffe 00000006 May 10 01:52:15 codel kern.warn kernel: [ 1234.027343] 831fa6f4 30000000 80360000 831c5cb8 83d44000 00000000 fffffe71 831c63c0 May 10 01:52:15 codel kern.warn kernel: [ 1234.035156] 831409c0 82aa9180 803013fc 831c5a10 00000000 83140e40 83d44000 00000000 May 10 01:52:15 codel kern.warn kernel: [ 1234.042968] fffffe71 82a26bc8 a79a55ba 00000001 82a80876 82a80886 00000000 831c5a10 May 10 01:52:15 codel kern.warn kernel: [ 1234.050781] ... May 10 01:52:15 codel kern.warn kernel: [ 1234.054687] Call Trace: May 10 01:52:15 codel kern.warn kernel: [ 1234.058593] [<831e5880>] ip6_frag_init+0x270/0xe38 [ipv6] May 10 01:52:15 codel kern.warn kernel: [ 1234.062500] May 10 01:52:15 codel kern.warn kernel: [ 1234.062500] May 10 01:52:15 codel kern.warn kernel: [ 1234.062500] Code: ac640088 ac62008c 26f0b194 <8e640004> 8e070100 26250008 26260018 afa40010 afa50018 May 10 01:52:15 codel kern.warn kernel: [ 1234.074218] Cpu 0 May 10 01:52:15 codel kern.warn kernel: [ 1234.078125] $ 0 : 00000000 00000001 00000000 00000950 May 10 01:52:15 codel kern.warn kernel: [ 1234.082031] $ 4 : 00000000 00000174 00000000 00000000 May 10 01:52:15 codel kern.warn kernel: [ 1234.089843] $ 8 : 00000000 00000000 00000000 00000030 May 10 01:52:15 codel kern.warn kernel: [ 1234.093750] $12 : 00000000 00000000 00000000 00000000 May 10 01:52:15 codel kern.warn kernel: [ 1234.097656] $16 : 831fb194 82a8086e 83140e40 82a80896 May 10 01:52:15 codel kern.warn kernel: [ 1234.105468] $20 : 80360000 80360000 831fa6f4 83200000 May 10 01:52:15 codel kern.warn kernel: [ 1234.109375] $24 : 00000000 831c6374 May 10 01:52:15 codel kern.warn kernel: [ 1234.113281] $28 : 82bbc000 82bbdc30 80360000 831c5cb8 May 10 01:52:15 codel kern.warn kernel: [ 1234.121093] Hi : 00000000 May 10 01:52:15 codel kern.warn kernel: [ 1234.121093] Lo : 00000001 May 10 01:52:15 codel kern.warn kernel: [ 1234.125000] epc : 831e5880 ip6_frag_init+0x270/0xe38 [ipv6] May 10 01:52:15 codel kern.warn kernel: [ 1234.132812] Tainted: G O May 10 01:52:15 codel kern.warn kernel: [ 1234.132812] ra : 831c5cb8 ip6_rcv_finish+0x2a8/0x4a8 [ipv6] May 10 01:52:15 codel kern.warn kernel: [ 1234.140625] Status: 1000fc03 KERNEL EXL IE May 10 01:52:15 codel kern.warn kernel: [ 1234.144531] Cause : 00800010 May 10 01:52:15 codel kern.warn kernel: [ 1234.148437] BadVA : 82a8089a May 10 01:52:15 codel kern.warn kernel: [ 1234.152343] PrId : 00019374 (MIPS 24Kc) May 10 01:52:15 codel kern.warn kernel: [truncated] [ 1234.156250] Modules linked in: sch_codel cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip May 10 01:52:15 codel kern.warn kernel: [ 1234.296875] Process hostapd (pid: 2166, threadinfo=82bbc000, task=82bd9d18, tls=772ee440) May 10 01:52:15 codel kern.warn kernel: [ 1234.308593] Stack : 00000000 831c5a5c 00000001 8020cc14 83140e40 82aa9180 00000000 83d44000 May 10 01:52:15 codel kern.warn kernel: [ 1234.316406] 00000000 82bbdc60 83140e40 83ec3600 831f5ad0 0000002c fffffffe 00000006 May 10 01:52:15 codel kern.warn kernel: [ 1234.324218] 831fa6f4 30000000 80360000 831c5cb8 83d44000 00000000 fffffe71 831c63c0 May 10 01:52:15 codel kern.warn kernel: [ 1234.332031] 83140e40 82aa9180 803013fc 831c5a10 00000000 00000000 83d44000 00000000 May 10 01:52:15 codel kern.warn kernel: [ 1234.339843] fffffe71 82a26bc8 a79a55ba 00000001 82a80876 82a80886 00000000 831c5a10 May 10 01:52:15 codel kern.warn kernel: [ 1234.347656] ... May 10 01:52:15 codel kern.warn kernel: [ 1234.351562] Call Trace: May 10 01:52:15 codel kern.warn kernel: [ 1234.355468] [<831e5880>] ip6_frag_init+0x270/0xe38 [ipv6] May 10 01:52:15 codel kern.warn kernel: [ 1234.359375] May 10 01:52:15 codel kern.warn kernel: [ 1234.359375] May 10 01:52:15 codel kern.warn kernel: [ 1234.359375] Code: ac640088 ac62008c 26f0b194 <8e640004> 8e070100 26250008 26260018 afa40010 afa50018 root@codel:/sys/kernel/debug/mips# May 10 01:52:18 codel daemon.info named[3820]: success resolving 'ns2.isc.ultradns.net/A' (in 'ultradns.net'?) after reducing the advertised EDNS UDP packet size to 512 octets May 10 01:52:19 codel daemon.info named[3820]: success resolving 'ns1.isc.ultradns.net/A' (in 'ultradns.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Updated by Dave Täht about 1 year ago
the neat thing about the previous test is that I was not running wirelessly
Updated by Dave Täht about 1 year ago
I thought count couldn't drop below 1.
qdisc mq 1: root
Sent 11734 bytes 52 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc codel 10: parent 1:1 ecn limit 1000p target 5.0ms interval 100.0ms
Sent 2075 bytes 18 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 256 count 0 lastcount 0 ldelay 0us
ecn_mark 0 drop_overlimit 0 states 1 : 0 0 0
qdisc codel 20: parent 1:2 ecn limit 1000p target 5.0ms interval 100.0ms
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 256 count 0 lastcount 0 ldelay 0us
ecn_mark 0 drop_overlimit 0 states 0 : 0 0 0
qdisc codel 30: parent 1:3 ecn limit 1000p target 5.0ms interval 100.0ms
Sent 9659 bytes 34 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 256 count 0 lastcount 0 ldelay 0us
ecn_mark 0 drop_overlimit 0 states 11 : 0 0 0
qdisc codel 40: parent 1:4 ecn limit 1000p target 5.0ms interval 100.0ms
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 256 count 0 lastcount 0 ldelay 0us
ecn_mark 0 drop_overlimit 0 states 0 : 0 0 0
Updated by Robert Bradley about 1 year ago
- File ip6_frag_init.patch added
Well, having looked at ip6_frag_init, I'm finding it difficult to see any obvious causes except for:
1 fq->saddr = *arg->src;
2 fq->daddr = *arg->dst;
Yes, that's dereferencing an in6_addr pointer (ultimately obtained from the incoming packet) and assigning the result to an aligned address. And yes, in6_addr structs were supposed to be marked packed. You may also need to mark frag_hdr (in /include/net/ipv6.h) as packed too, if it isn't already.
Updated by Robert Bradley about 1 year ago
This might be a little crazy, but would it be worth trying to stress-test the IPv4 equivalent (ip4_frag_init) too? "ping -s 4096 172.30.42.1" ought to manage it.
Updated by Robert Bradley about 1 year ago
- File ip6_frag_init.patch added
I think in that last patch, "sizeof(in6_addr)" might need to be "sizeof(struct in6_addr)". My local working copy had these changes uncommitted. :(
Updated by Dave Täht about 1 year ago
had already packed frag_hdr, thought using memcpy was crazy, trying it anyway.
Updated by Dave Täht about 1 year ago
also note jffs2 one. that proved hard to find. (no, haven't found it, breaking out heavy artillery for that one)
Updated by Dave Täht about 1 year ago
container_of?
Updated by Robert Bradley about 1 year ago
I don't think it can be container_of, since that is only doing pointer arithmetic, not actually dereferencing anything (from http://lxr.free-electrons.com/source/include/linux/kernel.h#L661):
1 #define container_of(ptr, type, member) ({ \
2 const typeof( ((type *)0)->member ) *__mptr = (ptr); \
3 (type *)( (char *)__mptr - offsetof(type,member) );})
In fact, I couldn't see anything in qfq that actually touched the network headers in skb, as opposed to kernel-derived fields. Hence why I'm hoping we can blame cls_u32...
(It could be cls_flow, but again, that's outsourcing everything potentially risky to skb_flow_dissect and reading the aligned skb fields afterwards. That function got looked at back in bug #360.)
Updated by Dave Täht about 1 year ago
with the current patch set, I never get any traps... until it crashes completely under heavy load. The only thing I can think of is that it's actually a bug or problem in the flash or jffs2 driver(s)
Updated by Robert Bradley about 1 year ago
It sounds like there's a few bugs affecting flash and JFFS2 at the moment. For example, this locking order bug in jffs2 was recently found and patched:
https://lkml.org/lkml/2012/3/29/528
That was merged into the mainline kernel today, along with a fix for a kernel oops in the dataflash driver:
Perhaps that would be worth testing?