Bug #371

route table updates perturbed by alignment errors

Added by Dave Täht on Apr 20, 2012. Updated on Apr 30, 2012.
New Normal Dave Täht

Description

While this is an artifact of bug #360 it begins to explain trouble
I’ve had all year with ipv6, ipv4 (to some extent) and routing.

It also explains the trouble I had with running a lot of filters in
the qfq qdisc, and now that I’m running sfq all the time, was showing up here.

Basically, trapping in the middle of a string of route updates is really bad
on the route table, often forcing a stream to halt entirely, and reset.
That too was hard to see until we were running fast enough to hit the keyhole window.

Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] [<831d5aa8>] ipv6_setsockopt+0x118/0x444 [ipv6]
Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] [<831cfbdc>] ip6_ins_rt+0x2a0/0x454 [ipv6]
Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] 
--
Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] [<831d5acc>] ipv6_setsockopt+0x13c/0x444 [ipv6]
Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] [<831cfbdc>] ip6_ins_rt+0x2a0/0x454 [ipv6]
Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] 
--
Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] [<831d5ad8>] ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] [<831cfbdc>] ip6_ins_rt+0x2a0/0x454 [ipv6]
Apr 20 19:55:06 OpenWrt kern.alert kernel: [ 7158.578125] 
--
Apr 20 19:55:23 OpenWrt kern.alert kernel: [ 7175.203125] [<831d5ab0>] ipv6_setsockopt+0x120/0x444 [ipv6]
Apr 20 19:55:23 OpenWrt kern.alert kernel: [ 7175.203125]

Attachments

  • ipv6_prefix_equal.patch (text/x-patch; 1.3 kiB) Unaligned access patches: ipv6_prefix_equal and addr_bit_set Robert Bradley Apr 21, 2012

History

Updated by Robert Bradley on Apr 21, 2012.
I’m posting this from Windows, so no patches, unfortunately. Looking at online copies of the source though (http://lxr.free-electrons.com/ and others), the likely suspects are addr_bit_set in net/ipv6/ip6_fib.c and __ipv6_prefix_equal in include/net/ipv6.h.
Updated by Robert Bradley on Apr 21, 2012.
Here are the patches for the changes suggested in comment 1.
Updated by Dave Täht on Apr 21, 2012.
my comment is that we were seeing the trap in setsockopt, but were hard to find during dissassembly which did imply they were in an inline. These patches seem somewhat plausible, so I’ll toss them in the next build. Thanks, as always.
Updated by Dave Täht on Apr 22, 2012.
Can’t honestly say if that patch did anything or not. Traps certainly ‘seem’ a little more infrequent.

Some more detail.

Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.367187] $20   : 00000010 82b2486e 82b2489e 80360000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.371093] $24   : 00000000 83246374                  
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.378906] $28   : 802fe000 802ffb70 80360000 801f44d0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.382812] Hi    : ffd87c98
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.386718] Lo    : 6672f4ca
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.390625] epc   : 83255b08 ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.394531]     Tainted: G           O
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.398437] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.402343] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.410156] Cause : 00800010
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.410156] BadVA : 82b248a6
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.414062] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [truncated] [  558.417968] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.562500] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.570312] Stack : 802ffc38 802ffb84 fffffff5 8324fd94 00000000 82b24896 83280000 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.578125]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.585937]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
root@OpenWrt:/sys/kernel/debug/mips# logread
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.527343] $ 8   : 00000000 8016d6f0 00000002 fea54801
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.535156] $12   : 8030cb8c 8030cc28 0000020b 00000000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.539062] $16   : 82238d80 832791e0 80416076 83d44000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.546875] $20   : 00000010 00000001 8041609e 00000001
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.550781] $24   : 00000000 83246374                  
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.554687] $28   : 802fe000 802ffb00 00000000 801f44d0
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.562500] Hi    : 007b3a5f
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.562500] Lo    : d90b3e61
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.566406] epc   : 83255afc ipv6_setsockopt+0x13c/0x444 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.574218]     Tainted: G           O
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.574218] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.582031] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.585937] Cause : 00800010
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.589843] BadVA : 80416082
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.589843] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:54:53 OpenWrt kern.warn kernel: [truncated] [  523.593750] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.738281] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.746093] Stack : 00000010 8009543c 83c190a0 80092430 83d32200 8041606e 83d44000 80416076
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.753906]         82b75900 8325722c 82ba4000 8324fbec 83d44000 82a9acb0 0000000a 82a9a148
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.765625]         802ffc74 802ffc84 00000000 804160ae 00000000 00000000 00000000 00000000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.773437]         00000000 00000000 804160b0 82acb088 80416096 00000087 82b14cc0 83d32200
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.781250]         80360000 83d44000 8327a6b4 30000000 80360000 83257f9c c1b6f568 82ac6400
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.789062]         ...
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.792968] Call Trace:
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.792968] [<83255afc>] ipv6_setsockopt+0x13c/0x444 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.800781] [<8324fbec>] ip6_ins_rt+0x2a0/0x454 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.804687] 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.808593] 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.808593] Code: 00e50018  8cc50008  70620000 <8c82000c> 8cc3000c  70620000  8c820008  00001812  70a22002 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.816406] Cpu 0
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.820312] $ 0   : 00000000 00000000 fea54801 79a6336b
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.824218] $ 4   : 80416076 95d345d9 82238d88 fe800003
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.832031] $ 8   : 00000000 8016d6f0 00000002 fea54801
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.835937] $12   : 8030cb8c 8030cc28 0000020b 00000000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.839843] $16   : 82238d80 832791e0 80416076 83d44000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.847656] $20   : 00000010 00000001 8041609e 00000001
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.851562] $24   : 00000000 83246374                  
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.855468] $28   : 802fe000 802ffb00 00000000 801f44d0
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.863281] Hi    : ffd6784f
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.863281] Lo    : 5f1e89cc
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.867187] epc   : 83255b08 ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.875000]     Tainted: G           O
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.878906] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.882812] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.886718] Cause : 00800010
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.890625] BadVA : 8041607e
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.890625] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:54:53 OpenWrt kern.warn kernel: [truncated] [  523.894531] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.039062] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.046875] Stack : 00000010 8009543c 83c190a0 80092430 83d32200 8041606e 83d44000 80416076
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.058593]         82b75900 8325722c 82ba4000 8324fbec 83d44000 82a9acb0 0000000a 82a9a148
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.066406]         802ffc74 802ffc84 00000000 804160ae 00000000 00000000 00000000 00000000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.074218]         00000000 00000000 804160b0 82acb088 80416096 00000087 82b14cc0 83d32200
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.082031]         80360000 83d44000 8327a6b4 30000000 80360000 83257f9c c1b6f568 82ac6400
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.089843]         ...
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.093750] Call Trace:
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.093750] [<83255b08>] ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.101562] [<8324fbec>] ip6_ins_rt+0x2a0/0x454 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.105468] 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.109375] 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.109375] Code: 8c82000c  8cc3000c  70620000 <8c820008> 00001812  70a22002  03e00008  00831021  8ca20048 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.453125] Cpu 0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.453125] $ 0   : 00000000 00000000 83255ad8 00000300
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.457031] $ 4   : 82b2489e 83d44000 82238d88 00000001
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.464843] $ 8   : 00000020 8016d6f0 00000002 fea9867b
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.468750] $12   : 7fb6b668 7fb6b6b8 00000002 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.472656] $16   : 82238d80 832791e0 82b2489e 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.480468] $20   : 00000010 82b2486e 82b2489e 80360000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.484375] $24   : 00000000 83246374                  
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.492187] $28   : 802fe000 802ffb70 80360000 801f44d0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.496093] Hi    : 00050cb2
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.500000] Lo    : de9a6fa4
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.500000] epc   : 83255ad8 ipv6_setsockopt+0x118/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.507812]     Tainted: G           O
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.511718] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.515625] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.519531] Cause : 00800010
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.523437] BadVA : 82b2489e
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.527343] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [truncated] [  557.531250] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.675781] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.683593] Stack : 802ffc38 802ffb84 fffffff5 8324fd94 00000000 82b24896 83280000 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.691406]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.699218]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.707031]         00000000 83d44000 8327a6b4 30000000 822abd80 00000088 82b24896 83d32200
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.714843]         80360000 83d44000 8327a6b4 30000000 80360000 8325fa84 802ffc90 802ffc90
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.722656]         ...
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.726562] Call Trace:
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.730468] [<83255ad8>] ipv6_setsockopt+0x118/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.734375] 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.738281] 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.738281] Code: 8c59002c  03200008  00000000 <8c870000> 8ca50080  8c820004  8cc30004  00e53826  8cc50000 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.746093] Cpu 0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.750000] $ 0   : 00000000 00000000 83255ad8 00000300
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.753906] $ 4   : 82b2489e 00000003 82238d88 fe800000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.761718] $ 8   : 00000020 8016d6f0 00000002 fea9867b
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.765625] $12   : 7fb6b668 7fb6b6b8 00000002 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.769531] $16   : 82238d80 832791e0 82b2489e 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.777343] $20   : 00000010 82b2486e 82b2489e 80360000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.781250] $24   : 00000000 83246374                  
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.785156] $28   : 802fe000 802ffb70 80360000 801f44d0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.792968] Hi    : 00050cb2
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.792968] Lo    : de9a6fa4
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.796875] epc   : 83255ae0 ipv6_setsockopt+0x120/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.804687]     Tainted: G           O
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.808593] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.812500] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.816406] Cause : 00800010
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.820312] BadVA : 82b248a2
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.820312] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [truncated] [  557.824218] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.968750] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.976562] Stack : 802ffc38 802ffb84 fffffff5 8324fd94 00000000 82b24896 83280000 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.988281]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.996093]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.003906]         00000000 83d44000 8327a6b4 30000000 822abd80 00000088 82b24896 83d32200
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.011718]         80360000 83d44000 8327a6b4 30000000 80360000 8325fa84 802ffc90 802ffc90
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.019531]         ...
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.023437] Call Trace:
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.023437] [<83255ae0>] ipv6_setsockopt+0x120/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.281250]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.289062]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.300781]         00000000 83d44000 8327a6b4 30000000 822abd80 00000088 82b24896 83d32200
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.308593]         80360000 83d44000 8327a6b4 30000000 80360000 8325fa84 802ffc90 802ffc90
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.316406]         ...
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.320312] Call Trace:
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.320312] [<83255afc>] ipv6_setsockopt+0x13c/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.328125] 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.328125] 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.328125] Code: 00e50018  8cc50008  70620000 <8c82000c> 8cc3000c  70620000  8c820008  00001812  70a22002 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.339843] Cpu 0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.339843] $ 0   : 00000000 00000000 fea9867b 79a6336b
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.347656] $ 4   : 82b2489e 95d345d9 82238d88 fe800003
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.351562] $ 8   : 00000020 8016d6f0 00000002 fea9867b
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.355468] $12   : 7fb6b668 7fb6b6b8 00000002 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.363281] $16   : 82238d80 832791e0 82b2489e 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.367187] $20   : 00000010 82b2486e 82b2489e 80360000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.371093] $24   : 00000000 83246374                  
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.378906] $28   : 802fe000 802ffb70 80360000 801f44d0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.382812] Hi    : ffd87c98
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.386718] Lo    : 6672f4ca
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.390625] epc   : 83255b08 ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.394531]     Tainted: G           O
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.398437] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.402343] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.410156] Cause : 00800010
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.410156] BadVA : 82b248a6
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.414062] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [truncated] [  558.417968] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.562500] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.570312] Stack : 802ffc38 802ffb84 fffffff5 8324fd94 00000000 82b24896 83280000 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.578125]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.585937]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
root@OpenWrt:/sys/kernel/debug/mips# logread
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.835937] $12   : 8030cb8c 8030cc28 0000020b 00000000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.839843] $16   : 82238d80 832791e0 80416076 83d44000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.847656] $20   : 00000010 00000001 8041609e 00000001
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.851562] $24   : 00000000 83246374                  
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.855468] $28   : 802fe000 802ffb00 00000000 801f44d0
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.863281] Hi    : ffd6784f
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.863281] Lo    : 5f1e89cc
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.867187] epc   : 83255b08 ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.875000]     Tainted: G           O
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.878906] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.882812] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.886718] Cause : 00800010
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.890625] BadVA : 8041607e
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  523.890625] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:54:53 OpenWrt kern.warn kernel: [truncated] [  523.894531] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.039062] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.046875] Stack : 00000010 8009543c 83c190a0 80092430 83d32200 8041606e 83d44000 80416076
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.058593]         82b75900 8325722c 82ba4000 8324fbec 83d44000 82a9acb0 0000000a 82a9a148
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.066406]         802ffc74 802ffc84 00000000 804160ae 00000000 00000000 00000000 00000000
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.074218]         00000000 00000000 804160b0 82acb088 80416096 00000087 82b14cc0 83d32200
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.082031]         80360000 83d44000 8327a6b4 30000000 80360000 83257f9c c1b6f568 82ac6400
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.089843]         ...
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.093750] Call Trace:
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.093750] [<83255b08>] ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.101562] [<8324fbec>] ip6_ins_rt+0x2a0/0x454 [ipv6]
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.105468] 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.109375] 
Apr 23 01:54:53 OpenWrt kern.warn kernel: [  524.109375] Code: 8c82000c  8cc3000c  70620000 <8c820008> 00001812  70a22002  03e00008  00831021  8ca20048 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.453125] Cpu 0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.453125] $ 0   : 00000000 00000000 83255ad8 00000300
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.457031] $ 4   : 82b2489e 83d44000 82238d88 00000001
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.464843] $ 8   : 00000020 8016d6f0 00000002 fea9867b
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.468750] $12   : 7fb6b668 7fb6b6b8 00000002 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.472656] $16   : 82238d80 832791e0 82b2489e 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.480468] $20   : 00000010 82b2486e 82b2489e 80360000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.484375] $24   : 00000000 83246374                  
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.492187] $28   : 802fe000 802ffb70 80360000 801f44d0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.496093] Hi    : 00050cb2
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.500000] Lo    : de9a6fa4
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.500000] epc   : 83255ad8 ipv6_setsockopt+0x118/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.507812]     Tainted: G           O
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.511718] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.515625] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.519531] Cause : 00800010
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.523437] BadVA : 82b2489e
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.527343] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [truncated] [  557.531250] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.675781] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.683593] Stack : 802ffc38 802ffb84 fffffff5 8324fd94 00000000 82b24896 83280000 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.691406]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.699218]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.707031]         00000000 83d44000 8327a6b4 30000000 822abd80 00000088 82b24896 83d32200
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.714843]         80360000 83d44000 8327a6b4 30000000 80360000 8325fa84 802ffc90 802ffc90
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.722656]         ...
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.726562] Call Trace:
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.730468] [<83255ad8>] ipv6_setsockopt+0x118/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.734375] 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.738281] 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.738281] Code: 8c59002c  03200008  00000000 <8c870000> 8ca50080  8c820004  8cc30004  00e53826  8cc50000 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.746093] Cpu 0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.750000] $ 0   : 00000000 00000000 83255ad8 00000300
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.753906] $ 4   : 82b2489e 00000003 82238d88 fe800000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.761718] $ 8   : 00000020 8016d6f0 00000002 fea9867b
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.765625] $12   : 7fb6b668 7fb6b6b8 00000002 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.769531] $16   : 82238d80 832791e0 82b2489e 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.777343] $20   : 00000010 82b2486e 82b2489e 80360000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.781250] $24   : 00000000 83246374                  
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.785156] $28   : 802fe000 802ffb70 80360000 801f44d0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.792968] Hi    : 00050cb2
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.792968] Lo    : de9a6fa4
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.796875] epc   : 83255ae0 ipv6_setsockopt+0x120/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.804687]     Tainted: G           O
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.808593] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.812500] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.816406] Cause : 00800010
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.820312] BadVA : 82b248a2
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.820312] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [truncated] [  557.824218] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.968750] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.976562] Stack : 802ffc38 802ffb84 fffffff5 8324fd94 00000000 82b24896 83280000 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.988281]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  557.996093]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.003906]         00000000 83d44000 8327a6b4 30000000 822abd80 00000088 82b24896 83d32200
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.011718]         80360000 83d44000 8327a6b4 30000000 80360000 8325fa84 802ffc90 802ffc90
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.019531]         ...
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.023437] Call Trace:
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.023437] [<83255ae0>] ipv6_setsockopt+0x120/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.281250]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.289062]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.300781]         00000000 83d44000 8327a6b4 30000000 822abd80 00000088 82b24896 83d32200
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.308593]         80360000 83d44000 8327a6b4 30000000 80360000 8325fa84 802ffc90 802ffc90
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.316406]         ...
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.320312] Call Trace:
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.320312] [<83255afc>] ipv6_setsockopt+0x13c/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.328125] 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.328125] 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.328125] Code: 00e50018  8cc50008  70620000 <8c82000c> 8cc3000c  70620000  8c820008  00001812  70a22002 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.339843] Cpu 0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.339843] $ 0   : 00000000 00000000 fea9867b 79a6336b
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.347656] $ 4   : 82b2489e 95d345d9 82238d88 fe800003
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.351562] $ 8   : 00000020 8016d6f0 00000002 fea9867b
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.355468] $12   : 7fb6b668 7fb6b6b8 00000002 00000000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.363281] $16   : 82238d80 832791e0 82b2489e 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.367187] $20   : 00000010 82b2486e 82b2489e 80360000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.371093] $24   : 00000000 83246374                  
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.378906] $28   : 802fe000 802ffb70 80360000 801f44d0
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.382812] Hi    : ffd87c98
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.386718] Lo    : 6672f4ca
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.390625] epc   : 83255b08 ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.394531]     Tainted: G           O
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.398437] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.402343] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.410156] Cause : 00800010
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.410156] BadVA : 82b248a6
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.414062] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [truncated] [  558.417968] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.562500] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.570312] Stack : 802ffc38 802ffb84 fffffff5 8324fd94 00000000 82b24896 83280000 83d44000
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.578125]         00000000 83258134 c1b6f568 82ac6400 83d44000 82022f00 c1b6f610 822abd80
Apr 23 01:55:28 OpenWrt kern.warn kernel: [  558.585937]         8108244c 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 23 01:55:34 OpenWrt kern.info kernel: [truncated] ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp xt_policy xt_esp ipt_ah xt_HL xt_hl xt_ecn ipt_ECN xt_CLAS
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.582031] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.589843] Stack : 00000010 8009543c 82b56140 80092430 83d32200 8227a86e 83d44000 8227a876
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.597656]         82b75900 8325722c 82ba4000 8324fbec 83d44000 82a9acb0 0000000a 82a9a148
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.609375]         802ffc74 802ffc84 00000000 8227a8ae 00000000 00000000 00000000 00000000
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.617187]         00000000 00000000 8227a8b0 82acb088 8227a896 00000087 832cc240 83d32200
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.625000]         80360000 83d44000 8327a6b4 30000000 80360000 83257f9c c1b6f568 82ac6400
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.632812]         ...
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.636718] Call Trace:
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.636718] [<83255b08>] ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.644531] [<8324fbec>] ip6_ins_rt+0x2a0/0x454 [ipv6]
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.699218] $28   : 802fe000 802ffb00 00000000 801f44d0
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.707031] Hi    : 00128594
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.707031] Lo    : 05ca59bc
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.710937] epc   : 83255ad8 ipv6_setsockopt+0x118/0x444 [ipv6]
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.718750]     Tainted: G           O
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.722656] ra    : 801f44d0 neigh_lookup+0x78/0x160
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.726562] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.730468] Cause : 00800010
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.734375] BadVA : 8041b876
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.738281] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:55:34 OpenWrt kern.warn kernel: [truncated] [  563.742187] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.886718] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.894531] Stack : 802ffb58 83259ee8 83c79100 802ffb98 83d32200 8041b86e 83d44000 8041b876
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.902343]         82b75900 8325722c 82ba4000 8324fbec 83d44000 82a9acb0 0000000a 82a9a148
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.910156]         802ffc74 802ffc84 00000000 8041b8ae 00000000 00000000 00000000 00000000
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.917968]         00000000 00000000 8041b8b0 82acb088 8041b896 00000087 82b14c00 83d32200
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.925781]         80360000 83d44000 8327a6b4 30000000 80360000 83257f9c c1b6f568 82ac6400
Apr 23 01:55:34 OpenWrt kern.warn kernel: [  563.933593]         ...
root@OpenWrt:/sys/kernel/debug/mips# logread -f
Apr 23 01:56:00 OpenWrt kern.info kernel: 0
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.148437] BadVA : 82b2b07e
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.152343] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:56:00 OpenWrt kern.warn kernel: [truncated] [  589.156250] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.300781] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.308593] Stack : 83d12140 80092c34 00000000 00000001 82b2b076 83280000 82b08540 83d32200
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.316406]         00000000 83258790 c1b6f568 82ac6400 83d44000 82022f00 00000000 83d12140
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.324218]         80092c64 82b2b0b6 82b2b0a6 00000000 82b2b0b6 00000000 82b2b0ae 00000000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.332031]         00000000 83d44000 8327a6b4 30000000 82b08540 00000086 82b2b096 83d32200
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.339843]         80360000 83d44000 8327a6b4 30000000 80360000 8325fa84 000000c3 83180a80
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.347656]         ...
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.351562] Call Trace:
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.355468] [<83255b08>] ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.359375] 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.359375] 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.359375] Code: 8c82000c  8cc3000c  70620000 <8c820008> 00001812  70a22002  03e00008  00831021  8ca20048 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.421875] epc   : 83255ad8 ipv6_setsockopt+0x118/0x444 [ipv6]
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.429687]     Tainted: G           O
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.433593] ra    : 801f4fd0 neigh_create+0x430/0x558
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.437500] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.441406] Cause : 00800010
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.445312] BadVA : 82b2b076
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.445312] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:56:00 OpenWrt kern.warn kernel: [truncated] [  589.449218] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.593750] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.601562] Stack : 1000fc03 ffd6784f 5f1e89cc 82b2b07e 00000010 801f4594 82b2b076 83280000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.613281]         82b08540 83d32200 00000000 00000000 83d44000 30000000 80360000 832587a8
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.621093]         c1b6f568 82b2b076 83d44000 82022f00 00000000 83d12140 80092c64 82b2b0b6
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.628906]         82b2b0a6 00000000 82b2b0b6 00000000 82b2b0ae 00000000 00000000 83d44000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.636718]         8327a6b4 30000000 82b08540 00000086 82b2b096 83d32200 80360000 83d44000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.644531]         ...
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.648437] Call Trace:
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.648437] [<83255ad8>] ipv6_setsockopt+0x118/0x444 [ipv6]
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.656250] 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.656250] 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.656250] Code: 8c59002c  03200008  00000000 <8c870000> 8ca50080  8c820004  8cc30004  00e53826  8cc50000 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.667968] Cpu 0
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.671875] $ 0   : 00000000 00000000 83255ad8 00000011
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.675781] $ 4   : 82b2b076 00000003 82238d88 fe800000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.679687] $ 8   : fe800000 00000000 0211bcff fea54801
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.687500] $12   : 8d000009 8030cc28 0000024c 00000000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.691406] $16   : 828be200 832791e0 82238d80 832792cc
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.695312] $20   : 83d44000 00000000 83d44000 00000010
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.703125] $24   : 00000000 80095334                  
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.707031] $28   : 802fe000 802ffb28 0001215f 801f4fd0
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.710937] Hi    : ffd6784f
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.714843] Lo    : 5f1e89cc
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.718750] epc   : 83255ae0 ipv6_setsockopt+0x120/0x444 [ipv6]
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.722656]     Tainted: G           O
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.726562] ra    : 801f4fd0 neigh_create+0x430/0x558
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.734375] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  589.738281] Cause : 00800010
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.000000] $24   : 00000000 80095334                  
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.003906] $28   : 802fe000 802ffb28 0001215f 801f4fd0
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.007812] Hi    : 007b3a5f
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.011718] Lo    : d90b3e61
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.015625] epc   : 83255afc ipv6_setsockopt+0x13c/0x444 [ipv6]
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.019531]     Tainted: G           O
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.023437] ra    : 801f4fd0 neigh_create+0x430/0x558
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.027343] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.035156] Cause : 00800010
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.035156] BadVA : 82b2b082
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.039062] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:56:00 OpenWrt kern.warn kernel: [truncated] [  590.042968] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.187500] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.195312] Stack : 1000fc03 ffd6784f 5f1e89cc 82b2b07e 00000010 801f4594 82b2b076 83280000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.203125]         82b08540 83d32200 00000000 00000000 83d44000 30000000 80360000 832587a8
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.210937]         c1b6f568 82b2b076 83d44000 82022f00 00000000 83d12140 80092c64 82b2b0b6
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.218750]         82b2b0a6 00000000 82b2b0b6 00000000 82b2b0ae 00000000 00000000 83d44000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.230468]         8327a6b4 30000000 82b08540 00000086 82b2b096 83d32200 80360000 83d44000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.238281]         ...
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.238281] Call Trace:
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.242187] [<83255afc>] ipv6_setsockopt+0x13c/0x444 [ipv6]
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.246093] 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.250000] 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.250000] Code: 00e50018  8cc50008  70620000 <8c82000c> 8cc3000c  70620000  8c820008  00001812  70a22002 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.261718] Cpu 0
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.261718] $ 0   : 00000000 00000000 fea54801 79a6336b
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.269531] $ 4   : 82b2b076 95d345d9 82238d88 fe800003
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.273437] $ 8   : fe800000 00000000 0211bcff fea54801
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.277343] $12   : 8d000009 8030cc28 0000024c 00000000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.285156] $16   : 828be200 832791e0 82238d80 832792cc
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.289062] $20   : 83d44000 00000000 83d44000 00000010
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.292968] $24   : 00000000 80095334                  
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.300781] $28   : 802fe000 802ffb28 0001215f 801f4fd0
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.304687] Hi    : ffd6784f
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.308593] Lo    : 5f1e89cc
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.308593] epc   : 83255b08 ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.316406]     Tainted: G           O
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.320312] ra    : 801f4fd0 neigh_create+0x430/0x558
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.324218] Status: 1000fc03    KERNEL EXL IE 
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.328125] Cause : 00800010
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.332031] BadVA : 82b2b07e
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.335937] PrId  : 00019374 (MIPS 24Kc)
Apr 23 01:56:00 OpenWrt kern.warn kernel: [truncated] [  590.339843] Modules linked in: cls_fw sch_htb sch_red sch_sfq cls_flow cls_u32 sch_qfq gpio_buttons ath79_wdt xt_hashlimit ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.484375] Process swapper (pid: 0, threadinfo=802fe000, task=803020c0, tls=00000000)
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.492187] Stack : 1000fc03 ffd6784f 5f1e89cc 82b2b07e 00000010 801f4594 82b2b076 83280000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.500000]         82b08540 83d32200 00000000 00000000 83d44000 30000000 80360000 832587a8
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.507812]         c1b6f568 82b2b076 83d44000 82022f00 00000000 83d12140 80092c64 82b2b0b6
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.515625]         82b2b0a6 00000000 82b2b0b6 00000000 82b2b0ae 00000000 00000000 83d44000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.523437]         8327a6b4 30000000 82b08540 00000086 82b2b096 83d32200 80360000 83d44000
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.535156]         ...
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.535156] Call Trace:
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.539062] [<83255b08>] ipv6_setsockopt+0x148/0x444 [ipv6]
Apr 23 01:56:00 OpenWrt kern.warn kernel: [  590.542968] 
Apr 23 01:56:38 OpenWrt kern.warn kernel: [  627.453125] Cp
Updated by Robert Bradley on Apr 23, 2012.
I think that last patch did virtually nothing based on the stack traces. I noticed though that the return address is somewhere in neigh_lookup, and so I think the next thing to try is altering __ipv6_neigh_lookup. It looks promising, since if the primary key were taken straight from a packet somewhere, that would definitely cause problems. The only problem is that I can’t see where it would get called, or why ipv6_setsockopt would get involved.
Updated by Stephen Walker on Apr 27, 2012.
Robert Bradley wrote:
> I think that last patch did virtually nothing based on the stack traces. I noticed though that the return address is somewhere in neigh_lookup, and so I think the next thing to try is altering __ipv6_neigh_lookup. It looks promising, since if the primary key were taken straight from a packet somewhere, that would definitely cause problems. The only problem is that I can’t see where it would get called, or why ipv6_setsockopt would get involved.

ipv6_neigh_lookup.patch worksforme

rootOnyx:~# cat /sys/kernel/debug/mips/unaligned_action
2
root@Onyx:~# cat /sys/kernel/debug/mips/unaligned_instructions
0@

(My local build does have a change to pack include/linux/icmp.h’s icmphdr to not trap upon pinging the router)

Updated by Dave Täht on Apr 27, 2012.
thx guys. you made my evening. (I’m still logging out tho)

stephen, I thought we’d covered that header in the current patchset already?

how does this puppy ‘feel’ now? I noticed as soon as we got below 10k traps/sec that things like ssh to the router became much smoother under load….

Updated by Stephen Walker on Apr 28, 2012.
Dave Täht wrote:
> stephen, I thought we’d covered that header in the current patchset already?

http://patchwork.openwrt.org/patch/2117/ only touches ICMPv6 & the existing 902-unaligned_access_hacks.patch didn’t touch ICMP at all.

how does this puppy ‘feel’ now? I noticed as soon as we got below 10k traps/sec that things like ssh to the router became much smoother under load….

I can’t say that I can feel any difference but I am not frequently logged in.

Updated by Dave Täht on Apr 30, 2012.
I have the combination of these patches up and running. I’ve seen as few as zero traps and as many as 1/sec… which is still quite nice. I only saw one router doing that, so I think it’s something with multicast, but didn’t look into it further.

However, see #379 . Behind every solution there’s another problem. At one level, I’m not too worried about it, it’s hard to drive the wireless radios hard enough to incur the issues, and similarly ethernet is on a switch for local devices…

This is a static export of the original bufferbloat.net issue database. As such, no further commenting is possible; the information is solely here for archival purposes.

Cerowrt Project

News items
List of Wiki pages
List of bugs
RSS feed

Recent Updates

Oct 20, 2023 Wiki page
What Can I Do About Bufferbloat?
Dec 3, 2022 Wiki page
Codel Wiki
Jun 11, 2022 Wiki page
More about Bufferbloat
Jun 11, 2022 Wiki page
Tests for Bufferbloat
Dec 7, 2021 Wiki page
Getting SQM Running Right

Find us elsewhere

Bufferbloat Mailing Lists
#bufferbloat on Twitter
Google+ group
Archived Bufferbloat pages from the Wayback Machine

Sponsors

Comcast Research Innovation Fund
Nlnet Foundation
Shuttleworth Foundation
GoFundMe

Bufferbloat Related Projects

OpenWrt Project
Congestion Control Blog
Flent Network Test Suite
Sqm-Scripts
The Cake shaper
AQMs in BSD
IETF AQM WG
CeroWrt (where it all started)

Network Performance Related Resources


Jim Gettys' Blog - The chairman of the Fjord
Toke's Blog - Karlstad University's work on bloat
Voip Users Conference - Weekly Videoconference mostly about voip
Candelatech - A wifi testing company that "gets it".