lkml.org 
[lkml]   [2006]   [Mar]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Kernel panic: Route cache, RCU, possibly FIB trie.

On Tue, 21 Mar 2006, Dipankar Sarma wrote:

> On Mon, Mar 20, 2006 at 10:44:21PM +0100, Jesper Dangaard Brouer wrote:
>>
>> Kernel panic report.
>>
>> Have experienced some kernel panic's on a production Linux box acting
>> as a router for a large number of customers.
>>
>> I have tried to track down the problem, and I think I have narrowed it
>> a bit down. My theory is that it is related to the route cache
>> (ip_dst_cache) or FIB, which cannot dealloacate route cache slab
>> elements (maybe RCU related). (I have seen my route cache increase to
>> around 520k entries using rtstat, before dying).
>>
>> I'm using the FIB trie system/algorithm (CONFIG_IP_FIB_TRIE). Think
>> that the error might be cause by the "fib_trie" code. See the syslog,
>> output below.
>>
>> Below are some kernel panic outputs from the console and some
>> interesting errors found in syslog.
>>
>> Kernel panic#1
>> --------------
>> EIP is at _stext+0x3feffd68/0x49
>> c03f7380
>> Call Trace:
>> [<c0103cc7>] show_stack+0x80/0x96
>> [<c0103e60>] show_registers+0x161/0x1c5
>> [<c0104057>] die+0x107/0x186
>> [<c0116c5f>] do_page_fault+0x2c6/0x57d
>> [<c0103997>] error_code+0x4f/0x54
>> [<c012fe7b>] __rcu_process_callbacks+0xaa/0xd3
>> [<c012feff>] rcu_process_callbacks+0x5b/0x65
>> [<c0124578>] tasklet_action+0x77/0xc9
>> [<c01241f1>] __do_softirq+0xc1/0xd6
>> [<c0124251>] do_softirq+0x4b/0x4d
>> [<c012433b>] irq_exit+0x47/0x49
>> [<c010533b>] do_IRQ+0x2b/0x3b
>> [<c010383e>] common_interrupt+0x1a/0x20
>> Code: Bad EIP value.
>> <0>Kernel panic - not syncing: Fatal exception in interrupt
>
> Bad eip in processing rcu callback often indicates that the object
> that embeds the rcu_head has already been freed. Can you enable
> slab debugging and see if this can be detected there in a different
> path ?

I have enabled slab debugging (CONFIG_DEBUG_SLAB=y) on my kernel with the
KDB patch applied (and all of the other debugging options I could find).

If you look (in my original mail and below) on the output from "Kernel
panic#3", there is a different code path which also fail. Related to
route cache flushing.


Do you have an explaination of the syslog listing, showing yet another
code path sleeping/failing??. In the fib_trie code. (I have recorded 12
of these).

Hilsen
Jesper Brouer

--
-------------------------------------------------------------------
Cand. scient datalog
Dept. of Computer Science, University of Copenhagen
-------------------------------------------------------------------


Syslog#1 (indicating a problem with the fib trie)
--------
Mar 20 18:00:04 hostname kernel: Debug: sleeping function called from invalid context at mm/slab.c:2472
Mar 20 18:00:04 hostname kernel: in_atomic():1, irqs_disabled():0
Mar 20 18:00:04 hostname kernel: [<c0103d9f>] dump_stack+0x1e/0x22
Mar 20 18:00:04 hostname kernel: [<c011cbe1>] __might_sleep+0xa6/0xae
Mar 20 18:00:04 hostname kernel: [<c014f3e9>] __kmalloc+0xd9/0xf3
Mar 20 18:00:04 hostname kernel: [<c014f5a4>] kzalloc+0x23/0x50
Mar 20 18:00:04 hostname kernel: [<c030ecd1>] tnode_alloc+0x3c/0x82
Mar 20 18:00:04 hostname kernel: [<c030edf6>] tnode_new+0x26/0x91
Mar 20 18:00:04 hostname kernel: [<c030f757>] halve+0x43/0x31d
Mar 20 18:00:04 hostname kernel: [<c030f090>] resize+0x118/0x27e
Mar 20 18:00:04 hostname kernel: [<c030fca1>] trie_rebalance+0x90/0x115
Mar 20 18:00:04 hostname kernel: [<c0310a58>] trie_leaf_remove+0x123/0x1c7
Mar 20 18:00:04 hostname kernel: [<c031101d>] fn_trie_flush+0x5c/0x84
Mar 20 18:00:04 hostname kernel: [<c030bb06>] fib_flush+0x2a/0x4c
Mar 20 18:00:04 hostname kernel: [<c030c905>] fib_disable_ip+0x47/0x49
Mar 20 18:00:04 hostname kernel: [<c030c9fb>] fib_netdev_event+0x87/0x94
Mar 20 18:00:04 hostname kernel: [<c012d15a>] notifier_call_chain+0x27/0x3f
Mar 20 18:00:04 hostname kernel: [<c02c02f2>] dev_close+0x6f/0x8d
Mar 20 18:00:04 hostname kernel: [<c02c19e0>] dev_change_flags+0x56/0x127
Mar 20 18:00:04 hostname kernel: [<f8b60806>] vlan_device_event+0x16b/0x16d [8021q]
Mar 20 18:00:04 hostname kernel: [<c012d15a>] notifier_call_chain+0x27/0x3f
Mar 20 18:00:04 hostname kernel: [<c02c02f2>] dev_close+0x6f/0x8d
Mar 20 18:00:04 hostname kernel: [<c02c19e0>] dev_change_flags+0x56/0x127
Mar 20 18:00:04 hostname kernel: [<c0304b95>] devinet_ioctl+0x232/0x5a2
Mar 20 18:00:04 hostname kernel: [<c0306e75>] inet_ioctl+0x5a/0x97
Mar 20 18:00:04 hostname kernel: [<c02b6c4f>] sock_ioctl+0xb0/0x236
Mar 20 18:00:04 hostname kernel: [<c0178473>] do_ioctl+0x7b/0x8b
Mar 20 18:00:04 hostname kernel: [<c017861d>] vfs_ioctl+0x62/0x1c1
Mar 20 18:00:04 hostname kernel: [<c01787bd>] sys_ioctl+0x41/0x64
Mar 20 18:00:04 hostname kernel: [<c0102ef9>] syscall_call+0x7/0xb



Kernel panic#3 with KDB
--------------
[<c01171eb>] do_page_fault+0x300/0x59c
[<c0103a1f>] error_code+0x4f/0x54
[<c02d8d07>] rt_cache_flush+0xf5/0xfe
[<c02d8d2b>] rt_secret_rebuild+0x1b/0x38
[<c0129344>] run_timer_softirq+0xd4/0x1c2
[<c0125115>] __do_softirq+0xc1/0xd6
[<c0125175>] do_softirq+0x4b/0x4d
[<c012525f>] irq_exit+0x47/0x49
[<c01139cc>] smp_apic_timer_interrupt+0x6f/0xee
[<c0103954>] apic_timer_interrupt+0x1c/0x24
[<c0100e10>] cpu_idle+0x89/0x9f
[<c01002dd>] _stext+0x45/0x49
[<c041aa3c>] start_kernel+0x1b5/0x1f6
[<c0100210>] 0xc0100210
Code: 01 f0 e8 40 de 04 00 a1 2c 3b 49 c0 8b 1c b8 85 db 74 07 c7 04 b8 00 00 00
00 03 35 28 3b 49 c0 89 f0 e8 d8 e1 04 00 85 db 74 37 <0f> b7 83 92 00 00 00 8b
33 8b 04 85 50 58 49 c0 85 c0 74 0d 8b

Entering kdb (current=0xc0392ba0, pid 0) on processor 0 Oops: Oops
due to oops @ 0xc02d8bc6
eax = 0xd8b62f84 ebx = 0xd9d75f04 ecx = 0x00000100 edx = 0xc02dda2e
esi = 0xd9d75f04 edi = 0x000077fc esp = 0xc0419e94 eip = 0xc02d8bc6
ebp = 0xc0419ea8 xss = 0xc02b0068 xcs = 0x00000060 eflags = 0x00010282
xds = 0xc02d007b xes = 0xc010007b origeax = 0xffffffff &regs = 0xc0419e60
[0]kdb>


[0]kdb> bt

Stack traceback for pid 0
0xc0392ba0 0 0 1 0 R 0xc0392d80 *swapper
EBP EIP Function (args)
0xc0419ea8 0xc02d8bc6 rt_run_flush+0x6b (0x0, 0xc0394ce4, 0x0, 0xc42ee, 0x102)
0xc0419ec8 0xc02d8d07 rt_cache_flush+0xf5 (0x0, 0xc0326d81, 0x0)
0xc0419edc 0xc02d8d2b rt_secret_rebuild+0x1b (0x0, 0xc1c0c520, 0x0, 0xc42ee, 0xc0419f00)
0xc0419f18 0xc0129344 run_timer_softirq+0xd4 (0xc040f308, 0x0, 0xa, 0xc1c0b2e0,0x46)
0xc0419f38 0xc0125115 __do_softirq+0xc1 (0xc1c0b2e0, 0x0)
0xc0419f48 0xc0125175 do_softirq+0x4b (0xc044d98c)
0xc0419f54 0xc012525f irq_exit+0x47 (0x0, 0xc0419f7c, 0xc044d988, 0xc0100d19, 0xc0418000)
0xc0419f74 0xc01139cc smp_apic_timer_interrupt+0x6f (0xc0100d19, 0x1, 0xc0418000, 0xc0418000, 0xc044d380)
0xc0419fb0 0xc0103954 apic_timer_interrupt+0x1c (0x602080b, 0x9f300, 0xc040c800)
0xc0100e10 cpu_idle+0x89
0xc0419fd8 0xc01002dd _stext+0x45


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2006-03-21 11:35    [W:0.582 / U:0.180 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site