Messages in this thread |  | | From | "Jonathan Earle" <> | Subject | Kernel oops in mm/slab.c [ kmem_cache_grow() ] with test4-8 | Date | Fri, 15 Sep 2000 14:02:17 -0500 |
| |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> <META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2652.35"> <TITLE>Kernel oops in mm/slab.c [ kmem_cache_grow() ] with test4-8</TITLE> </HEAD> <BODY>
<P><FONT SIZE=2>Hi, </FONT> </P>
<P><FONT SIZE=2>I've been having kernel oopses with the 2.4.0-test series and am including ksymoops processed output from both test4 and test5 kernels. The same oops happens in later kernels too (Tested with test6, test7 and test8).</FONT></P>
<P><FONT SIZE=2>The scenario is this:</FONT> </P>
<P><FONT SIZE=2>I have an incoming UDP stream at 1mbit. The router marks packets in this stream, according to port ranges, with 3 (or any # of) marks (via iptables v1.1.1). iproute2 builds new routing tables based on these marks, and mplsadm, with the tc patch, is called to build LSPs using these routing tables. Finally, the 3 egress LSPs are rate limited using tc (employing cbq classes) to a value less than the ingress rate (ie: I limited each LSP to 200kbit, for an aggregate egress output rate of 600kbit). When I start the traffic flowing from our generator, the box panics and freezes quite solidly. Policing via filters also crashes the box. If I move the egress rate limiting function to another box, it works okay.</FONT></P>
<P><FONT SIZE=2>I've also noted that the crash only occurs if I throttle the traffic flow to an egress rate which is less than the ingress rate (ie: ingress flow at 1mbit and egress flow at 1mbit works fine. If the egress rate is reduced, boom!)</FONT></P>
<P><FONT SIZE=2>I copied down the oopses and ran 'ksymoops < oops.txt > oops_proc.txt' and pasted them here. The first is from kernel 2.4.0-test4 and the second from 2.4.0-test5.</FONT></P>
<P><FONT SIZE=2>NEW: Here's the funny part. In mm/slab.c, the function kmem_cache_grow() contains a check as follows:</FONT> </P>
<P><FONT SIZE=2> /*</FONT> <BR><FONT SIZE=2> * The test for missing atomic flag is performed here, rather than</FONT> <BR><FONT SIZE=2> * the more obvious place, simply to reduce the critical path length</FONT> <BR><FONT SIZE=2> * in kmem_cache_alloc(). If a caller is seriously mis-behaving they</FONT> <BR><FONT SIZE=2> * will eventually be caught here (where it matters).</FONT> <BR><FONT SIZE=2> */</FONT> <BR><FONT SIZE=2> /* Commented out Sep 15 since it was crashing my router. */</FONT> <BR><FONT SIZE=2> /* if (in_interrupt() && (flags & SLAB_LEVEL_MASK) != SLAB_ATOMIC)</FONT> <BR><FONT SIZE=2> BUG(); */</FONT> </P>
<P><FONT SIZE=2>This is the check that fails and causes the oops. Not understanding what is actually being checked, and not knowing the repercussions of tampering with it, I commented out the check, recompiled and reran the test. I understand that this is not really a fix (it's more akin to just turning my head and pretending that the problem doesn't exist, but... it seems to work.) The result: Great joy and much celebration! I'm throwing 7.2mbps at the box, limiting the rate to 900kbit aggregate throughput and it's working! The numbers I'm getting also seem to jive with anticipated results.</FONT></P>
<P><FONT SIZE=2>Cheers! </FONT> <BR><FONT SIZE=2>Jon </FONT> </P>
<P><FONT SIZE=2>ksymoops 0.7c on i686 2.4.0-test4. Options used </FONT> <BR><FONT SIZE=2> -V (default) </FONT> <BR><FONT SIZE=2> -k /proc/ksyms (default) </FONT> <BR><FONT SIZE=2> -l /proc/modules (default) </FONT> <BR><FONT SIZE=2> -o /lib/modules/2.4.0-test4/ (default) </FONT> <BR><FONT SIZE=2> -m /usr/src/linux/System.map (default) </FONT> </P>
<P><FONT SIZE=2>Warning: You did not tell me where to find symbol information. I will </FONT> <BR><FONT SIZE=2>assume that the log matches the kernel and modules that are running </FONT> <BR><FONT SIZE=2>right now and I'll use the default options above for symbol resolution. </FONT> <BR><FONT SIZE=2>If the current kernel and/or modules do not match the log, you can get </FONT> <BR><FONT SIZE=2>more accurate output by telling me the kernel version and where to find </FONT> <BR><FONT SIZE=2>map, modules, ksyms etc. ksymoops -h explains the options. </FONT> </P>
<P><FONT SIZE=2>invalid operand: 0000 </FONT> <BR><FONT SIZE=2>CPU: 0 </FONT> <BR><FONT SIZE=2>EIP: 0010:[<c01277fd>] </FONT> <BR><FONT SIZE=2>Using defaults from ksymoops -t elf32-i386 -a i386 </FONT> <BR><FONT SIZE=2>EFLAGS: 00010286 </FONT> <BR><FONT SIZE=2>eax: 0000001b ebx: c7ffd0c0 ecx: 00000000 edx: 00000082 </FONT> <BR><FONT SIZE=2>esi: 00000246 edi: c7ffd0c0 ebp: 00000007 esp: c024fe70 </FONT> <BR><FONT SIZE=2>ds: 0018 es: 0018 ss: 0018 </FONT> <BR><FONT SIZE=2>Process swapper (pid:0, stackpage=c024f000) </FONT> <BR><FONT SIZE=2>Stack: c01fb794 c01fb834 00000412 c7ffd0c0 00000247 00000007 c024fed4 c7d1602e </FONT> <BR><FONT SIZE=2> c0127aaf c7ffd0c0 00000007 00000000 c7d170e0 c7d1602e c01eb196 00000008 </FONT> <BR><FONT SIZE=2> 00000007 00000000 c7d170e0 c7d1602e c7f8be00 00000000 c01b6aaf c7d170e0 </FONT> <BR><FONT SIZE=2>Call trace: [<c01fb794>][<c01fb834>][<c0127aaf>][<c01eb196>][<c01b6aaf>][<c01b6c6f>][<c01b6a84>] </FONT> <BR><FONT SIZE=2> [<c019b1c4>][<c01b6936>][<c01b6a84>][<c019efe3>][<c011b17f>][<c010b8ee>][<c01087e0>][<c01087e0>] </FONT> <BR><FONT SIZE=2> [<c010a518>][<c01087e0>][<c01087e0>][<c0100018>][<c0108803>][<c0108864>][<c0105000>][<c0100192>] </FONT> <BR><FONT SIZE=2>Code: 0f 0b 83 c4 0c c7 44 24 10 01 00 00 00 89 ee 83 e6 07 b8 03 </FONT> </P>
<P><FONT SIZE=2>>>EIP; c01277fd <kmem_cache_grow+69/254> <===== </FONT> <BR><FONT SIZE=2>Trace; c01fb794 <tvecs+1500/14d4c> </FONT> <BR><FONT SIZE=2>Trace; c01fb834 <tvecs+15a0/14d4c> </FONT> <BR><FONT SIZE=2>Trace; c0127aaf <kmalloc+73/ac> </FONT> <BR><FONT SIZE=2>Trace; c01eb196 <mpls_output+12/26c> </FONT> <BR><FONT SIZE=2>Trace; c01b6aaf <ip_rcv_finish+2b/21c> </FONT> <BR><FONT SIZE=2>Trace; c01b6c6f <ip_rcv_finish+1eb/21c> </FONT> <BR><FONT SIZE=2>Trace; c01b6a84 <ip_rcv_finish+0/21c> </FONT> <BR><FONT SIZE=2>Trace; c019b1c4 <nf_hook_slow+7c/b4> </FONT> <BR><FONT SIZE=2>Trace; c01b6936 <ip_rcv+356/38c> </FONT> <BR><FONT SIZE=2>Trace; c01b6a84 <ip_rcv_finish+0/21c> </FONT> <BR><FONT SIZE=2>Trace; c019efe3 <net_rx_action+123/1e8> </FONT> <BR><FONT SIZE=2>Trace; c011b17f <do_softirq+4f/70> </FONT> <BR><FONT SIZE=2>Trace; c010b8ee <do_IRQ+a6/b8> </FONT> <BR><FONT SIZE=2>Trace; c01087e0 <default_idle+0/28> </FONT> <BR><FONT SIZE=2>Trace; c01087e0 <default_idle+0/28> </FONT> <BR><FONT SIZE=2>Trace; c010a518 <ret_from_intr+0/20> </FONT> <BR><FONT SIZE=2>Trace; c01087e0 <default_idle+0/28> </FONT> <BR><FONT SIZE=2>Trace; c01087e0 <default_idle+0/28> </FONT> <BR><FONT SIZE=2>Trace; c0100018 <startup_32+18/13a> </FONT> <BR><FONT SIZE=2>Trace; c0108803 <default_idle+23/28> </FONT> <BR><FONT SIZE=2>Trace; c0108864 <cpu_idle+3c/50> </FONT> <BR><FONT SIZE=2>Trace; c0105000 <empty_bad_page+0/1000> </FONT> <BR><FONT SIZE=2>Trace; c0100192 <L6+0/2> </FONT> <BR><FONT SIZE=2>Code; c01277fd <kmem_cache_grow+69/254> </FONT> <BR><FONT SIZE=2>00000000 <_EIP>: </FONT> <BR><FONT SIZE=2>Code; c01277fd <kmem_cache_grow+69/254> <===== </FONT> <BR><FONT SIZE=2> 0: 0f 0b ud2a <===== </FONT> <BR><FONT SIZE=2>Code; c01277ff <kmem_cache_grow+6b/254> </FONT> <BR><FONT SIZE=2> 2: 83 c4 0c add $0xc,%esp </FONT> <BR><FONT SIZE=2>Code; c0127802 <kmem_cache_grow+6e/254> </FONT> <BR><FONT SIZE=2> 5: c7 44 24 10 01 00 00 movl $0x1,0x10(%esp,1) </FONT> <BR><FONT SIZE=2>Code; c0127809 <kmem_cache_grow+75/254> </FONT> <BR><FONT SIZE=2> c: 00 </FONT> <BR><FONT SIZE=2>Code; c012780a <kmem_cache_grow+76/254> </FONT> <BR><FONT SIZE=2> d: 89 ee mov %ebp,%esi </FONT> <BR><FONT SIZE=2>Code; c012780c <kmem_cache_grow+78/254> </FONT> <BR><FONT SIZE=2> f: 83 e6 07 and $0x7,%esi </FONT> <BR><FONT SIZE=2>Code; c012780f <kmem_cache_grow+7b/254> </FONT> <BR><FONT SIZE=2> 12: b8 03 00 00 00 mov $0x3,%eax </FONT> </P>
<P><FONT SIZE=2>Aiee, killing interrupt handler </FONT> <BR><FONT SIZE=2>Kernel panic: Attempted to kill the idle task! </FONT> </P>
<P><FONT SIZE=2>1 warning issued. Results may not be reliable. </FONT> </P> <BR>
<P><FONT SIZE=2>*********************</FONT> <BR><FONT SIZE=2>** Kernel 2.4.0-test5 </FONT> </P>
<P><FONT SIZE=2>ksymoops 0.7c on i686 2.4.0-test5. Options used </FONT> <BR><FONT SIZE=2> -V (default) </FONT> <BR><FONT SIZE=2> -k /proc/ksyms (default) </FONT> <BR><FONT SIZE=2> -l /proc/modules (default) </FONT> <BR><FONT SIZE=2> -o /lib/modules/2.4.0-test5/ (default) </FONT> <BR><FONT SIZE=2> -m /usr/src/linux/System.map (default) </FONT> </P>
<P><FONT SIZE=2>Warning: You did not tell me where to find symbol information. I will </FONT> <BR><FONT SIZE=2>assume that the log matches the kernel and modules that are running </FONT> <BR><FONT SIZE=2>right now and I'll use the default options above for symbol resolution. </FONT> <BR><FONT SIZE=2>If the current kernel and/or modules do not match the log, you can get </FONT> <BR><FONT SIZE=2>more accurate output by telling me the kernel version and where to find </FONT> <BR><FONT SIZE=2>map, modules, ksyms etc. ksymoops -h explains the options. </FONT> </P>
<P><FONT SIZE=2>invalid operand: 0000 </FONT> <BR><FONT SIZE=2>CPU: 0 </FONT> <BR><FONT SIZE=2>EIP: 0010:[<c01270ed>] </FONT> <BR><FONT SIZE=2>Using defaults from ksymoops -t elf32-i386 -a i386 </FONT> <BR><FONT SIZE=2>EFLAGS: 00010282 </FONT> <BR><FONT SIZE=2>eax: 0000001b ebx: c7ffd0c0 ecx: c02379cc edx: 00000000 </FONT> <BR><FONT SIZE=2>esi: 00000007 edi: c7ffd0c0 ebp: c024dee8 esp: c024de80 </FONT> <BR><FONT SIZE=2>ds: 0018 es: 0018 ss: 0018 </FONT> <BR><FONT SIZE=2>Process swapper (pid:0, stackpage=c024d000) </FONT> <BR><FONT SIZE=2>Stack: c01fa0fa c01fa19a 00000412 c7ffd0c0 00000246 00000007 c024dee8 c6702842 </FONT> <BR><FONT SIZE=2> c670282e c012736f c7ffd0c0 00000007 00000000 c7e70140 c670282e c01e96a6 </FONT> <BR><FONT SIZE=2> 00000008 00000008 00000000 c7e70140 c670282e c7f8be00 00000000 c01b4f67 </FONT> <BR><FONT SIZE=2>Call trace: [<c01fa0fa>][<c01fa198>][<c012736f>][<c01e96a6>][<c01b4f67>][<c01b511f>][<c0199a8c>] </FONT> <BR><FONT SIZE=2> [<c01b4de9>][<c01b4f3c>][<c019d833>][<c011ad5f>][<c010b84e>][<c01087e0>][<c01087e0>][<c010a478>] </FONT> <BR><FONT SIZE=2> [<c01087e0>][<c01087e0>][<c0100018>][<c0108803>][<c0108864>][<c0105000>][<c0100192>] </FONT> <BR><FONT SIZE=2>Code: 0f 0b 83 c4 0c c7 44 24 10 01 00 00 00 89 f5 83 e5 07 68 03 </FONT> </P>
<P><FONT SIZE=2>>>EIP; c01270ed <kmem_cache_grow+69/228> <===== </FONT> <BR><FONT SIZE=2>Trace; c01fa0fa <tvecs+15fe/156c4> </FONT> <BR><FONT SIZE=2>Trace; c01fa198 <tvecs+169c/156c4> </FONT> <BR><FONT SIZE=2>Trace; c012736f <kmalloc+6f/a8> </FONT> <BR><FONT SIZE=2>Trace; c01e96a6 <mpls_output+12/26c> </FONT> <BR><FONT SIZE=2>Trace; c01b4f67 <ip_rcv_finish+2b/214> </FONT> <BR><FONT SIZE=2>Trace; c01b511f <ip_rcv_finish+1e3/214> </FONT> <BR><FONT SIZE=2>Trace; c0199a8c <nf_hook_slow+c4/f8> </FONT> <BR><FONT SIZE=2>Trace; c01b4de9 <ip_rcv+ed/11c> </FONT> <BR><FONT SIZE=2>Trace; c01b4f3c <ip_rcv_finish+0/214> </FONT> <BR><FONT SIZE=2>Trace; c019d833 <net_rx_action+123/1e8> </FONT> <BR><FONT SIZE=2>Trace; c011ad5f <do_softirq+4f/70> </FONT> <BR><FONT SIZE=2>Trace; c010b84e <do_IRQ+a6/b8> </FONT> <BR><FONT SIZE=2>Trace; c01087e0 <default_idle+0/28> </FONT> <BR><FONT SIZE=2>Trace; c01087e0 <default_idle+0/28> </FONT> <BR><FONT SIZE=2>Trace; c010a478 <ret_from_intr+0/20> </FONT> <BR><FONT SIZE=2>Trace; c01087e0 <default_idle+0/28> </FONT> <BR><FONT SIZE=2>Trace; c01087e0 <default_idle+0/28> </FONT> <BR><FONT SIZE=2>Trace; c0100018 <startup_32+18/13a> </FONT> <BR><FONT SIZE=2>Trace; c0108803 <default_idle+23/28> </FONT> <BR><FONT SIZE=2>Trace; c0108864 <cpu_idle+3c/50> </FONT> <BR><FONT SIZE=2>Trace; c0105000 <empty_bad_page+0/1000> </FONT> <BR><FONT SIZE=2>Trace; c0100192 <L6+0/2> </FONT> <BR><FONT SIZE=2>Code; c01270ed <kmem_cache_grow+69/228> </FONT> <BR><FONT SIZE=2>00000000 <_EIP>: </FONT> <BR><FONT SIZE=2>Code; c01270ed <kmem_cache_grow+69/228> <===== </FONT> <BR><FONT SIZE=2> 0: 0f 0b ud2a <===== </FONT> <BR><FONT SIZE=2>Code; c01270ef <kmem_cache_grow+6b/228> </FONT> <BR><FONT SIZE=2> 2: 83 c4 0c add $0xc,%esp </FONT> <BR><FONT SIZE=2>Code; c01270f2 <kmem_cache_grow+6e/228> </FONT> <BR><FONT SIZE=2> 5: c7 44 24 10 01 00 00 movl $0x1,0x10(%esp,1) </FONT> <BR><FONT SIZE=2>Code; c01270f9 <kmem_cache_grow+75/228> </FONT> <BR><FONT SIZE=2> c: 00 </FONT> <BR><FONT SIZE=2>Code; c01270fa <kmem_cache_grow+76/228> </FONT> <BR><FONT SIZE=2> d: 89 f5 mov %esi,%ebp </FONT> <BR><FONT SIZE=2>Code; c01270fc <kmem_cache_grow+78/228> </FONT> <BR><FONT SIZE=2> f: 83 e5 07 and $0x7,%ebp </FONT> <BR><FONT SIZE=2>Code; c01270ff <kmem_cache_grow+7b/228> </FONT> <BR><FONT SIZE=2> 12: 68 03 00 00 00 push $0x3 </FONT> </P>
<P><FONT SIZE=2>Aiee, killing interrupt handler </FONT> <BR><FONT SIZE=2>Kernel panic: Attempted to kill the idle task! </FONT> </P>
<P><FONT SIZE=2>1 warning issued. Results may not be reliable. </FONT> </P>
</BODY> </HTML> |  |