Messages in this thread | | | From | "Kai Harrekilde-Petersen" <> | Subject | [2.2.14] Double Oops in fs layer | Date | Thu, 16 Mar 2000 18:32:28 +0100 |
| |
I am trying to determine the cause of a double Oops caused during stress testing on new server.
I suspect flaky hardware, but I am not very versed in in diagnosing hardware errors from Oops'es.
The hardware: Athlon-700, 768MB RAM, one 36Gig SCSI-U2W disk on a Tekram DC-390U2W (sym53c895) controller. LVD/SE terminator in place (can't tell if its active or passive).
Kernel: Stock 2.2.14, compiled to the system in case.
The stress loops: . In one VT: "while [ $? == 0 ]; do make clean && make -j8 bzImage; done" (on the 2.2.14 kernel tree). . In another VT: "while [ $? == 0 ]; do bonnie -d . -s 1024 -m lvd; done" (top was running in a third VT).
I had stressed the system the night before with just a 'make -j4 bzImage' loop, without problems.
Funnily enough, the Oopses hit 'top' and 'kflushd', not bonnie or the make loop. $DEITY only knows why (and maybe Linus ... :-)
The Oopses passed through ksymoops (see below for source code pinpoints):
ksymoops 2.3.3 on i686 2.2.14. Options used -V (specified) -k /proc/ksyms (default) -L (specified) -O (specified) -m /boot/System.map-2.2.14 (specified)
Unable to handle kernel paging request at virtual address 43429ca8 current->tss.cr3 = 1f90d000, %cr3 = 1f90d000 *pde = 00000000 Oops: 0000 CPU: 0 EIP: 0010:[<c01309e2>] Using defaults from ksymoops -t elf32-i386 -a i386 EFLAGS: 00010a83 eax: 43429c40 ebx: 0004000a ecx: c0219c90 edx: 43429c40 esi: effd1c00 edi: effd1c00 ebp: d31ded80 esp: d24cbea8 ds: 0018 es: 0018 ss: 0018 Process top (pid: 1840, process nr: 62, stackpage=d24cb000) Stack: c0130ce0 effd1c00 0004000a c0219c90 0004000a c01f34e4 effd1c00 00040002 c01403fa effd1c00 0004000a c01f34e4 00040000 c01cc535 c0140a73 effd1c00 0004000a c01f34e4 fffffff4 d31ded80 d6a88dd0 d31dee80 d6a88dd0 ffffffea Call Trace: [<c0130ce0>] [<c01403fa>] [<c01cc535>] [<c0140a73>] [<c012afe7>] [<c012b1d0>] [<c012b339>] [<c0123cee>] [<c0123f58>] [<c0109094>] Code: 39 70 68 75 0d 39 58 18 75 08 ff 40 1c 5b 5e c3 89 f6 8b 12
>>EIP; c01309e2 <find_inode+16/34> <===== Trace; c0130ce0 <iget+30/6c> Trace; c01403fa <proc_get_inode+3e/f0> Trace; c01cc535 <cprt+1bb3/7ac8> Trace; c0140a73 <proc_lookup+93/c4> Trace; c012afe7 <real_lookup+4f/a4> Trace; c012b1d0 <lookup_dentry+10c/1ac> Trace; c012b339 <open_namei+6d/2ec> Trace; c0123cee <filp_open+46/f8> Trace; c0123f58 <sys_open+38/94> Trace; c0109094 <system_call+34/38> Code; c01309e2 <find_inode+16/34> 00000000 <_EIP>: Code; c01309e2 <find_inode+16/34> <===== 0: 39 70 68 cmpl %esi,0x68(%eax) <===== Code; c01309e5 <find_inode+19/34> 3: 75 0d jne 12 <_EIP+0x12> c01309f4 <find_inode+28/34> Code; c01309e7 <find_inode+1b/34> 5: 39 58 18 cmpl %ebx,0x18(%eax) Code; c01309ea <find_inode+1e/34> 8: 75 08 jne 12 <_EIP+0x12> c01309f4 <find_inode+28/34> Code; c01309ec <find_inode+20/34> a: ff 40 1c incl 0x1c(%eax) Code; c01309ef <find_inode+23/34> d: 5b popl %ebx Code; c01309f0 <find_inode+24/34> e: 5e popl %esi Code; c01309f1 <find_inode+25/34> f: c3 ret Code; c01309f2 <find_inode+26/34> 10: 89 f6 movl %esi,%esi Code; c01309f4 <find_inode+28/34> 12: 8b 12 movl (%edx),%edx
Unable to handle kernel paging request at virtual address 000064b4 current->tss.cr3 = 00101000, %cr3 = 00101000 *pde = 00000000 Oops: 0002 CPU: 0 EIP: 0010:[<c012566f>] EFLAGS: 00010202 eax: 00006480 ebx: 00000000 ecx: eafbfc00 edx: efc6b8f8 esi: eafbfc00 edi: 00000754 ebp: 00000000 esp: effe9fc4 ds: 0018 es: 0018 ss: 0018 Process kflushd (pid: 2, process nr: 2, stackpage=effe900) Stack: eafbfc00 00000001 eafbfba0 c0126e56 eafbfc00 00000f00 efffbfcc c0106000 000000a0 00000003 eafbfc00 c0107b2b 00000000 00000f00 c0203fd8 Call Trace: [<c0126e56>] [<c0106000>] [<c0107b2b>] Code: 89 50 34 c7 01 00 00 00 00 89 02 c7 41 34 00 00 00 00 ff 0d
>>EIP; c012566f <remove_from_queues+a3/130> <===== Trace; c0126e56 <bdflush+d2/220> Trace; c0106000 <get_options+0/74> Trace; c0107b2b <kernel_thread+23/30> Code; c012566f <remove_from_queues+a3/130> 00000000 <_EIP>: Code; c012566f <remove_from_queues+a3/130> <===== 0: 89 50 34 movl %edx,0x34(%eax) <===== Code; c0125672 <remove_from_queues+a6/130> 3: c7 01 00 00 00 movl $0x0,(%ecx) Code; c0125677 <remove_from_queues+ab/130> 8: 00 Code; c0125678 <remove_from_queues+ac/130> 9: 89 02 movl %eax,(%edx) Code; c012567a <remove_from_queues+ae/130> b: c7 41 34 00 00 movl $0x0,0x34(%ecx) Code; c012567f <remove_from_queues+b3/130> 10: 00 00 Code; c0125681 <remove_from_queues+b5/130> 12: ff 0d 00 00 00 decl 0x0 Code; c0125686 <remove_from_queues+ba/130> 17: 00
The first Oops arises in fs/inode.c: /* * Called with the inode lock held. */ static struct inode * find_inode(struct super_block * sb, unsigned long ino, struct list_head *head) { struct list_head *tmp; struct inode * inode;
tmp = head; for (;;) { tmp = tmp->next; inode = NULL; if (tmp == head) break; inode = list_entry(tmp, struct inode, i_hash); if (inode->i_sb != sb) /* <=== OOPS OCCURS HERE */ continue; if (inode->i_ino != ino) continue; inode->i_count++; break; } return inode; }
The second in fs/buffer.c (inlined into remove_from_queues): static inline void remove_from_hash_queue(struct buffer_head * bh) { struct buffer_head **pprev = bh->b_pprev; if (pprev) { struct buffer_head * next = bh->b_next; if (next) { next->b_pprev = pprev; /* <=== OOPS OCCURS HERE */ bh->b_next = NULL; } *pprev = next; bh->b_pprev = NULL; nr_hashed_buffers--; } }
In both cases, the pointer deref is the direct cause (cf the asm).
I've been trying to identify whether it's a parity error causing the problem, or it is SCSI related.
The DIMMs are non-parity (I know I know, but I didn't order the system, OK?), so this is high on my list of possible villains. OTOH, the memory blocks are from a reputedly high-quality vendor (Memory Card Technology), so I'm reluctant to make a blanket statement about bad DIMMs.
I'd be very happy for suggestions as to the cause of these oopses. I can try to reproduce them if need be (the box isn't getting released for 'production' before this has been resolved).
TIA,
Kai -- Kai Harrekilde-Petersen Exbit Technology A/S
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |