lkml.org 
[lkml]   [2013]   [Mar]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: PROBLEM: All CPUs in soft lockup
On Wed, Mar 27, 2013 at 12:55:41PM +1100, Robert Norris wrote:
> The console shows a new "BUG: soft lockup" line every few seconds

Looking closer, the whole thing starts with a _hard_ lockup.

2013-03-26T08:33:39.921834-04:00 imap30 kernel: [185090.090328] Watchdog detected hard LOCKUP on cpu 3

(also in the logs of the other two servers I mentioned).

Looking down to where the watchdog interrupt comes in:

2013-03-26T08:33:39.921870-04:00 imap30 kernel: [185090.090426] <<EOE>> <IRQ> [<ffffffff8112a5b1>] ? end_buffer_async_read+0x79/0xff

Disassembling:

0xffffffff8112a57a <+66>: mov %rbx,%rdi
0xffffffff8112a57d <+69>: callq 0xffffffff81129265 <buffer_io_error>
0xffffffff8112a582 <+74>: lock orb $0x2,0x0(%rbp)
0xffffffff8112a587 <+79>: mov 0x0(%rbp),%rax
0xffffffff8112a58b <+83>: test $0x8,%ah
0xffffffff8112a58e <+86>: jne 0xffffffff8112a594 <end_buffer_async_read+92>
0xffffffff8112a590 <+88>: ud2
0xffffffff8112a592 <+90>: jmp 0xffffffff8112a592 <end_buffer_async_read+90>

That lock at +74 is presumably the offender here. Which is line 275 of fs/buffer.c:

275 SetPageError(page);

So another CPU has these page flags locked right now, and isn't keen to
release that lock?

I don't know how to debug this further. What's the next step?

Thanks,
Rob.


\
 
 \ /
  Last update: 2013-03-28 02:01    [W:0.118 / U:0.036 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site