lkml.org 
[lkml]   [2008]   [Sep]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: BUG: soft lockup in 2.6.25.5

----- "Andrew Morton" <akpm@linux-foundation.org> wrote:

> On Fri, 19 Sep 2008 13:49:16 +1000 Brett Pemberton <brett@vpac.org>
> wrote:
>
> > I'm getting about 3-5 machines in a cluster of 95 hanging with
> >
> > BUG: soft lockup - CPU#7 stuck for 61s! [pdflush:321]
> >
> > per week. Nothing in common each time, different users running
> > different jobs on different nodes.
> >
> > The most recent is at the end of this email, .config is attached.
> >
> > Googling is scary. Many people reporting these, but never any
> response.
> > It's happening on enough separate nodes that I can't believe it's
> > hardware, although they are identical machines:
> >
> > - 2x Quad-Core AMD Opteron(tm) Processor 2356
> > - 32gb ram
> > - 4 x sata drives
> >
> > Running CentOS 5.2 with a kernel.org kernel
> > Has been happening with a variety of kernels from 2.6.25 - present.
>
> Yes, it's a false positive. With a lot of memory and a random-access
> or lot-of-files writing behaviour, it can take tremendous amounts of
> time to get everything stored on the disk.
>
> Not sure what to do about it, really. Perhaps touch the softlockup
> detector somewhere in the writeback code.
>
> > I'd love any advice on where to turn to next and what avenues to
> pursue.
>
> Set /proc/sys/kernel/softlockup_thresh to zero to shut it up :(
>

Hmm,

I'd love to believe it's a false positive, but I guess I didn't mention that
once a machine gets one of these, it incrementally increases load until it
falls over a few (hours, days) later.

When I've noticed, and have been able to log in and run top before it falls over,
I can see that one or two cores have their processes stuck in a wait state, while
the others continue as normal.

This is consistent. I've never had a node hit this BUG: and not eventually
die (lose all network connectivity, sit at console waiting for login but
not registering keystrokes) within a week. The node today hit lockup around
2 hours after registering this BUG:

Surely turning off the detection via the proc file will just mean this will
happen silently in the future?

cheers,

/ Brett


\
 
 \ /
  Last update: 2008-09-19 12:17    [W:0.074 / U:0.032 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site