Messages in this thread | | | Date | Mon, 19 Dec 2005 16:34:35 -0600 | From | Cliff Wickman <> | Subject | Re: Very rare crash in prune_dcache |
| |
We've seen the below on at 2.6.5 kernel (SuSE SLES9) at SGI. Does it look like your crash?
The panic is by kswapd0:
<1>Unable to handle kernel NULL pointer dereference (address 0000000000000078) <4>kswapd0[122]: Oops 8813272891392 [1]
whose stack shows: [<a0000001001cecf0>] clear_inode+0x1b0/0x2c0 [<a0000001001d03d0>] generic_drop_inode+0x3b0/0x400 [<a0000001001ccf30>] iput+0x130/0x1c0 [<a00000020b6f0cd0>] nfs_dentry_iput+0x170/0x1c0 [nfs] [<a0000001001ca050>] prune_dcache+0x510/0x540 [<a0000001001ca0c0>] shrink_dcache_memory+0x40/0x80 [<a00000010014c360>] shrink_slab+0x2e0/0x440
Both generic_shutdown_super()'s calls to shrink_dcache_parent() or shrink_dcache_anon(), and kswapd0's call to shrink_dcache_memory() call prune_dcache(). I suspect a race condition inside prune_dcache().
The prune_dcache() function: lock dcache_lock scan the dentry_unused list of dentry's for a given number ("count") of dentry's to free: if a dentry to free, call prune_one_dentry() dentry_iput() unlock dcache_lock iput() any associated inode d_free() the dentry lock dcache_lock unlock dcache_lock
Two processors entering prune_dcache() near the same time will both scan the dentry_unused list and could try to iput() the same inode twice. That is because the dcache_lock is released while running iput().
I suppose the dcache_lock must be released here because the iput() may take a long time. And the dcache_lock is used many places in the system to protect the dentry cache's lists.
It would seem to me that a straighforward fix would be to add another lock to protect just the scan of the dentry_unused list only here in prune_dcache()
-Cliff Wickman
On Mon, Dec 19, 2005 at 04:38:55PM -0500, John F Flynn III wrote: > Good evening, folks... > > We have been experiencing a very rare (on average once every two to > three months) crash on some of our servers. > > uname -a: > Linux cheetah 2.6.9-22.0.1.ELsmp #1 SMP Thu Oct 27 13:14:25 CDT 2005 > i686 i686 i386 GNU/Linux > > (This is a CentOS provided kernel) > > Here is a photo of the bottom of the panic. Unfortunately the kernel has > no chance to log this anywhere else: > > http://www.cs.fiu.edu/~flynnj/cheetah-crash.jpg > > > The crash appears to be in prune_dcache, and has happened on several > distinct machines, so we do not believe it is a hardware problem. > > If anyone has pointers on what bug could be causing this crash, or if > it's been fixed in newer kernels we could try, it would be greatly > appreciated. This only seems to happen on loaded production machines, > and it happens so rarely that more detailed debugging is nearly impossible. > > Thanks in advance, > -John Flynn > > -- > John Flynn flynnj@cs.fiu.edu > ========================================================= > Systems and Network Administration /\_/\ > School of Computer Science ( O.O ) > Florida International University > < > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/
-- Cliff Wickman Silicon Graphics, Inc. cpw@sgi.com (651) 683-3824 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |