lkml.org 
[lkml]   [2009]   [Jun]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: 2.6.29.1: nfsd: page allocation failure - nfsd or kernel problem?
David Rientjes wrote:
> On Thu, 18 Jun 2009, Michael Tokarev wrote:
>
>> David Rientjes wrote:
>>> On Thu, 18 Jun 2009, Michael Tokarev wrote:
>>>
>>>>> http://bugzilla.kernel.org/show_bug.cgi?id=13518
>>>> Does not look similar.
>>>>
>>>> I repeated the issue here. The slab which is growing here is buffer_head.
>>>> It's growing slowly -- right now, after ~5 minutes of constant writes over
>>>> nfs, its size is 428423 objects, growing at about 5000 objects/minute
>>>> rate.
>>>> When stopping writing, the cache shrinks slowly back to an acceptable
>>>> size, probably when the data gets actually written to disk.
>>> Not sure if you're referring to the bugzilla entry or Justin's reported
>>> issue. Justin's issue is actually allocating a skbuff_head_cache slab while
>>> the system is oom.
>> We have the same issue - I replied to Justin's initial email with exactly
>> the same trace as him. I didn't see your reply up until today, -- the one
>> you're referring to below.
>>
>
> If it's the exact same trace, then the page allocation failure is
> occurring as the result of slab's growth of the skbuff_head_cache cache,
> not buffer_head.

See http://lkml.org/lkml/2009/6/16/550 -- second message in this thread
is mine, it shows exactly the same trace.

> So it appears as though the issue you're raising is that buffer_head is
> consuming far too much memory, which causes the system to be oom when
> attempting a GFP_ATOMIC allocation for skbuff_head_cache and is otherwise
> unseen with alloc_buffer_head() because it is allowed to invoke direct
> reclaim:
>
> $ grep -r alloc_buffer_head\( fs/*
> fs/buffer.c: bh = alloc_buffer_head(GFP_NOFS);
> fs/buffer.c:struct buffer_head *alloc_buffer_head(gfp_t gfp_flags)
> fs/gfs2/log.c: bh = alloc_buffer_head(GFP_NOFS | __GFP_NOFAIL);
> fs/jbd/journal.c: new_bh = alloc_buffer_head(GFP_NOFS|__GFP_NOFAIL);
> fs/jbd2/journal.c: new_bh = alloc_buffer_head(GFP_NOFS|__GFP_NOFAIL);

Might be.

Here, I see the following scenario. With freshly booted server, 1.9Gb RAM,
slabtop shows about 11K entries in buffer_head slab, and about 1.7Gb free RAM.

When starting writing from another machine to this one over nfs, buffer_head
slab grows quite rapidly up to about 450K entries (total size 48940K) and
free memory drops to almost zero -- this happens in first 1..2 minutes
(GigE network, writing from /dev/zero using dd).

The cache does not grow further -- just because there's no free memory for
growing. On a 4Gb machine it grows up to about 920K objects.

And from time to time during write the same warning occurs. And slows
down write from ~70Mb/sec (it is almost the actual speed of the target
drive - it can do ~80Mb/sec) to almost zero for several seconds.

>> As far as I can see, the warning itself, while harmless, indicates some
>> deeper problem. Namely, we shouldn't have an OOM condition - the system
>> is doing nothing but NFS, there's only one NFS client which writes single
>> large file, the system has 2GB (or 4Gb on another machine) RAM. It should
>> not OOM to start with.
>
> Thanks to the page allocation failure that Justin posted earlier, which
> shows the state of the available system memory, it shows that the machine
> truly is oom. You seem to have isolated that to an enormous amount of
> buffer_head slab, which is a good start.

It's not really slabs it seems. In my case the total amount of buffer_heads
is about 49Mb which is very small compared with the amount of memory on the
system. But as far as I can *guess* buffer_head is just that - head, a
pointer to some other place... Unwritten or cached data?

Note that the only way to shrink that buffer_head cache back is to remove
the file in question on the server.

>> Well, there ARE side-effects actually. When the issue happens, the I/O
>> over NFS slows down to almost zero bytes/sec for some while, and resumes
>> slowly after about half a minute - sometimes faster, sometimes slower.
>> Again, the warning itself is harmless, but it shows a deeper issue. I
>> don't think it's wise to ignore the sympthom -- the actual cause should
>> be fixed instead. I think.
>
> Since the GFP_ATOMIC allocation cannot trigger reclaim itself, it must
> rely on other allocations or background writeout to free the memory and
> this will be considerably slower than a blocking allocation. The page
> allocation failure messages from Justin's post indicate there are 0 pages
> under writeback at the time of oom yet ZONE_NORMAL has reclaimable memory;
> this is the result of the nonblocking allocation.

So... what's the "consensus" so far? Just shut up the warning as you
initially proposed?

At least I don't see any immediately alternative. Well, but I don't know
kernel internals either :)

Thanks!

/mjt


\
 
 \ /
  Last update: 2009-06-18 20:01    [W:0.056 / U:11.156 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site