lkml.org 
[lkml]   [2008]   [Dec]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: Page alloc failures under network/disk IO load
    From
    Date
    On Thu, 2008-12-04 at 13:54 -0500, Dan Noé wrote:
    > On Thu, 04 Dec 2008 09:42:08 +0100
    > Peter Zijlstra <peterz@infradead.org> wrote:
    >
    > > On Thu, 2008-12-04 at 09:23 +0100, Peter Zijlstra wrote:
    > > > On Wed, 2008-12-03 at 22:27 -0500, Dan Noé wrote:
    > > > > This is on Linux 2.6.28-rc7, on a Core 2 Duo. The system has
    > > > > plenty of memory:
    > > > >
    > > > > total used free shared buffers
    > > > > cached
    > > > > Mem: 1893 1822 70 0 0
    > > >
    > > > filled to the brim with data
    > > >
    > > > > 1573
    > > > > -/+ buffers/cache: 249 1644
    > > > > Swap: 1906 37 1868
    > > > >
    > > > > I am using rsync to transfer data onto this system. The
    > > > > filesystem is XFS, and the target drive is a 1TB Western Digital
    > > > > on ata_piix. The system files are on a RAID 1 (Linux md, also on
    > > > > ata_piix).
    > > > >
    > > > > Periodically I get page allocation failures, from
    > > > > __netdev_alloc_skb. I suppose this causes the driver to drop
    > > > > packets and thus hurts performance.
    > > >
    > > > There isn't much we can do about that, memory is filled and your
    > > > network card tries to allocate memory in a mode that doesn't allow
    > > > freeing some.
    > > >
    > > > Looking at the timestamps its not very frequent, so it doesn't hurt
    > > > performance much if anything. If you're really bothered with this,
    > > > you could quiet it by sticking in a __GFP_NOWARN in
    > > > __netdev_alloc_skb() or something..
    > >
    > > Another thing you can do is increase /proc/sys/vm/min_free_kbytes
    >
    > I'm a bit confused because on another system (2.6.26.3) I never see
    > messages like this despite having the same amount of physical RAM in
    > each. The 2.6.26.3 system is also under more active use, and has more
    > userspace memory usage. On that system:
    >
    > total used free shared buffers
    > cached Mem: 2017 1681 335 0
    > 99 603 -/+ buffers/cache: 979 1037
    > Swap: 972 137 835
    >
    > dpn@colobus:~$ cat /proc/sys/vm/min_free_kbytes
    > 3816
    >
    > Yet on the system where I saw the allocation failures:
    >
    > dpn@trout:~/kernels/linux-2.6$ cat /proc/sys/vm/min_free_kbytes
    > 5711
    >
    > If I understand it correctly the issue is that __netdev_alloc_skb must
    > make a GFP_ATOMIC allocation, which fails because the page cache must
    > evict pages before there is sufficient memory. And
    > min_free_kbytes allows tuning of the point where try_to_free_pages is
    > called and thus the "reserve" memory available. Is that correct?

    yes

    > Wouldn't a higher min_free_kbytes mean less likelihood of GFP_ATOMIC
    > allocations failing? Or are these allocations failing on my 2.6.26.3
    > system and I don't know it because of different config options?
    >
    > Why am I seeing this on the system with the *higher* min_free_kbytes?

    Higher burst rate? For the reserve pool to dry out, you need a high rate
    of incoming packets. If one machine has a steady workload and the other
    a bursty one, that could be the full difference.

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2008-12-05 08:15    [W:0.024 / U:59.668 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site