[lkml]   [2001]   [Apr]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: Re: nfs performance at high loads
I had sent in a note on nfs performance issues some time back,
and Mark Hemment had been kind enough to point out to the
zerocopy networking patch. Well, we tried with it, and it does
seem to have some improvement, but it seems to have screwed up
nfs performance a bit, because we see a LOT of rpc failures for
all kinds of calls, starting from lookup, to read and writes.
Could this possibly be triggered by this patch ( picked up from
davem's site for 2.4.0 ).
On the other hand, we do plan to migrate to 2.4.2. Can somebody
update me or provide pointers to info. as to whether we can
expect some of these problems have been resolved in 2.4.2? We
should soon be testing on 2.4.2

Get your own "800" number
Voicemail, fax, email, and a lot more

---- On Wed, 4 Apr 2001, Mark Hemment (

> I believe David Miller's latest zero-copy patches might help
> In his patch, the pull-up buffer is now allocated near the
top of
> stack
> (in the sunrpc code), so it can be a blocking allocation.
> This doesn't fix the core VM problems, but does relieve the
> _slightly_ on the VM (I assume, haven't tried David's patch
> One of the core problems is that the VM keeps no measure of
> page fragmentation in the free page pool. The system reaches
the state
> of
> having plenty of free single pages (so kswapd and friends
aren't kicked
> - or if they are, they do no or little word), and very few
buddied pages
> (which you need for some of the NFS requests).
> Unfortunately, even with keeping a mesaure of fragmentation,
> insuring work is done when it is reached, doesn't solve the
> problem.
> When a large order request comes in, the inactive_clean page
list is
> reaped. As reclaim_page() simply selects the "oldest" page it
can, with
> no regard as to whether it will buddy (now, or 'possibily in
the near
> future), this list is quickly shrunk by a large order request
- far too
> quickly for a well behaved system.
> An NFS write request, with an 8K block size, needs an
order-2 (16K)
> pull
> up buffer (we shouldn't really be pulling the header into the
> buffer
> as the data - perhaps we aren't any more?). On a well used
system, an
> order-2 _blocking_ allocation ends up populating the order-0
and order-1
> with quite a few pages from the inactive_clean.
> This then triggers another problem. :(
> As large (non-zero) order requests are always from the
> zones, these zones tend to have a lot of free-pages (put there
by the
> blind reclaim_page() - well, once you can do a blocking
allocation they
> are, or when the fragmentation kicking is working).
> New allocations for pages for the page-cache often ignore
> zone (it reaches a steady state), and so is passed over by the
loop at
> the
> head of __alloc_pages()).
> However, NORMAL and DMA zones tend to be above pages_low
(due to the
> reason above), and so new page-cache pages came from these
zones. On a
> HIGHMEM system this leads to thrashing of the NORMAL zone,
while the
> HIGHMEM zone stays (relatively) quiet.
> Note: To make matters even worse under this condition,
pulling pages
> out
> of the NORMAL zone is exactly what you don't want to happen!
It would
> be
> much better if they could be left alone for a (short) while to
give them
> chance to buddy - Linux (at present) doesn't care about the
budding of
> pages in the HIGHMEM zone (no non-zero allocations come from
> I was working on these problems (and some others) a few
months back,
> and
> will to return to them shortly. Unfortunately, the changes
started to
> look too large for 2.4....
> Also, for NFS, the best solution now might be to give the
nfsd threads
> a
> receive buffer. With David's patches, the pull-up occurs in
the context
> of a thread, making this possible.
> This doesn't solve the problem for other subsystems which do
> order page allocations, but (perhaps) they have a low enough
> not
> to be of real issue.
> Kapish,
> Note: Ensure you put a "sync" in your /etc/exports - the
> behaviour was "async" (not legal for a valid SpecFS run).
> Mark
> On Wed, 4 Apr 2001, Alan Cox wrote:
> > > We have been seeing some problems with running nfs
> > > at very high loads and were wondering if somebody could
> > > some pointers to where the problem lies.
> > > The system is a 2.4.0 kernel on a 6.2 Red at distribution
( so
> >
> > Use 2.2.19. The 2.4 VM is currently too broken to survive
high I/O
> benchmark
> > tests without going silly
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in
> > the body of a message to
> > More majordomo info at
> > Please read the FAQ at
> >
> -
> To unsubscribe from this list: send the line "unsubscribe
> in
> the body of a message to
> More majordomo info at
> Please read the FAQ at

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
More majordomo info at
Please read the FAQ at

 \ /
  Last update: 2005-03-22 12:52    [W:0.068 / U:1.268 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site