lkml.org 
[lkml]   [2008]   [Apr]   [30]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Announce: Semaphore-Removal tree
On Wed, Apr 30, 2008 at 04:06:13AM -0600, Matthew Wilcox wrote:
> On Tue, Apr 29, 2008 at 10:09:30AM +1000, David Chinner wrote:
> > > 2. l_flushsema
> > >
> > > This seems to be a completion. ie you're using it to wait for the log
> > > to be flushed.
> >
> > Yes, that could probably be a completion. I'm assuming that a completion
> > can handle several thousand waiting processes, right?
>
> By the way ... is it common that you get several thousand waiting
> processes? I ask because you wake them all up, then the herd thunders
> into the l_icloglock spinlock. Or is this a worst-case scenario that
> happens once in a blue moon?

I'm thinking of a certain 2048p machine at NASA where they run
MPI jobs that do synchronised exit() calls with about 6-7 open
file descriptors that all run into ->release at the same time
and try to do EOF truncation transcations all at the same time.

We get the first 100-200 processes filling all the log buffers
and forcing them to disk, and then the rest start waiting on
the flush sema.

Given that we're already pushing 4096p support for the next gen
machines, this problem isn't going to get any better...

> If l_flushsema does typically get more than one waiter, we can instead
> wake the waiters one at a time.

The current code does them one at a time, but in such a way that is
not any better than a thundering herd. wake_up_all() is probably
a more efficient thundering herd as it doesn't require picking up
and dropping a spinlock for every task being woken.

I think that we need to redesign this code to prevent the thundering
herd problem - it's not problem solvable by just changing the wakeup
call. Besides, the evidence points to the fact that a thundering
herd on the flush sema is not the limiting factor in thoughput.
There's still several layers of the onion to peel before
we get to that point.

Cheers,

Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group


\
 
 \ /
  Last update: 2008-04-30 13:05    [W:0.075 / U:0.280 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site