lkml.org 
[lkml]   [2009]   [May]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH RFC] v5 expedited "big hammer" RCU grace periods

* Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:

> On Tue, May 19, 2009 at 10:58:25AM +0200, Ingo Molnar wrote:
> >
> > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> >
> > > On Mon, May 18, 2009 at 05:42:41PM +0200, Ingo Molnar wrote:
> > > >
> > > > * Paul E. McKenney <paulmck@linux.vnet.ibm.com> wrote:
> > > >
> > > > > > i might be missing something fundamental here, but why not just
> > > > > > have per CPU helper threads, all on the same waitqueue, and wake
> > > > > > them up via a single wake_up() call? That would remove the SMP
> > > > > > cross call (wakeups do immediate cross-calls already).
> > > > >
> > > > > My concern with this is that the cache misses accessing all the
> > > > > processes on this single waitqueue would be serialized, slowing
> > > > > things down. In contrast, the bitmask that smp_call_function()
> > > > > traverses delivers on the order of a thousand CPUs' worth of bits
> > > > > per cache miss. I will give it a try, though.
> > > >
> > > > At least if you go via the migration threads, you can queue up
> > > > requests to them locally. But there's going to be cachemisses
> > > > _anyway_, since you have to access them all from a single CPU,
> > > > and then they have to fetch details about what to do, and then
> > > > have to notify the originator about completion.
> > >
> > > Ah, so you are suggesting that I use smp_call_function() to run
> > > code on each CPU that wakes up that CPU's migration thread? I
> > > will take a look at this.
> >
> > My suggestion was to queue up a dummy 'struct migration_req' up with
> > it (change migration_req::task == NULL to mean 'nothing') and simply
> > wake it up using wake_up_process().
>
> OK. I was thinking of just using wake_up_process() without the
> migration_req structure, and unconditionally setting a per-CPU
> variable from within migration_thread() just before the list_empty()
> check. In your approach we would need a NULL-pointer check just
> before the call to __migrate_task().
>
> > That will force a quiescent state, without the need for any extra
> > information, right?
>
> Yep!
>
> > This is what the scheduler code does, roughly:
> >
> > wake_up_process(rq->migration_thread);
> > wait_for_completion(&req.done);
> >
> > and this will always have to perform well. The 'req' could be put
> > into PER_CPU, and a loop could be done like this:
> >
> > for_each_online_cpu(cpu)
> > wake_up_process(cpu_rq(cpu)->migration_thread);
> >
> > for_each_online_cpu(cpu)
> > wait_for_completion(&per_cpu(req, cpu).done);
> >
> > hm?
>
> My concern is the linear slowdown for large systems, but this
> should be OK for modest systems (a few 10s of CPUs). However, I
> will try it out -- it does not need to be a long-term solution,
> after all.

I think there is going to be a linear slowdown no matter what -
because sending that many IPIs is going to be linear. (there are no
'broadcast to all' IPIs anymore - on x86 we only have them if all
physical APIC IDs are 7 or smaller.)

Also, no matter what scheme we use, the target CPU does have to be
processed somehow and it does have to signal completion back somehow
- which generates cachemisses.

I think what probaby matters most is to go simple, and to use
established kernel primitives - and the above is really typical
pattern for things like TLB flushes to a process having a presence
on every physical CPU. Those aspects will be kept reasonably fast
and balanced on all hardware that matters. (and if not, people will
notice any TLB flush/shootdown linear slowdowns and will address it)

I could be wrong though ... maybe someone can get some numbers from
a really large system?

Ingo


\
 
 \ /
  Last update: 2009-05-19 14:49    [W:0.137 / U:0.092 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site