lkml.org 
[lkml]   [2012]   [Apr]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: CPU Hotplug rework
On Sat, Apr 07, 2012 at 01:45:44AM +0530, Srivatsa S. Bhat wrote:
> On 04/06/2012 04:36 AM, Paul E. McKenney wrote:
>
> > Hello,
> >
> > Here is my attempt at a summary of the discussion.
> >
>
>
> Thanks for the summary, it is really helpful :-)
>
> > Srivatsa, I left out the preempt_disable() pieces, but would be happy
> > to add them in when you let me know what you are thinking to do for
> > de-stop_machine()ing CPU hotplug.
> >
>
>
> Ok..
>
>
> > Thanx, Paul
> >
> > ------------------------------------------------------------------------
> >
> > CPU-hotplug work breakout:
> >
> > 1. Read and understand the current generic code.
> > Srivatsa Bhat has done this, as have Paul E. McKenney and
> > Peter Zijlstra to a lesser extent.
>
>
> "lesser extent"?? Hell no! :-) ;-)

Certainly to a lesser extent on my part, but yes, I should not speak
for Peter.

> > 2. Read and understand the architecture-specific code, looking
> > for opportunities to consolidate additional function into
> > core code.
> >
> > a. Carry out any indicated consolidation.
> >
> > b. Convert all architectures to make use of the
> > consolidated implementation.
> >
> > Not started. Low priority from a big.LITTLE perspective.
>
>
> Recently this unexpectedly assumed high priority due to some scheduler
> changes and things got fixed up temporarily. And in that context,
> Peter Zijlstra gave some more technical pointers on what is wrong and needs
> to be done right. Link: https://lkml.org/lkml/2012/3/22/149
>
> Nikunj (in CC) has offered to work with me on this consolidation.

Very cool! I have added the following:

------------------------------------------------------------------------
CONSOLIDATE ARCHITECTURE-SPECIFIC CPU-HOTPLUG CODE

1. Ensure that all CPU_STARTING notifiers complete before the
incoming CPU is marked online (the blackfin architecture
fails to do this).

2. Ensure that interrupts are disabled throughout the CPU_STARTING
notifiers. Currently, blackfin, cris, m32r, mips, sh, sparc64,
um, and x86 fail to do this properly.

3. Ensure that all architectures that use CONFIG_USE_GENERIC_SMP_HELPERS
hold ipi_call_lock() over the entire CPU-online process. Currently,
alpha, arm, m32r, mips, sh, and sparc32 seem to fail to do this
properly.

4. Additional memory barriers are likely to be needed, for example,
an smp_wmb() after setting cpu_active and an smp_rmb() in
select_fallback_rq() before reading cpu_active.

Srivatsa Bhat (srivatsa.bhat@linux.vnet.ibm.com) and Nikunj A Dadhania
(nikunj@linux.vnet.ibm.com) are taking on this work.

------------------------------------------------------------------------
Please let me know if adjustments are needed.

> > 3. Address the current kthread creation/teardown/migration
> > performance issues. (More details below.)
> >
> > Highest priority from a big.LITTLE perspective.
> >
> > 4. Wean CPU-hotplug offlining from stop_machine().
> > (More details below.)
> >
> > Moderate priority from a big.LITTLE perspective.
> >
> >
> > ADDRESSING KTHREAD CREATION/TEARDOWN/MIGRATION PERFORMANCE ISSUES
> >
> > 1. Evaluate approaches. Approaches currently under
> > consideration include:
> >
> > a. Park the kthreads rather than tearing them down or
> > migrating them. RCU currently takes this sort of
> > approach. Note that RCU currently relies on both
> > preempt_disable() and local_bh_disable() blocking the
> > current CPU from going offline.
> >
> > b. Allow in-kernel kthreads to avoid the delay
> > required to work around a bug in old versions of
> > bash. (This bug is a failure to expect receiving
> > a SIGCHILD signal corresponding to a child
> > created by a fork() system call that has not yet
> > returned.)
> >
> > This might be implemented using an additional
> > CLONE_ flag. This should allow kthreads to
> > be created and torn down much more quickly.
> >
> > c. Have some other TBD way to "freeze" a kthread.
> > (As in "your clever idea here".)
> >
> > 2. Implement the chosen approach or approaches. (Different
> > kernel subsystems might have different constraints, possibly
> > requiring different kthread handling.)
> >
> >
> > WEAN CPU-HOTPLUG OFFLINING FROM stop_machine()
> >
> >
> > 1. CPU_DYING notifier fixes needed as of 3.2:
> >
> > o vfp_hotplug(): I believe that this works as-is.
> > o s390_nohz_notify(): I believe that this works as-is.
> > o x86_pmu_notifier(): I believe that this works as-is.
> > o perf_ibs_cpu_notifier(): I don't know enough about
> > APIC to say.
> > o tboot_cpu_callback(): I believe that this works as-is,
> > but this one returns NOTIFY_BAD to a CPU_DYING notifier,
> > which is badness. But it looks like that case is a
> > "cannot happen" case. Still needs to be fixed.
> > o clockevents_notify(): This one acquires a global lock,
> > so it should be safe as-is.
> > o console_cpu_notify(): This one takes the same action
> > for CPU_ONLINE, CPU_DEAD, CPU_DOWN_FAILED, and
> > CPU_UP_CANCELLED that it does for CPU_DYING, so it
> > should be OK.
> > * rcu_cpu_notify(): This one needs adjustment as noted
> > above, but nothing major. Patch has been posted,
> > probably needs a bit of debugging.
> > o migration_call(): I defer to Peter on this one.
> > It looks to me like it is written to handle other
> > CPUs, but...
> > * workqueue_cpu_callback(): Might need help, does a
> > non-atomic OR.
> > o kvm_cpu_hotplug(): Uses a global spinlock, so should
> > be OK as-is.
> >
> > 2. Evaluate designs for stop_machine()-free CPU hotplug.
> > Implement the chosen design. An outline for a particular
> > design is shown below, but the actual design might be
> > quite different.
> >
> > 3. Fix issues with CPU Hotplug callback registration. Currently
> > there is no totally-race-free way to register callbacks and do
> > setup for already online cpus.
> >
> > Srivatsa had posted an incomplete patchset some time ago
> > regarding this, which gives an idea of the direction he had
> > in mind.
> > http://thread.gmane.org/gmane.linux.kernel/1258880/focus=15826
>
> Gah, this has been "incomplete" for quite some time now.. I'll try to speed up
> things a bit :-)

Sounds good to me! ;-)

Thanx, Paul



\
 
 \ /
  Last update: 2012-04-09 18:49    [from the cache]
©2003-2011 Jasper Spaans