lkml.org 
[lkml]   [2010]   [Oct]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 2/2 v2] rcu,cleanup: simplify the code when cpu is dying
    On Fri, Oct 22, 2010 at 03:35:13PM +0800, Lai Jiangshan wrote:
    > On 10/21/2010 03:25 AM, Paul E. McKenney wrote:
    > > On Wed, Oct 20, 2010 at 02:13:06PM +0800, Lai Jiangshan wrote:
    > >> When we handle cpu notify DYING, the whole system is stopped except
    > >> current CPU, so we can touch any data, and we remove the orphan_cbs_tail
    > >> and send the callbacks to the dest CPU directly.
    > >
    > > Queued along with the documentation/comment patch below, thank you!!!
    > > (Of course, please let me know if you see problems with my patch.)
    >
    > Your patch is good for me, please queue it, thanks.

    Very good, done!

    > > One remaining question... You use cpumask_any() to select the destination
    > > CPU, which sounds good until you look at its definition. The problem
    > > is that cpumask_any() always chooses the lowest-numbered online CPU.
    > > So imagine a (say) 64-CPU system and suppose that CPU 0 remains online.
    > > Suppose further that the other 63 CPUs each execute some workload that
    > > generates lots of RCU callbacks (perhaps creating then removing a large
    > > source tree), and periodically go offline and come back online.
    > >
    > > All of the RCU callbacks from CPUs 1-63 could easily end up getting
    > > dumped onto CPU 0's callback lists. It is easy to imagine that CPU 0
    > > might not be able to invoke these callbacks as fast as the other CPUs
    > > could generate them.
    > >
    > > Or am I missing something?
    >
    > It happens in the worst case. It may also happen before this patch.
    >
    > Before this patch, the callback move to the receive-CPU who handles the CPU_DEAD
    > event, and this CPU may be always cpu#0 in the worst case, the problem happens.
    >
    > And it's not help if I introduce a choose_receive_cpu_very_smart(),
    > Suppose further that the other 63 CPUs each execute some workload that
    > generates lots of RCU callbacks (perhaps creating then removing a large
    > source tree), and periodically go offline and come back online. In worse
    > case, in some period, there is only cpu#0 online, So all of the RCU callbacks
    > from CPUs 1-63 could easily end up getting dumped onto CPU 0's callback lists.
    > It is easy to imagine that CPU 0 might not be able to invoke these callbacks
    > as fast as the other CPUs could generate them.
    >
    > Another bad case(it may happens without this patch/with this patch
    > /with choose_receive_cpu_very_smart()):
    > Live-Lock, suppose cpu#A and cpu#B periodically go offline and come
    > back online, the callback may be moved from A to B and from B to A
    > periodically, no callback is handled.

    Agreed, it -could- happen before in the worst case, but it required very
    bad luck for the task adopting the callbacks to always be the same.
    In contrast, cpumask_any() will always pick on the same CPU.

    That said, your approach called out below is intriguing...

    > To fix these problems(it does really very hardly happen), we must force
    > all adopted callbacks are called before next cpu-offline. so we can use
    > work_on_cpu() or rcu_barrier() to do this. To make the code simpler, I will
    > use rcu_barrier().

    This approach is nice, but requires extensive testing -- a start would
    be a script that randomly onlines and offlines CPUs while rcutorture
    is running in the background. If you have not already done so, could
    you please give this an over-the-weekend test on the largest system
    you have access to?

    Thanx, Paul


    \
     
     \ /
      Last update: 2010-10-22 18:13    [W:0.025 / U:59.012 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site