[lkml]   [2011]   [Sep]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [PATCH v1] sched: fix nohz idle load balancer issues
    On Tue, 2011-09-27 at 12:53 -0700, Venki Pallipadi wrote:
    > On Mon, Sep 26, 2011 at 4:50 AM, Srivatsa Vaddagiri
    > <> wrote:
    > >
    > > Reviewing idle load balancer code and testing it with some
    > > trace_printk(), I observed the following:
    > >
    > > 1. I had put a trace_printk() in nohz_idle_balance() as below:
    > >
    > > nohz_idle_balance()
    > > {
    > >
    > > if (idle != CPU_IDLE || !this_rq->nohz_balance_kick)
    > > return;
    > >
    > > ..
    > >
    > > trace_printk("Running rebalance for %d\n", balance_cpu);
    > >
    > > rebalance_domains(balance_cpu, CPU_IDLE);
    > > }
    > >
    > > I *never* got that printed during the test. Further investigation
    > > revealed that ilb_cpu was bailing out early because idle =
    > > CPU_NOT_IDLE i.e ilb_cpu was no longer idle_at_tick by the time it
    > > got around to handle the kick. As a result, no one was truly
    > > doing a load balance on behalf of sleeping idle cpus.

    One of the reasons why we saw lib_cpu not idle is probably because that
    info was stale.

    Consider this scenario.

    a. got a tick when the cpu was busy, so idle_at_tick was not set
    b. cpu went idle
    c. same cpu got the kick IPI from other busy cpu
    d. and as it has idle_at_tick not set, it couldn't proceed with the nohz
    idle balance.

    > > This patch is an attempt to solve above issues observed with idle load
    > > balancer.
    > >
    > > - The patch causes a ilb_cpu (that was kicked) to kick another idle
    > > cpu in case it was found to be !idle_at_tick (so that another idle cpu
    > > can do load balance on behalf of idle cpus). This fixes issue #1
    > Some comments:
    > Another potential change here is to
    > - either reverse the order of rebalance_domains() and
    > nohz_idle_balance() in run_rebalance_domains()
    > - or to kick another idle CPU in case of need_resched() in nohz_idle_balance.
    > This should help with idle balance of tickless CPUs when ilb CPU gets
    > a new task through load balance and hence aborts ilb.

    I think we are mostly likely seeing the above mentioned scenario.

    Also Vatsa, there is a deadlock associated by using
    __smp_call_funciton_single() in the nohz_balancer_kick(). So I am
    planning to remove the IPI that is used to kick the nohz balancer and
    instead use the resched_cpu logic to kick the nohz balancer.

    I will post this patch mostly tomorrow. That patch will not use the
    idle_at_tick check in the nohz_idle_balance(). So that should address
    your issue in some cases if not most.


     \ /
      Last update: 2011-09-28 01:51    [W:0.024 / U:3.572 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site