lkml.org 
[lkml]   [2011]   [Sep]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    Date
    From
    SubjectRe: [PATCH 1/2] sched: Fix "divide error: 0000" in find_busiest_group
    On Wed, Jul 20, 2011 at 05:32:08AM +0200, Mike Galbraith wrote:

    > On Wed, 2011-07-20 at 04:29 +0200, Peter Zijlstra wrote:
    > > On Wed, 2011-07-20 at 04:26 +0200, Mike Galbraith wrote:
    > > > On Tue, 2011-07-19 at 23:17 +0200, Peter Zijlstra wrote:
    > > > > On Tue, 2011-07-19 at 14:58 -0600, Terry Loftin wrote:
    > > > > > Correct the protection expression in update_cpu_power() to avoid setting
    > > > > > rq->cpu_power to zero.
    > > > >
    > > > > Firstly you fail to mention what kernel this is again, secondly this
    > > > > should never happen in the first place, so this fix is wrong. At best it
    > > > > papers over another bug.
    > > > >
    > > > > > Signed-off-by: Terry Loftin <terry.loftin@hp.com>
    > > > > > Signed-off-by: Bob Montgomery <bob.montgomery@hp.com>
    > > > > > ---
    > > > > > diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
    > > > > > index 0c26e2d..9c50020 100644
    > > > > > --- a/kernel/sched_fair.c
    > > > > > +++ b/kernel/sched_fair.c
    > > > > > @@ -2549,7 +2549,7 @@ static void update_cpu_power(struct sched_domain *sd, int cpu)
    > > > > > power *= scale_rt_power(cpu);
    > > > > > power >>= SCHED_LOAD_SHIFT;
    > > > > >
    > > > > > - if (!power)
    > > > > > + if ((u32)power == 0)
    > > > > > power = 1;
    > > > > >
    > > > > > cpu_rq(cpu)->cpu_power = power;
    > > >
    > > > I put that (and a bunch more protection+warnings) in an enterprise
    > > > kernel so it would not explode, but would gather some data. The entire
    > > > world has been utterly silent, except for a gaggle of POWER7 boxen,
    > > > which manage to convince scale_rt_power() to return negative values.
    > > >
    > > > Turning on PRINTK_TIME made these boxen go silent. A printk with
    > > > timestamps, which doesn't happen, hides the problem. Tilt.
    > >
    > > Did those kernels contain the scale_rt_power() hunk from commit
    > > aa483808516ca5cacfa0e5849691f64fec25828e? Venki thought that might cure
    > > sure woes, but since we never could reproduce...
    >
    > Yeah, that commit is present.

    We just hit what seems to be this bug on a box running 2.6.36 since
    around the time it was built (Nov 8, 2010). It's a 16 core box (dual quad
    with HT) and runs all sorts of stuff all day long, and suddenly hit this
    divide error.

    Commit aa483808516ca5cacfa0e5849691f64fec25828e is present.

    find_busiest_group() seems to have been inlined into load_balance():

    0xffffffff8104d55f <+1151>: jne 0xffffffff8104d568 <load_balance+1160>
    0xffffffff8104d561 <+1153>: mov $0x1,%cl
    0xffffffff8104d563 <+1155>: mov $0x1,%esi
    0xffffffff8104d568 <+1160>: movslq -0x16c(%rbp),%rdx
    0xffffffff8104d56f <+1167>: mov $0x14bc0,%rax
    0xffffffff8104d576 <+1174>: mov -0x7e601720(,%rdx,8),%rdx
    0xffffffff8104d57e <+1182>: mov %rcx,0x7e0(%rax,%rdx,1)
    0xffffffff8104d586 <+1190>: mov %esi,0x8(%r8)
    0xffffffff8104d58a <+1194>: nopw 0x0(%rax,%rax,1)
    0xffffffff8104d590 <+1200>: mov -0x138(%rbp),%rcx
    0xffffffff8104d597 <+1207>: mov -0x68(%rbp),%rsi
    0xffffffff8104d59b <+1211>: xor %edx,%edx
    0xffffffff8104d59d <+1213>: mov 0x8(%rcx),%edi
    0xffffffff8104d5a0 <+1216>: mov %rsi,%rax
    0xffffffff8104d5a3 <+1219>: mov -0x60(%rbp),%rcx
    0xffffffff8104d5a7 <+1223>: shl $0xa,%rax
    0xffffffff8104d5ab <+1227>: div %rdi <--------------
    0xffffffff8104d5ae <+1230>: mov %rax,-0x70(%rbp)
    0xffffffff8104d5b2 <+1234>: xor %eax,%eax
    0xffffffff8104d5b4 <+1236>: test %rcx,%rcx
    0xffffffff8104d5b7 <+1239>: je 0xffffffff8104d5c5 <load_balance+1253>

    rax, rdx, and rdi were 0 here. Fuzzy picture available on request.

    Other than some indirection, I don't see any changes in this area that
    would fix this bug since 2.6.36, either. Perhaps the !power test in
    update_cpu_power() should be copied to update_group_power()? This still
    seems like papering over another issue, though...

    Perhaps this might discover something:

    diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
    index bc8ee99..b31cd3d 100644
    --- a/kernel/sched_fair.c
    +++ b/kernel/sched_fair.c
    @@ -2682,6 +2682,7 @@ static void update_group_power(struct sched_domain *sd, int cpu)
    } while (group != child->groups);

    sdg->sgp->power = power;
    + BUG_ON(!power);
    }

    /*
    Simon-


    \
     
     \ /
      Last update: 2011-09-01 19:39    [W:0.027 / U:0.084 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site