lkml.org 
[lkml]   [2011]   [Sep]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patch in this message
/
Date
From
SubjectRe: [PATCH 1/2] sched: Fix "divide error: 0000" in find_busiest_group
On Wed, Jul 20, 2011 at 05:32:08AM +0200, Mike Galbraith wrote:

> On Wed, 2011-07-20 at 04:29 +0200, Peter Zijlstra wrote:
> > On Wed, 2011-07-20 at 04:26 +0200, Mike Galbraith wrote:
> > > On Tue, 2011-07-19 at 23:17 +0200, Peter Zijlstra wrote:
> > > > On Tue, 2011-07-19 at 14:58 -0600, Terry Loftin wrote:
> > > > > Correct the protection expression in update_cpu_power() to avoid setting
> > > > > rq->cpu_power to zero.
> > > >
> > > > Firstly you fail to mention what kernel this is again, secondly this
> > > > should never happen in the first place, so this fix is wrong. At best it
> > > > papers over another bug.
> > > >
> > > > > Signed-off-by: Terry Loftin <terry.loftin@hp.com>
> > > > > Signed-off-by: Bob Montgomery <bob.montgomery@hp.com>
> > > > > ---
> > > > > diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
> > > > > index 0c26e2d..9c50020 100644
> > > > > --- a/kernel/sched_fair.c
> > > > > +++ b/kernel/sched_fair.c
> > > > > @@ -2549,7 +2549,7 @@ static void update_cpu_power(struct sched_domain *sd, int cpu)
> > > > > power *= scale_rt_power(cpu);
> > > > > power >>= SCHED_LOAD_SHIFT;
> > > > >
> > > > > - if (!power)
> > > > > + if ((u32)power == 0)
> > > > > power = 1;
> > > > >
> > > > > cpu_rq(cpu)->cpu_power = power;
> > >
> > > I put that (and a bunch more protection+warnings) in an enterprise
> > > kernel so it would not explode, but would gather some data. The entire
> > > world has been utterly silent, except for a gaggle of POWER7 boxen,
> > > which manage to convince scale_rt_power() to return negative values.
> > >
> > > Turning on PRINTK_TIME made these boxen go silent. A printk with
> > > timestamps, which doesn't happen, hides the problem. Tilt.
> >
> > Did those kernels contain the scale_rt_power() hunk from commit
> > aa483808516ca5cacfa0e5849691f64fec25828e? Venki thought that might cure
> > sure woes, but since we never could reproduce...
>
> Yeah, that commit is present.

We just hit what seems to be this bug on a box running 2.6.36 since
around the time it was built (Nov 8, 2010). It's a 16 core box (dual quad
with HT) and runs all sorts of stuff all day long, and suddenly hit this
divide error.

Commit aa483808516ca5cacfa0e5849691f64fec25828e is present.

find_busiest_group() seems to have been inlined into load_balance():

0xffffffff8104d55f <+1151>: jne 0xffffffff8104d568 <load_balance+1160>
0xffffffff8104d561 <+1153>: mov $0x1,%cl
0xffffffff8104d563 <+1155>: mov $0x1,%esi
0xffffffff8104d568 <+1160>: movslq -0x16c(%rbp),%rdx
0xffffffff8104d56f <+1167>: mov $0x14bc0,%rax
0xffffffff8104d576 <+1174>: mov -0x7e601720(,%rdx,8),%rdx
0xffffffff8104d57e <+1182>: mov %rcx,0x7e0(%rax,%rdx,1)
0xffffffff8104d586 <+1190>: mov %esi,0x8(%r8)
0xffffffff8104d58a <+1194>: nopw 0x0(%rax,%rax,1)
0xffffffff8104d590 <+1200>: mov -0x138(%rbp),%rcx
0xffffffff8104d597 <+1207>: mov -0x68(%rbp),%rsi
0xffffffff8104d59b <+1211>: xor %edx,%edx
0xffffffff8104d59d <+1213>: mov 0x8(%rcx),%edi
0xffffffff8104d5a0 <+1216>: mov %rsi,%rax
0xffffffff8104d5a3 <+1219>: mov -0x60(%rbp),%rcx
0xffffffff8104d5a7 <+1223>: shl $0xa,%rax
0xffffffff8104d5ab <+1227>: div %rdi <--------------
0xffffffff8104d5ae <+1230>: mov %rax,-0x70(%rbp)
0xffffffff8104d5b2 <+1234>: xor %eax,%eax
0xffffffff8104d5b4 <+1236>: test %rcx,%rcx
0xffffffff8104d5b7 <+1239>: je 0xffffffff8104d5c5 <load_balance+1253>
rax, rdx, and rdi were 0 here. Fuzzy picture available on request.

Other than some indirection, I don't see any changes in this area that
would fix this bug since 2.6.36, either. Perhaps the !power test in
update_cpu_power() should be copied to update_group_power()? This still
seems like papering over another issue, though...

Perhaps this might discover something:

diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index bc8ee99..b31cd3d 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -2682,6 +2682,7 @@ static void update_group_power(struct sched_domain *sd, int cpu)
} while (group != child->groups);

sdg->sgp->power = power;
+ BUG_ON(!power);
}

/*
Simon-


\
 
 \ /
  Last update: 2011-09-01 19:39    [from the cache]
©2003-2011 Jasper Spaans