lkml.org 
[lkml]   [2010]   [Dec]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [BUG] 2.6.37-rc3 massive interactivity regression on ARM
On Sun, Dec 05, 2010 at 01:17:02PM +0000, Russell King - ARM Linux wrote:
> On Sun, Dec 05, 2010 at 01:32:37PM +0100, Mikael Pettersson wrote:
> > Mikael Pettersson writes:
> > > The scenario is that I do a remote login to an ARM build server,
> > > use screen to start a sub-shell, in that shell start a largish
> > > compile job, detach from that screen, and from the original login
> > > shell I occasionally monitor the compile job with top or ps or
> > > by attaching to the screen.
> > > With kernels 2.6.37-rc2 and -rc3 this causes the machine to become
> > > very sluggish: top takes forever to start, once started it shows no
> > > activity from the compile job (it's as if it's sleeping on a lock),
> > > and ps also takes forever and shows no activity from the compile job.
> > >
> > > Rebooting into 2.6.36 eliminates these issues.
> > >
> > > I do pretty much the same thing (remote login -> screen -> compile job)
> > > on other archs, but so far I've only seen the 2.6.37-rc misbehaviour
> > > on ARM EABI, specifically on an IOP n2100. (I have access to other ARM
> > > sub-archs, but haven't had time to test 2.6.37-rc on them yet.)
> > >
> > > Has anyone else seen this? Any ideas about the cause?
> >
> > (Re-followup since I just realised my previous followups were to Rafael's
> > regressions mailbot rather than the original thread.)
> >
> > > The bug is still present in 2.6.37-rc4. I'm currently trying to bisect it.
> >
> > git bisect identified
> >
> > [305e6835e05513406fa12820e40e4a8ecb63743c] sched: Do not account irq time to current task
> >
> > as the cause of this regression. Reverting it from 2.6.37-rc4 (requires some
> > hackery due to subsequent changes in the same area) restores sane behaviour.
> >
> > The original patch submission talks about irq-heavy scenarios. My case is the
> > exact opposite: UP, !PREEMPT, NO_HZ, very low irq rate, essentially 100% CPU
> > bound in userspace but expected to schedule quickly when needed (e.g. running
> > top or ps or just hitting CR in one shell while another runs a compile job).
> >
> > I've reproduced the misbehaviour with 2.6.37-rc4 on ARM/mach-iop32x and
> > ARM/mach-ixp4xx, but ARM/mach-kirkwood does not misbehave, and other archs
> > (x86 SMP, SPARC64 UP and SMP, PowerPC32 UP, Alpha UP) also do not misbehave.
> >
> > So it looks like an ARM-only issue, possibly depending on platform specifics.
> >
> > One difference I noticed between my Kirkwood machine and my ixp4xx and iop32x
> > machines is that even though all have CONFIG_NO_HZ=y, the timer irq rate is
> > much higher on Kirkwood, even when the machine is idle.
>
> The above patch you point out is fundamentally broken.
>
> + rq->clock = sched_clock_cpu(cpu);
> + irq_time = irq_time_cpu(cpu);
> + if (rq->clock - irq_time > rq->clock_task)
> + rq->clock_task = rq->clock - irq_time;
>
> This means that we will only update rq->clock_task if it is smaller than
> rq->clock. So, eventually over time, rq->clock_task becomes the maximum
> value that rq->clock can ever be. Or in other words, the maximum value
> of sched_clock_cpu().
>
> Once that has been reached, although rq->clock will wrap back to zero,
> rq->clock_task will not, and so (I think) task execution time accounting
> effectively stops dead.
>
> I guess this hasn't been noticed on x86 as they have a 64-bit sched_clock,
> and so need to wait a long time for this to be noticed. However, on ARM
> where we tend to have 32-bit counters feeding sched_clock(), this value
> will wrap far sooner.

I'm not so sure about this - certainly that if() statement looks very
suspicious above. As irq_time_cpu() will always be zero, can you try
removing the conditional?
In any case, sched_clock_cpu() should be resilient against sched_clock()
wrapping. However, your comments about it being iop32x and ixp4xx
(both of which are 32-bit-counter-to-ns based implementations) and
kirkwood being a 32-bit-extended-to-63-bit-counter-to-ns implementation
does make me wonder...


\
 
 \ /
  Last update: 2010-12-05 15:21    [from the cache]
©2003-2011 Jasper Spaans