Messages in this thread | | | Date | Sun, 5 Dec 2010 17:07:36 +0100 | From | Mikael Pettersson <> | Subject | Re: [BUG] 2.6.37-rc3 massive interactivity regression on ARM |
| |
Russell King - ARM Linux writes: > On Sun, Dec 05, 2010 at 01:17:02PM +0000, Russell King - ARM Linux wrote: > > On Sun, Dec 05, 2010 at 01:32:37PM +0100, Mikael Pettersson wrote: > > > Mikael Pettersson writes: > > > > The scenario is that I do a remote login to an ARM build server, > > > > use screen to start a sub-shell, in that shell start a largish > > > > compile job, detach from that screen, and from the original login > > > > shell I occasionally monitor the compile job with top or ps or > > > > by attaching to the screen. > > > > > > > > With kernels 2.6.37-rc2 and -rc3 this causes the machine to become > > > > very sluggish: top takes forever to start, once started it shows no > > > > activity from the compile job (it's as if it's sleeping on a lock), > > > > and ps also takes forever and shows no activity from the compile job. > > > > > > > > Rebooting into 2.6.36 eliminates these issues. > > > > > > > > I do pretty much the same thing (remote login -> screen -> compile job) > > > > on other archs, but so far I've only seen the 2.6.37-rc misbehaviour > > > > on ARM EABI, specifically on an IOP n2100. (I have access to other ARM > > > > sub-archs, but haven't had time to test 2.6.37-rc on them yet.) > > > > > > > > Has anyone else seen this? Any ideas about the cause? > > > > > > (Re-followup since I just realised my previous followups were to Rafael's > > > regressions mailbot rather than the original thread.) > > > > > > > The bug is still present in 2.6.37-rc4. I'm currently trying to bisect it. > > > > > > git bisect identified > > > > > > [305e6835e05513406fa12820e40e4a8ecb63743c] sched: Do not account irq time to current task > > > > > > as the cause of this regression. Reverting it from 2.6.37-rc4 (requires some > > > hackery due to subsequent changes in the same area) restores sane behaviour. > > > > > > The original patch submission talks about irq-heavy scenarios. My case is the > > > exact opposite: UP, !PREEMPT, NO_HZ, very low irq rate, essentially 100% CPU > > > bound in userspace but expected to schedule quickly when needed (e.g. running > > > top or ps or just hitting CR in one shell while another runs a compile job). > > > > > > I've reproduced the misbehaviour with 2.6.37-rc4 on ARM/mach-iop32x and > > > ARM/mach-ixp4xx, but ARM/mach-kirkwood does not misbehave, and other archs > > > (x86 SMP, SPARC64 UP and SMP, PowerPC32 UP, Alpha UP) also do not misbehave. > > > > > > So it looks like an ARM-only issue, possibly depending on platform specifics. > > > > > > One difference I noticed between my Kirkwood machine and my ixp4xx and iop32x > > > machines is that even though all have CONFIG_NO_HZ=y, the timer irq rate is > > > much higher on Kirkwood, even when the machine is idle. > > > > The above patch you point out is fundamentally broken. > > > > + rq->clock = sched_clock_cpu(cpu); > > + irq_time = irq_time_cpu(cpu); > > + if (rq->clock - irq_time > rq->clock_task) > > + rq->clock_task = rq->clock - irq_time; > > > > This means that we will only update rq->clock_task if it is smaller than > > rq->clock. So, eventually over time, rq->clock_task becomes the maximum > > value that rq->clock can ever be. Or in other words, the maximum value > > of sched_clock_cpu(). > > > > Once that has been reached, although rq->clock will wrap back to zero, > > rq->clock_task will not, and so (I think) task execution time accounting > > effectively stops dead. > > > > I guess this hasn't been noticed on x86 as they have a 64-bit sched_clock, > > and so need to wait a long time for this to be noticed. However, on ARM > > where we tend to have 32-bit counters feeding sched_clock(), this value > > will wrap far sooner. > > I'm not so sure about this - certainly that if() statement looks very > suspicious above. As irq_time_cpu() will always be zero, can you try > removing the conditional? > > In any case, sched_clock_cpu() should be resilient against sched_clock() > wrapping. However, your comments about it being iop32x and ixp4xx > (both of which are 32-bit-counter-to-ns based implementations) and > kirkwood being a 32-bit-extended-to-63-bit-counter-to-ns implementation > does make me wonder...
I ran two tests on my iop32x machine:
1. Made the above-mentioned assignment to rq->clock_task unconditional. That cured the interactivity regressions.
2. Restored the conditional assignment to rq->clock_task and disabled the platform-specific sched_clock() so the kernel used the generic one. That too cured the interactivity regressions.
I then repeated these tests on my ixp4xx machine, with the same results.
I'll try to come up with a fix for the ixp4xx and plat-iop 32-bit sched_clock()s.
| |