lkml.org 
[lkml]   [2011]   [Apr]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: 2.6.39-rc4+: Kernel leaking memory during FS scanning, regression?
On Thu, Apr 28, 2011 at 12:06:11AM +0200, Thomas Gleixner wrote:
> On Wed, 27 Apr 2011, Bruno Prémont wrote:
> > On Wed, 27 April 2011 Bruno Prémont wrote:
> > Voluntary context switches stay constant from the time on SLABs pile up.
> > (which makes sense as it doesn't run get CPU slices anymore)
> >
> > > > Can you please enable CONFIG_SCHED_DEBUG and provide the output of
> > > > /proc/sched_stat when the problem surfaces and a minute after the
> > > > first snapshot?
> >
> > hm, did you mean CONFIG_SCHEDSTAT or /proc/sched_debug?
> >
> > I did use CONFIG_SCHED_DEBUG (and there is no /proc/sched_stat) so I took
> > /proc/sched_debug which exists... (attached, taken about 7min and +1min
> > after SLABs started piling up), though build processes were SIGSTOPped
> > during first minute.
>
> Oops. /proc/sched_debug is the right thing.
>
> > printk wrote (in case its timestamp is useful, more below):
> > [ 518.480103] sched: RT throttling activated
>
> Ok. Aside of the fact that the CPU time accounting is completely hosed
> this is pointing to the root cause of the problem.
>
> kthread_rcu seems to run in circles for whatever reason and the RT
> throttler catches it. After that things go down the drain completely
> as it should get on the CPU again after that 50ms throttling break.

Ah. This could happen if there was a huge number of callbacks, in
which case blimit would be set very large and kthread_rcu could then
go CPU-bound. And this workload was generating large numbers of
callbacks due to filesystem operations, right?

So, perhaps I should kick kthread_rcu back to SCHED_NORMAL if blimit
has been set high. Or have some throttling of my own. I must confess
that throttling kthread_rcu for two hours seems a bit harsh. ;-)

If this was just throttling kthread_rcu for a few hundred milliseconds,
or even for a second or two, things would be just fine.

Left to myself, I will put together a patch that puts callback processing
down to SCHED_NORMAL in the case where there are huge numbers of
callbacks to be processed.

> Though we should not ignore the fact, that the RT throttler hit, but
> none of the RT tasks actually accumulated runtime.
>
> So there is a couple of questions:
>
> - Why does the scheduler detect the 950 ms RT runtime, but does
> not accumulate that runtime to any thread
>
> - Why is the runtime accounting totally hosed
>
> - Why does that not happen (at least not reproducible) with
> TREE_RCU

This one I can answer -- In Linus's tree, TREE_RCU still uses softirq,
so there is no RCU kthread, so there is nothing to throttle other
than ksoftirqd itself.

Thanx, Paul

> I need some sleep now, but I will try to come up with sensible
> debugging tomorrow unless Paul or someone else beats me to it.
>
> Thanks,
>
> tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2011-04-28 00:29    [from the cache]
©2003-2011 Jasper Spaans