lkml.org 
[lkml]   [2011]   [Apr]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: 2.6.39-rc4+: Kernel leaking memory during FS scanning, regression?
    On Thu, Apr 28, 2011 at 12:06:11AM +0200, Thomas Gleixner wrote:
    > On Wed, 27 Apr 2011, Bruno Prémont wrote:
    > > On Wed, 27 April 2011 Bruno Prémont wrote:
    > > Voluntary context switches stay constant from the time on SLABs pile up.
    > > (which makes sense as it doesn't run get CPU slices anymore)
    > >
    > > > > Can you please enable CONFIG_SCHED_DEBUG and provide the output of
    > > > > /proc/sched_stat when the problem surfaces and a minute after the
    > > > > first snapshot?
    > >
    > > hm, did you mean CONFIG_SCHEDSTAT or /proc/sched_debug?
    > >
    > > I did use CONFIG_SCHED_DEBUG (and there is no /proc/sched_stat) so I took
    > > /proc/sched_debug which exists... (attached, taken about 7min and +1min
    > > after SLABs started piling up), though build processes were SIGSTOPped
    > > during first minute.
    >
    > Oops. /proc/sched_debug is the right thing.
    >
    > > printk wrote (in case its timestamp is useful, more below):
    > > [ 518.480103] sched: RT throttling activated
    >
    > Ok. Aside of the fact that the CPU time accounting is completely hosed
    > this is pointing to the root cause of the problem.
    >
    > kthread_rcu seems to run in circles for whatever reason and the RT
    > throttler catches it. After that things go down the drain completely
    > as it should get on the CPU again after that 50ms throttling break.

    Ah. This could happen if there was a huge number of callbacks, in
    which case blimit would be set very large and kthread_rcu could then
    go CPU-bound. And this workload was generating large numbers of
    callbacks due to filesystem operations, right?

    So, perhaps I should kick kthread_rcu back to SCHED_NORMAL if blimit
    has been set high. Or have some throttling of my own. I must confess
    that throttling kthread_rcu for two hours seems a bit harsh. ;-)

    If this was just throttling kthread_rcu for a few hundred milliseconds,
    or even for a second or two, things would be just fine.

    Left to myself, I will put together a patch that puts callback processing
    down to SCHED_NORMAL in the case where there are huge numbers of
    callbacks to be processed.

    > Though we should not ignore the fact, that the RT throttler hit, but
    > none of the RT tasks actually accumulated runtime.
    >
    > So there is a couple of questions:
    >
    > - Why does the scheduler detect the 950 ms RT runtime, but does
    > not accumulate that runtime to any thread
    >
    > - Why is the runtime accounting totally hosed
    >
    > - Why does that not happen (at least not reproducible) with
    > TREE_RCU

    This one I can answer -- In Linus's tree, TREE_RCU still uses softirq,
    so there is no RCU kthread, so there is nothing to throttle other
    than ksoftirqd itself.

    Thanx, Paul

    > I need some sleep now, but I will try to come up with sensible
    > debugging tomorrow unless Paul or someone else beats me to it.
    >
    > Thanks,
    >
    > tglx

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2011-04-28 00:29    [W:0.024 / U:30.488 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site