lkml.org 
[lkml]   [2012]   [Sep]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH tip/core/rcu 11/15] rcu: Avoid spurious RCU CPU stall warnings
    On Thu, Sep 06, 2012 at 05:19:18PM +0200, Peter Zijlstra wrote:
    > On Thu, 2012-09-06 at 11:07 -0400, Steven Rostedt wrote:
    > > On Thu, 2012-09-06 at 16:56 +0200, Peter Zijlstra wrote:
    > > > On Thu, 2012-08-30 at 11:56 -0700, Paul E. McKenney wrote:
    > > > >
    > > > > If a given CPU avoids the idle loop but also avoids starting a new
    > > > > RCU grace period for a full minute, RCU can issue spurious RCU CPU
    > > > > stall warnings. This commit fixes this issue by adding a check for
    > > > > ongoing grace period to avoid these spurious stall warnings.
    > > >
    > > > How would it avoid starting a new period for over a minute? fqs should
    > > > happen, right? And holding rcu_read_lock() for over a minute surely is a
    > > > bug.
    > >
    > > I can see this happening in test cases, but it would seem weird on a
    > > normal system. That is, for preempt rcu, having a process scheduled out
    > > holding an rcu_read_lock() for over a minute could happen on a really
    > > stressed out system. But for such a case, I don't think a warning is out
    > > of question.
    >
    > One would hope that fqs would boost things.. but yeah, if your app is
    > spinning above the rcu boost prio you're still toast. But in that case
    > you're right, a warning is fully deserved.

    Here are a few other ways that stalls can happen:

    o A CPU looping in an RCU read-side critical section.

    o A CPU looping with interrupts disabled. This condition can
    result in RCU-sched and RCU-bh stalls.

    o A CPU looping with preemption disabled. This condition can
    result in RCU-sched stalls and, if ksoftirqd is in use, RCU-bh
    stalls.

    o A CPU looping with bottom halves disabled. This condition can
    result in RCU-sched and RCU-bh stalls.

    o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel
    without invoking schedule().

    o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might
    happen to preempt a low-priority task in the middle of an RCU
    read-side critical section. This is especially damaging if
    that low-priority task is not permitted to run on any other CPU,
    in which case the next RCU grace period can never complete, which
    will eventually cause the system to run out of memory and hang.
    While the system is in the process of running itself out of
    memory, you might see stall-warning messages.

    o A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that
    is running at a higher priority than the RCU softirq threads.
    This will prevent RCU callbacks from ever being invoked,
    and in a CONFIG_TREE_PREEMPT_RCU kernel will further prevent
    RCU grace periods from ever completing. Either way, the
    system will eventually run out of memory and hang. In the
    CONFIG_TREE_PREEMPT_RCU case, you might see stall-warning
    messages.

    o A hardware or software issue shuts off the scheduler-clock
    interrupt on a CPU that is not in dyntick-idle mode. This
    problem really has happened, and seems to be most likely to
    result in RCU CPU stall warnings for CONFIG_NO_HZ=n kernels.

    o A bug in the RCU implementation.

    o A hardware failure. This is quite unlikely, but has occurred
    at least once in real life. A CPU failed in a running system,
    becoming unresponsive, but not causing an immediate crash.
    This resulted in a series of RCU CPU stall warnings, eventually
    leading the realization that the CPU had failed.

    Thanx, Paul



    \
     
     \ /
      Last update: 2012-09-07 00:01    [W:4.194 / U:0.056 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site