lkml.org 
[lkml]   [2009]   [Aug]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [BUG] race of RCU vs NOHU
    On Fri, 21 Aug 2009 08:54:18 -0700
    "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

    > On Wed, Aug 12, 2009 at 09:32:33AM +0200, Martin Schwidefsky wrote:
    > > On Tue, 11 Aug 2009 11:04:07 -0700
    > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
    > >
    > > > On Tue, Aug 11, 2009 at 05:17:51PM +0200, Martin Schwidefsky wrote:
    > > > > On Tue, 11 Aug 2009 07:52:22 -0700
    > > > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
    > > > >
    > > > > > On Tue, Aug 11, 2009 at 12:56:53PM +0200, Martin Schwidefsky wrote:
    > > > > > > On Mon, 10 Aug 2009 08:08:07 -0700
    > > > > > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
    > > > > > >
    > > > > > > > On Mon, Aug 10, 2009 at 02:25:35PM +0200, Martin Schwidefsky wrote:
    > > > > > > > > On Fri, 7 Aug 2009 07:29:57 -0700
    > > > > > > > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
    > > > > > > > >
    > > > > > > > > > On Fri, Aug 07, 2009 at 03:15:29PM +0200, Martin Schwidefsky wrote:
    > > > > > > > > > > Hi Paul,
    > > > > > > > > > > I analysed a dump of a hanging 2.6.30 system and found what I think is
    > > > > > > > > > > a bug of RCU vs NOHZ. There are a number of patches ontop of that
    > > > > > > > > > > kernel but they should be independent of the bug.
    > > > > > > > > > >
    > > > > > > > > > > The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up
    > > > > > > > > > > recently, cpu #1 has been sleeping for 5 minutes, but there is a pending
    > > > > > > > > > > rcu batch. The timer wheel for cpu #1 is empty, it will continue to
    > > > > > > > > > > sleep for NEXT_TIMER_MAX_DELTA ticks.
    > > > > > > > > >
    > > > > > > > > > Congratulations, Martin! You have exercised what to date has been a
    > > > > > > > > > theoretical bug identified last year by Manfred Spraul. The fix is to
    > > > > > > > > > switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in
    > > > > > > > > > 2.6.29.
    > > > > > > > > >
    > > > > > > > > > Of course, if you need to work with an old kernel version, you might
    > > > > > > > > > still need a patch, perhaps for the various -stable versions. If so,
    > > > > > > > > > please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE
    > > > > > > > > > rather than backwards on CONFIG_RCU_CLASSIC.
    > > > > > > > >
    > > > > > > > > SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is
    > > > > > > > > present there and I think it needs to be fixed :-/
    > > > > > > >
    > > > > > > > I was afraid of that. ;-)
    > > > > > > >
    > > > > > > > Given that there are some other theoretical bugs in Classic RCU involving
    > > > > > > > interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more
    > > > > > > > sense than playing whack-a-mole on Classic RCU bugs?
    > > > > > >
    > > > > > > Fine with me but I don't know if SuSE/Novell is willing to accept such a
    > > > > > > big change for an existing distribution. I've put Ihno and Greg on Cc.
    > > > > >
    > > > > > Good point! While they are thinking about the tradeoff between
    > > > > > whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was
    > > > > > to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel
    > > > > > version(s) should I backport it to?
    > > > >
    > > > > We found the bug with kernel version 2.6.30 - the kernel on our test systems
    > > > > still use classic RCU. For us it is easy to switch to tree-RCU, no patch
    > > > > required.
    > > >
    > > > Ah! Could you please send me the test you use? My tests were
    > > > insufficient to force this problem to happen.
    > >
    > > There is no specific test, just a regular system boot. The boot did not
    > > finish and our tester took a dump. This boot failure seems to happen from
    > > time to time.
    >
    > OK. Has CONFIG_TREE_RCU been working for you? If so, which variant
    > of 2.6.27 do you need a backport to?

    We changed the configuration of our test kernels to CONFIG_TREE_RCU. So
    far the problem has not shown up again. As we a dealing with a rare race
    here this has to be taken with a grain of salt.

    --
    blue skies,
    Martin.

    "Reality continues to ruin my life." - Calvin.



    \
     
     \ /
      Last update: 2009-08-31 15:25    [W:0.071 / U:1.608 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site