lkml.org 
[lkml]   [2009]   [Aug]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [BUG] race of RCU vs NOHU
On Fri, 21 Aug 2009 08:54:18 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> On Wed, Aug 12, 2009 at 09:32:33AM +0200, Martin Schwidefsky wrote:
> > On Tue, 11 Aug 2009 11:04:07 -0700
> > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> >
> > > On Tue, Aug 11, 2009 at 05:17:51PM +0200, Martin Schwidefsky wrote:
> > > > On Tue, 11 Aug 2009 07:52:22 -0700
> > > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > >
> > > > > On Tue, Aug 11, 2009 at 12:56:53PM +0200, Martin Schwidefsky wrote:
> > > > > > On Mon, 10 Aug 2009 08:08:07 -0700
> > > > > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > > > >
> > > > > > > On Mon, Aug 10, 2009 at 02:25:35PM +0200, Martin Schwidefsky wrote:
> > > > > > > > On Fri, 7 Aug 2009 07:29:57 -0700
> > > > > > > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > > > > > >
> > > > > > > > > On Fri, Aug 07, 2009 at 03:15:29PM +0200, Martin Schwidefsky wrote:
> > > > > > > > > > Hi Paul,
> > > > > > > > > > I analysed a dump of a hanging 2.6.30 system and found what I think is
> > > > > > > > > > a bug of RCU vs NOHZ. There are a number of patches ontop of that
> > > > > > > > > > kernel but they should be independent of the bug.
> > > > > > > > > >
> > > > > > > > > > The systems has 4 cpus and uses classic RCU. cpus #0, #2 and #3 woke up
> > > > > > > > > > recently, cpu #1 has been sleeping for 5 minutes, but there is a pending
> > > > > > > > > > rcu batch. The timer wheel for cpu #1 is empty, it will continue to
> > > > > > > > > > sleep for NEXT_TIMER_MAX_DELTA ticks.
> > > > > > > > >
> > > > > > > > > Congratulations, Martin! You have exercised what to date has been a
> > > > > > > > > theoretical bug identified last year by Manfred Spraul. The fix is to
> > > > > > > > > switch from CONFIG_RCU_CLASSIC to CONFIG_RCU_TREE, which was added in
> > > > > > > > > 2.6.29.
> > > > > > > > >
> > > > > > > > > Of course, if you need to work with an old kernel version, you might
> > > > > > > > > still need a patch, perhaps for the various -stable versions. If so,
> > > > > > > > > please let me know -- otherwise, I will focus forward on CONFIG_RCU_TREE
> > > > > > > > > rather than backwards on CONFIG_RCU_CLASSIC.
> > > > > > > >
> > > > > > > > SLES11 is 2.6.27 and uses classic RCU. The not-so theoretical bug is
> > > > > > > > present there and I think it needs to be fixed :-/
> > > > > > >
> > > > > > > I was afraid of that. ;-)
> > > > > > >
> > > > > > > Given that there are some other theoretical bugs in Classic RCU involving
> > > > > > > interrupts and CONFIG_NO_HZ, would backporting CONFIG_TREE_RCU make more
> > > > > > > sense than playing whack-a-mole on Classic RCU bugs?
> > > > > >
> > > > > > Fine with me but I don't know if SuSE/Novell is willing to accept such a
> > > > > > big change for an existing distribution. I've put Ihno and Greg on Cc.
> > > > >
> > > > > Good point! While they are thinking about the tradeoff between
> > > > > whack-a-mole on Classic RCU and backporting CONFIG_TREE_RCU, if I was
> > > > > to send you a patch backporting CONFIG_TREE_RCU, to exactly which kernel
> > > > > version(s) should I backport it to?
> > > >
> > > > We found the bug with kernel version 2.6.30 - the kernel on our test systems
> > > > still use classic RCU. For us it is easy to switch to tree-RCU, no patch
> > > > required.
> > >
> > > Ah! Could you please send me the test you use? My tests were
> > > insufficient to force this problem to happen.
> >
> > There is no specific test, just a regular system boot. The boot did not
> > finish and our tester took a dump. This boot failure seems to happen from
> > time to time.
>
> OK. Has CONFIG_TREE_RCU been working for you? If so, which variant
> of 2.6.27 do you need a backport to?

We changed the configuration of our test kernels to CONFIG_TREE_RCU. So
far the problem has not shown up again. As we a dealing with a rare race
here this has to be taken with a grain of salt.

--
blue skies,
Martin.

"Reality continues to ruin my life." - Calvin.



\
 
 \ /
  Last update: 2009-08-31 15:25    [W:0.061 / U:0.312 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site