lkml.org 
[lkml]   [2009]   [Oct]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH RFC tip/core/rcu 1/3] rcu: The Bloatwatch Edition, v7
On Wed, Oct 14, 2009 at 10:05:25AM +0800, Lai Jiangshan wrote:
> Paul E. McKenney wrote:
> > On Wed, Oct 14, 2009 at 08:37:18AM +0800, Lai Jiangshan wrote:
> >> Paul E. McKenney wrote:
> >>>> It's an old issue.
> >>>> It's not only about RCUTINY, it's also about other rcu implementations:
> >>>>
> >>>> rcu_enter_nohz()/rcu_exit_nohz() are not called in pairs.
> >>>>
> >>>> irq_exit() calls tick_nohz_stop_sched_tick() which calls rcu_enter_nohz(),
> >>>> where is the corresponding rcu_exit_nohz()?
> >>>> (or tick_nohz_restart_sched_tick())?
> >>> The tick_nohz_restart_sched_tick() function is called from the various
> >>> per-architecture cpu_idle() functions (or default_idle() or whatever
> >>> name that the architecture uses). For example, in:
> >>>
> >>> arch/x86/kernel/process_64.c
> >>>
> >>> the cpu_idle() function invokes tick_nohz_restart_sched_tick() just
> >>> before invoking schedule() to exit the idle loop.
> >>>
> >>> And, as you say, tick_nohz_restart_sched_tick() invokes rcu_exit_nohz().
> >> These tick_nohz_restart_sched_tick() which are called from the various
> >> per-architecture cpu_idle() functions are not the opposite of
> >> the tick_nohz_stop_sched_tick() in *irq_exit()*. So I figure that
> >> rcu_enter_nohz()/rcu_exit_nohz() are not called in pairs.
> >
> > OK, let's start with rcu_enter_nohz(), which tells RCU that the running
> > CPU is going into dyntick-idle mode, and thus should be ignored by RCU.
> > Let's do the idle loop first:
> >
> > o Upon entry to the idle() loop (using cpu_idle() in
> > arch/x86/kernel/process_64.c for this exercise),
> > we invoke tick_nohz_stop_sched_tick(1), which says we
> > are in an idle loop. (This is in contrast to the call
> > from irq_exit(), where we are not in the idle loop.)
> >
> > o tick_nohz_stop_sched_tick() invokes rcu_enter_nohz(),
> > does a bunch of timer checking, and returns. If anything
> > indicated that entering dyntick-idle mode would be bad,
> > we raise TIMER_SOFTIRQ to kick us out of this mode.
> >
> > Either way, we return to the idle loop.
> >
> > o The idle loops until need_resched(). Upon exit from the
> > idle loop, we call tick_nohz_restart_sched_tick(), which
> > invokes rcu_exit_nohz(), which tells RCU to start paying
> > attention to this CPU once more.
> >
> > OK, now for interrupts.
> >
> > o The hardware interrupt handlers invoke irq_enter(), which in
> > turn invokes rcu_irq_enter(). This has no real effect (other
> > than incrementing a counter) if the interrupt did not come
> > from dyntick-idle mode.
> >
> > Either way, RCU is now paying attention to RCU read-side
> > critical sections on this CPU.
> >
> > o Upon return from interrupt, the hardware interrupt handlers
> > invoke irq_exit(), which in turn invokes rcu_irq_exit().
> > This has no real effect (other than decrementing a counter)
> > if the interrupt is not returning to dyntick-idle mode.
> >
> > However, if the interrupt -is- returning to dyntick-idle
> > mode, then RCU will stop paying attention to RCU read-side
> > critical sections on this CPU.
>
>
> You haven't explain the tick_nohz_stop_sched_tick() in *irq_exit()*.
> (tick_nohz_stop_sched_tick() calls rcu_enter_nohz())
>
> void irq_exit(void)
> {
> ....
> rcu_irq_exit(); /* This is OK, the opposite is in irq_enter() */
> if (idle_cpu(smp_processor_id()) && !in_interrupt() && !need_resched())
> tick_nohz_stop_sched_tick(0); /* where is the opposite ??? */
> ....
> }
>
> This means if the interrupt -is- returning to dyntick-idle mode,
> rcu_enter_nohz() is called again.
>
> Take this flow as example:
>
> cpu_idle():
> while(1) {
>
> tick_nohz_stop_sched_tick()
> rcu_enter_nohz() *****

=== now RCU is in no_hz mode.

> ------->interrupt happen
> irq_enter()

rcu_irq_enter()

=== now RCU is no longer in no_hz mode.

> irq_exit()

rcu_irq_exit()

=== now RCU is in no_hz mode again.

> tick_nohz_stop_sched_tick()

=== but tick_nohz_stop_sched_tick() is passed "0" as the argument.
=== I might be missing something, but doesn't this prevent
=== rcu_enter_nohz() from being called at this point?

> rcu_enter_nohz() *****
> <-------interrupt returns
> tick_nohz_restart_sched_tick()
> rcu_exit_nohz() *****

=== now RCU is no longer in no_hz mode.

> } /* while(1) */
>
>
> You can see that rcu_enter_nohz() is called twice and
> rcu_exit_nohz() is only called once in this flow.
>
> It's because tick_nohz_stop_sched_tick()/tick_nohz_restart_sched_tick()
> are not called in pairs, so rcu_enter_nohz() and rcu_exit_nohz()
> are not called in pairs either.

I believe that the checks in tick_nohz_stop_sched_tick() prevent this
scenario from happening, but could easily be mistaken. I am not seeing
the WARN_ON_RATELIMIT() in rcu_exit_nohz(), however.

Thanx, Paul

> Lai
>
> >
> > So I do believe that rcu_enter_nohz() and rcu_exit_nohz() are in fact
> > invoked in pairs. One strange thing about this is that the idle loop
> > first invokes rcu_enter_nohz(), then invokes rcu_exit_nohz(), while
> > an interrupt handler first invokes rcu_irq_enter() and then invokes
> > rcu_irq_exit(). So the idle loop enters dyntick-idle mode and then
> > leaves it, while an interrupt handler might leave dyntick-idle mode and
> > then re-enter it.
> >
> > Or am I still missing something here?
> >
> > Thanx, Paul
> >
> >
>


\
 
 \ /
  Last update: 2009-10-14 04:59    [W:0.072 / U:1.520 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site