| From | "Paul E. McKenney" <> | Subject | [PATCH tip/core/rcu 09/40] rcu: Make rcu_all_qs() do deferred NOCB wakeups | Date | Wed, 12 Apr 2017 10:39:54 -0700 |
| |
If a CONFIG_RCU_NOCB_CPUS kernel invokes call_rcu() with interrupts disabled, wakeups must be deferred in order to avoid self-deadlock in the cases where the disabled interrupts are due to scheduler locks being held. In this case, a flag is set and is checked on entry to extended quiescent states (usermode, idle), on exit from the RCU_SOFTIRQ handler, when the CPU in question goes offline, and on a subsequent invocation of call_rcu(). However, a given CPU could avoid all of those states for a considerable length of time.
This commit therefore allows an invocation of rcu_all_qs() to do the wakeup. It also makes the wakeup function clear the deferred-wakeup flag.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> --- kernel/rcu/tree.c | 6 ++++++ kernel/rcu/tree_plugin.h | 3 ++- 2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e8843bf0f5e1..eaa2b6d7abe2 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -522,6 +522,7 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch); void rcu_all_qs(void) { unsigned long flags; + struct rcu_state *rsp; barrier(); /* Avoid RCU read-side critical sections leaking down. */ if (unlikely(raw_cpu_read(rcu_sched_qs_mask))) { @@ -544,6 +545,11 @@ void rcu_all_qs(void) rcu_sched_qs(); preempt_enable(); } + for_each_rcu_flavor(rsp) { + preempt_disable(); + do_nocb_deferred_wakeup(this_cpu_ptr(rsp->rda)); + preempt_enable(); + } this_cpu_inc(rcu_qs_ctr); barrier(); /* Avoid RCU read-side critical sections leaking up. */ } diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 5a149091d262..0e2114e8076b 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -2201,7 +2201,8 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp) { int ndw; - if (!rcu_nocb_need_deferred_wakeup(rdp)) + if (!rcu_nocb_need_deferred_wakeup(rdp) || + unlikely(rcu_scheduler_active != RCU_SCHEDULER_RUNNING)) return; ndw = READ_ONCE(rdp->nocb_defer_wakeup); WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOGP_WAKE_NOT); -- 2.5.2
|