lkml.org 
[lkml]   [2018]   [Apr]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    From
    Subject[PATCH tip/core/rcu 11/21] rcu: Switch __rcu_process_callbacks() to rcu_accelerate_cbs()
    Date
    The __rcu_process_callbacks() function currently checks to see if
    the current CPU needs a grace period and also if there is any other
    reason to kick off a new grace period. This is one of the fail-safe
    checks that has been rendered unnecessary by the changes that increase
    the accuracy of rcu_gp_cleanup()'s estimate as to whether another grace
    period is required. Because this particular fail-safe involved acquiring
    the root rcu_node structure's ->lock, which has seen excessive contention
    in real life, this fail-safe needs to go.

    However, one check must remain, namely the check for newly arrived
    RCU callbacks that have not yet been associated with a grace period.
    One might hope that the checks in __note_gp_changes(), which is invoked
    indirectly from rcu_check_quiescent_state(), would suffice, but this
    function won't be invoked at all if RCU is idle. It is therefore necessary
    to replace the fail-safe checks with a simpler check for newly arrived
    callbacks during an RCU idle period, which is exactly what this commit
    does. This change removes the final call to rcu_start_gp(), so this
    function is removed as well.

    Note that lockless use of cpu_needs_another_gp() is racy, but that
    these races are harmless in this case. If RCU really is idle, the
    values will not change, so the return value from cpu_needs_another_gp()
    will be correct. If RCU is not idle, the resulting redundant call to
    rcu_accelerate_cbs() will be harmless, and might even have the benefit
    of reducing grace-period latency a bit.

    This commit also moves interrupt disabling into the "if" statement to
    improve real-time response a bit.

    Reported-by: Nicholas Piggin <npiggin@gmail.com>
    Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
    ---
    kernel/rcu/tree.c | 53 +++++++++++++++--------------------------------------
    1 file changed, 15 insertions(+), 38 deletions(-)

    diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
    index 6396a3d10be9..fbacc486ed4c 100644
    --- a/kernel/rcu/tree.c
    +++ b/kernel/rcu/tree.c
    @@ -2335,34 +2335,6 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,
    }

    /*
    - * Similar to rcu_start_gp_advanced(), but also advance the calling CPU's
    - * callbacks. Note that rcu_start_gp_advanced() cannot do this because it
    - * is invoked indirectly from rcu_advance_cbs(), which would result in
    - * endless recursion -- or would do so if it wasn't for the self-deadlock
    - * that is encountered beforehand.
    - *
    - * Returns true if the grace-period kthread needs to be awakened.
    - */
    -static bool rcu_start_gp(struct rcu_state *rsp)
    -{
    - struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
    - struct rcu_node *rnp = rcu_get_root(rsp);
    - bool ret = false;
    -
    - /*
    - * If there is no grace period in progress right now, any
    - * callbacks we have up to this point will be satisfied by the
    - * next grace period. Also, advancing the callbacks reduces the
    - * probability of false positives from cpu_needs_another_gp()
    - * resulting in pointless grace periods. So, advance callbacks
    - * then start the grace period!
    - */
    - ret = rcu_advance_cbs(rsp, rnp, rdp) || ret;
    - ret = rcu_start_gp_advanced(rsp, rnp, rdp) || ret;
    - return ret;
    -}
    -
    -/*
    * Report a full set of quiescent states to the specified rcu_state data
    * structure. Invoke rcu_gp_kthread_wake() to awaken the grace-period
    * kthread if another grace period is required. Whether we wake
    @@ -2889,22 +2861,27 @@ __rcu_process_callbacks(struct rcu_state *rsp)
    unsigned long flags;
    bool needwake;
    struct rcu_data *rdp = raw_cpu_ptr(rsp->rda);
    + struct rcu_node *rnp;

    WARN_ON_ONCE(!rdp->beenonline);

    /* Update RCU state based on any recent quiescent states. */
    rcu_check_quiescent_state(rsp, rdp);

    - /* Does this CPU require a not-yet-started grace period? */
    - local_irq_save(flags);
    - if (cpu_needs_another_gp(rsp, rdp)) {
    - raw_spin_lock_rcu_node(rcu_get_root(rsp)); /* irqs disabled. */
    - needwake = rcu_start_gp(rsp);
    - raw_spin_unlock_irqrestore_rcu_node(rcu_get_root(rsp), flags);
    - if (needwake)
    - rcu_gp_kthread_wake(rsp);
    - } else {
    - local_irq_restore(flags);
    + /* No grace period and unregistered callbacks? */
    + if (!rcu_gp_in_progress(rsp) &&
    + rcu_segcblist_is_enabled(&rdp->cblist)) {
    + local_irq_save(flags);
    + if (rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL)) {
    + local_irq_restore(flags);
    + } else {
    + rnp = rdp->mynode;
    + raw_spin_lock_rcu_node(rnp); /* irqs disabled. */
    + needwake = rcu_accelerate_cbs(rsp, rnp, rdp);
    + raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
    + if (needwake)
    + rcu_gp_kthread_wake(rsp);
    + }
    }

    /* If there are callbacks ready, invoke them. */
    --
    2.5.2
    \
     
     \ /
      Last update: 2018-04-23 05:05    [W:4.045 / U:0.456 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site