Messages in this thread | | | Date | Mon, 9 May 2016 22:25:54 -0400 | From | Waiman Long <> | Subject | Re: [PATCH] locking/rwsem: Optimize write lock slowpath |
| |
On 05/09/2016 03:16 PM, Jason Low wrote: > When acquiring the rwsem write lock in the slowpath, we first try > to set count to RWSEM_WAITING_BIAS. When that is successful, > we then atomically add the RWSEM_WAITING_BIAS in cases where > there are other tasks on the wait list. This causes write lock > operations to often issue multiple atomic operations. > > We can instead make the list_is_singular() check first, and then > set the count accordingly, so that we issue at most 1 atomic > operation when acquiring the write lock and reduce unnecessary > cacheline contention. > > Signed-off-by: Jason Low<jason.low2@hp.com> > --- > kernel/locking/rwsem-xadd.c | 20 +++++++++++++------- > 1 file changed, 13 insertions(+), 7 deletions(-) > > diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c > index df4dcb8..23c33e6 100644 > --- a/kernel/locking/rwsem-xadd.c > +++ b/kernel/locking/rwsem-xadd.c > @@ -258,14 +258,20 @@ EXPORT_SYMBOL(rwsem_down_read_failed); > static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem) > { > /* > - * Try acquiring the write lock. Check count first in order > - * to reduce unnecessary expensive cmpxchg() operations. > + * Avoid trying to acquire write lock if count isn't RWSEM_WAITING_BIAS. > */ > - if (count == RWSEM_WAITING_BIAS&& > - cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS, > - RWSEM_ACTIVE_WRITE_BIAS) == RWSEM_WAITING_BIAS) { > - if (!list_is_singular(&sem->wait_list)) > - rwsem_atomic_update(RWSEM_WAITING_BIAS, sem); > + if (count != RWSEM_WAITING_BIAS) > + return false; > + > + /* > + * Acquire the lock by trying to set it to ACTIVE_WRITE_BIAS. If there > + * are other tasks on the wait list, we need to add on WAITING_BIAS. > + */ > + count = list_is_singular(&sem->wait_list) ? > + RWSEM_ACTIVE_WRITE_BIAS : > + RWSEM_ACTIVE_WRITE_BIAS + RWSEM_WAITING_BIAS; > + > + if (cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS, count) == RWSEM_WAITING_BIAS) { > rwsem_set_owner(sem); > return true; > }
Acked-by: Waiman Long<Waiman.Long@hpe.com>
| |