Messages in this thread | | | Date | Sun, 26 Feb 2023 16:31:15 -0500 | Subject | Re: [PATCH 3/6] locking/rwsem: Rework writer wakeup | From | Waiman Long <> |
| |
On 2/26/23 07:00, Peter Zijlstra wrote: > On Thu, Feb 23, 2023 at 04:38:08PM -0500, Waiman Long wrote: > >>> @@ -1143,54 +1138,36 @@ rwsem_down_write_slowpath(struct rw_sema >>> } else { >>> atomic_long_or(RWSEM_FLAG_WAITERS, &sem->count); >>> } >>> + raw_spin_unlock_irq(&sem->wait_lock); >>> /* wait until we successfully acquire the lock */ >>> - set_current_state(state); >>> trace_contention_begin(sem, LCB_F_WRITE); >>> for (;;) { >>> - if (rwsem_try_write_lock(sem, &waiter)) { >>> - /* rwsem_try_write_lock() implies ACQUIRE on success */ >>> + set_current_state(state); >>> + if (!smp_load_acquire(&waiter.task)) { >>> + /* Matches rwsem_waiter_wake()'s smp_store_release(). */ >>> break; >>> } >>> - >>> - raw_spin_unlock_irq(&sem->wait_lock); >>> - >>> - if (signal_pending_state(state, current)) >>> - goto out_nolock; >>> - >>> - /* >>> - * After setting the handoff bit and failing to acquire >>> - * the lock, attempt to spin on owner to accelerate lock >>> - * transfer. If the previous owner is a on-cpu writer and it >>> - * has just released the lock, OWNER_NULL will be returned. >>> - * In this case, we attempt to acquire the lock again >>> - * without sleeping. >>> - */ >>> - if (waiter.handoff_set) { >>> - enum owner_state owner_state; >>> - >>> - owner_state = rwsem_spin_on_owner(sem); >>> - if (owner_state == OWNER_NULL) >>> - goto trylock_again; >>> + if (signal_pending_state(state, current)) { >>> + raw_spin_lock_irq(&sem->wait_lock); >>> + if (waiter.task) >>> + goto out_nolock; >>> + raw_spin_unlock_irq(&sem->wait_lock); >>> + /* Ordered by sem->wait_lock against rwsem_mark_wake(). */ >>> + break; >>> } >>> - >>> schedule_preempt_disabled(); >>> lockevent_inc(rwsem_sleep_writer); >>> - set_current_state(state); >>> -trylock_again: >>> - raw_spin_lock_irq(&sem->wait_lock); >>> } >>> __set_current_state(TASK_RUNNING); >>> - raw_spin_unlock_irq(&sem->wait_lock); >>> lockevent_inc(rwsem_wlock); >>> trace_contention_end(sem, 0); >>> return sem; >>> out_nolock: >>> - __set_current_state(TASK_RUNNING); >>> - raw_spin_lock_irq(&sem->wait_lock); >>> rwsem_del_wake_waiter(sem, &waiter, &wake_q); >>> + __set_current_state(TASK_RUNNING); >>> lockevent_inc(rwsem_wlock_fail); >>> trace_contention_end(sem, -EINTR); >>> return ERR_PTR(-EINTR); >> I believe it is better to change state inside the wait_lock critical section >> to provide a release barrier for free. > I can't follow... a release for what? Note that the reader slowpath has > this exact form already.\
You are right. I forgot that we don't need synchronization when setting state to TASK_RUNNING.
Cheers, Longman
| |