Messages in this thread Patch in this message | | | From | Linus Torvalds <> | Date | Sun, 19 Sep 2021 13:10:45 -0700 | Subject | Re: [GIT pull] locking/urgent for v5.15-rc2 |
| |
On Sun, Sep 19, 2021 at 11:28 AM Thomas Gleixner <tglx@linutronix.de> wrote: > > > - atomic_add(READER_BIAS - bias, &rwb->readers); > + /* > + * _release() is needed in case that reader is in fast path, pairing > + * with atomic_try_cmpxchg() in rwbase_read_trylock(), provides RELEASE > + */ > + (void)atomic_add_return_release(READER_BIAS - bias, &rwb->readers);
Ugh. This really needs fixing.
atomic_add() is already much more than release-ordered on x86, and atomic_add_return_release() is much more expensive on some uarchs.
I think it should be easy to add a atomic_add_release() function, and it might be as simple as the attached patch, allowing architectures to add their own arch_atomic_add_release() as needed.
I've pulled this, but please don't do things like the above hack.
Linus arch/x86/include/asm/atomic.h | 2 ++ include/linux/atomic/atomic-arch-fallback.h | 4 ++++ include/linux/atomic/atomic-instrumented.h | 7 +++++++ 3 files changed, 13 insertions(+)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 5e754e895767..6c5814177c73 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -55,6 +55,8 @@ static __always_inline void arch_atomic_add(int i, atomic_t *v) : "ir" (i) : "memory"); } +#define arch_atomic_add_release arch_atomic_add + /** * arch_atomic_sub - subtract integer from atomic variable * @i: integer value to subtract diff --git a/include/linux/atomic/atomic-arch-fallback.h b/include/linux/atomic/atomic-arch-fallback.h index a3dba31df01e..ae437d961bd1 100644 --- a/include/linux/atomic/atomic-arch-fallback.h +++ b/include/linux/atomic/atomic-arch-fallback.h @@ -165,6 +165,10 @@ arch_atomic_set_release(atomic_t *v, int i) #define arch_atomic_set_release arch_atomic_set_release #endif +#ifndef arch_atomic_add_release +#define arch_atomic_add_release (void)arch_atomic_add_return_release +#endif + #ifndef arch_atomic_add_return_relaxed #define arch_atomic_add_return_acquire arch_atomic_add_return #define arch_atomic_add_return_release arch_atomic_add_return diff --git a/include/linux/atomic/atomic-instrumented.h b/include/linux/atomic/atomic-instrumented.h index a0f654370da3..485b89804b9d 100644 --- a/include/linux/atomic/atomic-instrumented.h +++ b/include/linux/atomic/atomic-instrumented.h @@ -56,6 +56,13 @@ atomic_add(int i, atomic_t *v) arch_atomic_add(i, v); } +static __always_inline void +atomic_add_release(int i, atomic_t *v) +{ + instrument_atomic_read_write(v, sizeof(*v)); + arch_atomic_add_release(i, v); +} + static __always_inline int atomic_add_return(int i, atomic_t *v) { | |