Messages in this thread | | | From | Jan Glauber <> | Subject | Re: [RFC] Disable lockref on arm64 | Date | Thu, 2 May 2019 08:38:41 +0000 |
| |
On Wed, May 01, 2019 at 05:01:40PM +0100, Will Deacon wrote: > Hi Jan, > > [+Peter and Linus, since they enjoy this stuff] > > On Mon, Apr 29, 2019 at 02:52:11PM +0000, Jan Glauber wrote: > > I've been looking into performance issues that were reported for several > > test-cases, for instance an nginx benchmark. > > Could you share enough specifics here so that we can reproduce the issue > locally, please? That would help us in our attempts to develop a fix without > simply disabling the option for everybody else.
I can send my test-case which is a trivial open-read-close loop with one thread per CPU and increasing read sizes.
> > It turned out the issue we have on ThunderX2 is the file open-close sequence > > with small read sizes. If the used files are opened read-only the > > lockref code (enabled by ARCH_USE_CMPXCHG_LOCKREF) is used. > > > > The lockref CMPXCHG_LOOP uses an unbound (as long as the associated > > spinlock isn't taken) while loop to change the lock count. This behaves > > badly under heavy contention (~25x retries for one cmpxchg to succeed > > with 28 threads operating on the same file). In case of a NUMA system > > it also behaves badly as the access from the other socket is much slower. > > It's surprising that this hasn't been reported on x86. I suspect their > implementation of cmpxchg is a little more forgiving under contention. > > > The fact that on ThunderX2 cpu_relax() turns only into one NOP > > instruction doesn't help either. On Intel pause seems to block the thread > > much longer, avoiding the heavy contention thereby. > > NOPing out the yield instruction seems like a poor choice for an SMT CPU > such as TX2. That said, the yield was originally added to cpu_relax() as > a scheduling hint for QEMU.
The issue is not limited to SMT, it also shows without SMT.
> > With the queued spinlocks implementation I can see a major improvement > > when I disable lockref. A trivial open-close test-case improves by > > factor 2 while system time is decreasing also 2x. Looking at kernel compile > > and dbench numbers didn't show any regression with lockref disabled. > > > > Can we simply disable lockref? Is anyone else seeing this issue? Is there > > an arm64 platform that actually implements yield? > > There are two issues with disabling lockref like this: > > 1. It's a compile-time thing, so systems that would benefit from the code > are unfairly penalised. > > 2. You're optimising for the contended case at the cost of the > uncontended case, which should actually be the common case as well.
I completely agree with 2). Nevertheless limiting the retry attempts like Linus suggested looks like a fair change that should not penalize anyone and would still help the contented case.
--Jan
> Now, nobody expects contended CAS to scale well, so the middle ground > probably involves backing off to the lock under contention, a bit like > an optimistic trylock(). Unfortunately, that will need some tuning, hence > my initial request for a reproducer. > > Cheers, > > Will
| |