Messages in this thread | | | Date | Wed, 21 Nov 2018 07:36:09 +0100 | From | Ingo Molnar <> | Subject | Re: [PATCH] x86: only use ERMS for user copies for larger sizes |
| |
[ Cc:-ed a few other gents and lkml. ]
* Jens Axboe <axboe@kernel.dk> wrote:
> Hi, > > So this is a fun one... While I was doing the aio polled work, I noticed > that the submitting process spent a substantial amount of time copying > data to/from userspace. For aio, that's iocb and io_event, which are 64 > and 32 bytes respectively. Looking closer at this, and it seems that > ERMS rep movsb is SLOWER for smaller copies, due to a higher startup > cost. > > I came up with this hack to test it out, and low and behold, we now cut > the time spent in copying in half. 50% less. > > Since these kinds of patches tend to lend themselves to bike shedding, I > also ran a string of kernel compilations out of RAM. Results are as > follows: > > Patched : 62.86s avg, stddev 0.65s > Stock : 63.73s avg, stddev 0.67s > > which would also seem to indicate that we're faster punting smaller > (< 128 byte) copies. > > CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz > > Interestingly, text size is smaller with the patch as well?! > > I'm sure there are smarter ways to do this, but results look fairly > conclusive. FWIW, the behaviorial change was introduced by: > > commit 954e482bde20b0e208fd4d34ef26e10afd194600 > Author: Fenghua Yu <fenghua.yu@intel.com> > Date: Thu May 24 18:19:45 2012 -0700 > > x86/copy_user_generic: Optimize copy_user_generic with CPU erms feature > > which contains nothing in terms of benchmarking or results, just claims > that the new hotness is better. > > Signed-off-by: Jens Axboe <axboe@kernel.dk> > --- > > diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h > index a9d637bc301d..7dbb78827e64 100644 > --- a/arch/x86/include/asm/uaccess_64.h > +++ b/arch/x86/include/asm/uaccess_64.h > @@ -29,16 +29,27 @@ copy_user_generic(void *to, const void *from, unsigned len) > { > unsigned ret; > > + /* > + * For smaller copies, don't use ERMS as it's slower. > + */ > + if (len < 128) { > + alternative_call(copy_user_generic_unrolled, > + copy_user_generic_string, X86_FEATURE_REP_GOOD, > + ASM_OUTPUT2("=a" (ret), "=D" (to), "=S" (from), > + "=d" (len)), > + "1" (to), "2" (from), "3" (len) > + : "memory", "rcx", "r8", "r9", "r10", "r11"); > + return ret; > + } > + > /* > * If CPU has ERMS feature, use copy_user_enhanced_fast_string. > * Otherwise, if CPU has rep_good feature, use copy_user_generic_string. > * Otherwise, use copy_user_generic_unrolled. > */ > alternative_call_2(copy_user_generic_unrolled, > - copy_user_generic_string, > - X86_FEATURE_REP_GOOD, > - copy_user_enhanced_fast_string, > - X86_FEATURE_ERMS, > + copy_user_generic_string, X86_FEATURE_REP_GOOD, > + copy_user_enhanced_fast_string, X86_FEATURE_ERMS, > ASM_OUTPUT2("=a" (ret), "=D" (to), "=S" (from), > "=d" (len)), > "1" (to), "2" (from), "3" (len)
So I'm inclined to do something like yours, because clearly the changelog of 954e482bde20 was at least partly false: Intel can say whatever they want, it's a fact that ERMS has high setup costs for low buffer sizes - ERMS is optimized for large size, cache-aligned copies mainly.
But the result is counter-intuitive in terms of kernel text footprint, plus the '128' is pretty arbitrary - we should at least try to come up with a break-even point where manual copy is about as fast as ERMS - on at least a single CPU ...
Thanks,
Ingo
| |