lkml.org 
[lkml]   [2020]   [Apr]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [PATCH v4 05/11] arm64: csum: Disable KASAN for do_csum()
From
Date
On 2020-04-24 10:41 am, David Laight wrote:
> From: Robin Murphy
>> Sent: 22 April 2020 12:02
> ..
>> Sure - I have a nagging feeling that it could still do better WRT
>> pipelining the loads anyway, so I'm happy to come back and reconsider
>> the local codegen later. It certainly doesn't deserve to stand in the
>> way of cross-arch rework.
>
> How fast does that loop actually run?

I've not characterised it in detail, but faster than any of the other
attempts so far ;)

> To my mind it seems to do a lot of operations on each 64bit value.
> I'd have thought that a loop based on:
> sum64 = *ptr;
> sum64_high = *ptr++ >> 32;
> and then fixing up the result would be faster.
>
> The x86-64 code is also bad!
> On intel cpu prior to haswell a simple:
> sum_64 += *ptr32++;
> is faster than the current code.
> (Although you can do a lot better even on ivy bridge.)

The aim here is to minimise load bandwidth - most Arm cores can slurp 16
bytes from L1 in a single load as quickly as any smaller amount, so
nibbling away in little 32-bit chunks would result in up to 4x more load
cycles. Yes, the C code looks ridiculous, but the other trick is that
most of those operations don't actually exist. Since a __uint128_t is
really backed by any two 64-bit GPRs - or if you're careful, one 64-bit
GPR and the carry flag - all those shifts and rotations are in fact
resolved by register allocation, so what we end up with is a very neat
loop of essentially just loads and 64-bit accumulation:

...
138: a94030c3 ldp x3, x12, [x6]
13c: a9412cc8 ldp x8, x11, [x6, #16]
140: a94228c4 ldp x4, x10, [x6, #32]
144: a94324c7 ldp x7, x9, [x6, #48]
148: ab03018d adds x13, x12, x3
14c: 510100a5 sub w5, w5, #0x40
150: 9a0c0063 adc x3, x3, x12
154: ab08016c adds x12, x11, x8
158: 9a0b0108 adc x8, x8, x11
15c: ab04014b adds x11, x10, x4
160: 9a0a0084 adc x4, x4, x10
164: ab07012a adds x10, x9, x7
168: 9a0900e7 adc x7, x7, x9
16c: ab080069 adds x9, x3, x8
170: 9a080063 adc x3, x3, x8
174: ab070088 adds x8, x4, x7
178: 9a070084 adc x4, x4, x7
17c: 910100c6 add x6, x6, #0x40
180: ab040067 adds x7, x3, x4
184: 9a040063 adc x3, x3, x4
188: ab010064 adds x4, x3, x1
18c: 9a030023 adc x3, x1, x3
190: 710100bf cmp w5, #0x40
194: aa0303e1 mov x1, x3
198: 54fffd0c b.gt 138 <do_csum+0xd8>
...

Instruction-wise, that's about as good as it can get short of
maintaining multiple accumulators and moving the pairwise folding out of
the loop. The main thing that I think is still left on the table is that
the load-to-use distances are pretty short and there's clearly scope to
spread out and amortise the load cycles better, which stands to benefit
both big and little cores.

Robin.

\
 
 \ /
  Last update: 2020-04-24 13:01    [W:0.188 / U:1.288 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site