lkml.org 
[lkml]   [2015]   [Apr]   [30]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH 0/13] Parallel struct page initialisation v4
On Wed, Apr 29, 2015 at 2:38 AM, nzimmer <nzimmer@sgi.com> wrote:
> On 04/28/2015 11:06 AM, Pekka Enberg wrote:
>> On Tue, Apr 28, 2015 at 5:36 PM, Mel Gorman <mgorman@suse.de> wrote:
>>> Struct page initialisation had been identified as one of the
>>> reasons why
>>> large machines take a long time to boot. Patches were posted a long
>>> time ago
>>> to defer initialisation until they were first used. This was
>>> rejected on
>>> the grounds it should not be necessary to hurt the fast paths. This
>>> series
>>> reuses much of the work from that time but defers the
>>> initialisation of
>>> memory to kswapd so that one thread per node initialises memory
>>> local to
>>> that node.
>>>
>>> After applying the series and setting the appropriate Kconfig
>>> variable I
>>> see this in the boot log on a 64G machine
>>>
>>> [ 7.383764] kswapd 0 initialised deferred memory in 188ms
>>> [ 7.404253] kswapd 1 initialised deferred memory in 208ms
>>> [ 7.411044] kswapd 3 initialised deferred memory in 216ms
>>> [ 7.411551] kswapd 2 initialised deferred memory in 216ms
>>>
>>> On a 1TB machine, I see
>>>
>>> [ 8.406511] kswapd 3 initialised deferred memory in 1116ms
>>> [ 8.428518] kswapd 1 initialised deferred memory in 1140ms
>>> [ 8.435977] kswapd 0 initialised deferred memory in 1148ms
>>> [ 8.437416] kswapd 2 initialised deferred memory in 1148ms
>>>
>>> Once booted the machine appears to work as normal. Boot times were
>>> measured
>>> from the time shutdown was called until ssh was available again.
>>> In the
>>> 64G case, the boot time savings are negligible. On the 1TB machine,
>>> the
>>> savings were 16 seconds.

> On an older 8 TB box with lots and lots of cpus the boot time, as
> measure from grub to login prompt, the boot time improved from 1484
> seconds to exactly 1000 seconds.
>
> I have time on 16 TB box tonight and a 12 TB box thursday and will
> hopefully have more numbers then.

Neat, and a roughly similar picture here.

On a 7TB, 1728-core NumaConnect system with 108 NUMA nodes, we're
seeing stock 4.0 boot in 7136s. This drops to 2159s, or a 70% reduction
with this patchset. Non-temporal PMD init [1] drops this to 1045s.

Nathan, what do you guys see with the non-temporal PMD patch [1]? Do
add a sfence at the ende label if you manually patch.

Thanks!
Daniel

[1] https://lkml.org/lkml/2015/4/23/350



\
 
 \ /
  Last update: 2015-04-30 18:41    [W:0.489 / U:0.196 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site