Messages in this thread | | | Date | Fri, 01 May 2015 00:10:27 +0800 | From | Daniel J Blueman <> | Subject | Re: [PATCH 0/13] Parallel struct page initialisation v4 |
| |
On Wed, Apr 29, 2015 at 2:38 AM, nzimmer <nzimmer@sgi.com> wrote: > On 04/28/2015 11:06 AM, Pekka Enberg wrote: >> On Tue, Apr 28, 2015 at 5:36 PM, Mel Gorman <mgorman@suse.de> wrote: >>> Struct page initialisation had been identified as one of the >>> reasons why >>> large machines take a long time to boot. Patches were posted a long >>> time ago >>> to defer initialisation until they were first used. This was >>> rejected on >>> the grounds it should not be necessary to hurt the fast paths. This >>> series >>> reuses much of the work from that time but defers the >>> initialisation of >>> memory to kswapd so that one thread per node initialises memory >>> local to >>> that node. >>> >>> After applying the series and setting the appropriate Kconfig >>> variable I >>> see this in the boot log on a 64G machine >>> >>> [ 7.383764] kswapd 0 initialised deferred memory in 188ms >>> [ 7.404253] kswapd 1 initialised deferred memory in 208ms >>> [ 7.411044] kswapd 3 initialised deferred memory in 216ms >>> [ 7.411551] kswapd 2 initialised deferred memory in 216ms >>> >>> On a 1TB machine, I see >>> >>> [ 8.406511] kswapd 3 initialised deferred memory in 1116ms >>> [ 8.428518] kswapd 1 initialised deferred memory in 1140ms >>> [ 8.435977] kswapd 0 initialised deferred memory in 1148ms >>> [ 8.437416] kswapd 2 initialised deferred memory in 1148ms >>> >>> Once booted the machine appears to work as normal. Boot times were >>> measured >>> from the time shutdown was called until ssh was available again. >>> In the >>> 64G case, the boot time savings are negligible. On the 1TB machine, >>> the >>> savings were 16 seconds.
> On an older 8 TB box with lots and lots of cpus the boot time, as > measure from grub to login prompt, the boot time improved from 1484 > seconds to exactly 1000 seconds. > > I have time on 16 TB box tonight and a 12 TB box thursday and will > hopefully have more numbers then.
Neat, and a roughly similar picture here.
On a 7TB, 1728-core NumaConnect system with 108 NUMA nodes, we're seeing stock 4.0 boot in 7136s. This drops to 2159s, or a 70% reduction with this patchset. Non-temporal PMD init [1] drops this to 1045s.
Nathan, what do you guys see with the non-temporal PMD patch [1]? Do add a sfence at the ende label if you manually patch.
Thanks! Daniel
[1] https://lkml.org/lkml/2015/4/23/350
| |