lkml.org 
[lkml]   [2021]   [Nov]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: [PATCH 2/2] kbuild: pass --stream-size --no-content-size to zstd
Date


> On Nov 20, 2021, at 8:53 AM, Alex Xu (Hello71) <alex_y_xu@yahoo.ca> wrote:
>
> Otherwise, it allocates 2 GB of memory at once. Even though the majority
> of this memory is never touched, the default heuristic overcommit
> refuses this request if less than 2 GB of RAM+swap is currently
> available. This results in "zstd: error 11 : Allocation error : not
> enough memory" and the kernel failing to build.
>
> When the size is specified, zstd will reduce the memory request
> appropriately. For typical kernel sizes of ~32 MB, the largest mmap
> request will be reduced to 512 MB, which will succeed on all but the
> smallest devices.
>
> For inputs around this size, --stream-size --no-content-size may
> slightly decrease the compressed size, or slightly increase it:
> https://github.com/facebook/zstd/issues/2848.

I like the change in general. However, I have a question of portability.
The --stream-size option was added in zstd-1.4.4 released November
2019 [0]. That means the build will fail for people with older zstd
versions. As a reference, Ubuntu Bionic (18.04) includes zstd-1.3.3,
and Ubuntu Focal (20.04) has zstd-1.4.4. So Bionic users will have to
install zstd manually to build the kernel.

I’d prefer to err on the side of caution for portability. I’d say once
Bionic hits end of life, it’d be safe to require zstd-1.4.4 or newer.
However, I see the need to reduce memory usage, as this issue has
come up before.

Could we detect the zstd version with `zstd --version` and pass
--stream-size if we detect the version is >= v1.4.4? That add some
complexity, but I think it would be worth it. Just be sure to document
why we are detecting support for --stream-size, and why it is important
to pass to zstd.

Best,
Nick Terrell

[0] https://github.com/facebook/zstd/releases/tag/v1.4.4

> Signed-off-by: Alex Xu (Hello71) <alex_y_xu@yahoo.ca>
> ---
> scripts/Makefile.lib | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index ca901814986a..13d756fbcdc7 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -468,10 +468,10 @@ quiet_cmd_xzmisc = XZMISC $@
> # be used because it would require zstd to allocate a 128 MB buffer.
>
> quiet_cmd_zstd = ZSTD $@
> - cmd_zstd = { cat $(real-prereqs) | $(ZSTD) -19; $(size_append); } > $@
> + cmd_zstd = { cat $(real-prereqs) | $(ZSTD) --stream-size=$(total_size) --no-content-size -19; $(size_append); } > $@
>
> quiet_cmd_zstd22 = ZSTD22 $@
> - cmd_zstd22 = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
> + cmd_zstd22 = { cat $(real-prereqs) | $(ZSTD) --stream-size=$(total_size) --no-content-size -22 --ultra; $(size_append); } > $@
>
> # ASM offsets
> # ---------------------------------------------------------------------------
> --
> 2.34.0
>

\
 
 \ /
  Last update: 2021-11-22 20:20    [W:0.056 / U:0.188 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site