Messages in this thread |  | | Subject | Re: [PATCH] erofs: support adjust lz4 history window size | From | Huang Jianan <> | Date | Tue, 23 Feb 2021 10:03:59 +0800 |
| |
Hi Xiang,
On 2021/2/22 12:44, Gao Xiang wrote: > Hi Jianan, > > On Thu, Feb 18, 2021 at 08:00:49PM +0800, Huang Jianan via Linux-erofs wrote: >> From: huangjianan <huangjianan@oppo.com> >> >> lz4 uses LZ4_DISTANCE_MAX to record history preservation. When >> using rolling decompression, a block with a higher compression >> ratio will cause a larger memory allocation (up to 64k). It may >> cause a large resource burden in extreme cases on devices with >> small memory and a large number of concurrent IOs. So appropriately >> reducing this value can improve performance. >> >> Decreasing this value will reduce the compression ratio (except >> when input_size <LZ4_DISTANCE_MAX). But considering that erofs >> currently only supports 4k output, reducing this value will not >> significantly reduce the compression benefits. >> >> Signed-off-by: Huang Jianan <huangjianan@oppo.com> >> Signed-off-by: Guo Weichao <guoweichao@oppo.com> >> --- >> fs/erofs/decompressor.c | 13 +++++++++---- >> fs/erofs/erofs_fs.h | 3 ++- >> fs/erofs/internal.h | 3 +++ >> fs/erofs/super.c | 3 +++ >> 4 files changed, 17 insertions(+), 5 deletions(-) >> >> diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c >> index 1cb1ffd10569..94ae56b3ff71 100644 >> --- a/fs/erofs/decompressor.c >> +++ b/fs/erofs/decompressor.c >> @@ -36,22 +36,27 @@ static int z_erofs_lz4_prepare_destpages(struct z_erofs_decompress_req *rq, >> struct page *availables[LZ4_MAX_DISTANCE_PAGES] = { NULL }; >> unsigned long bounced[DIV_ROUND_UP(LZ4_MAX_DISTANCE_PAGES, >> BITS_PER_LONG)] = { 0 }; >> + unsigned int lz4_distance_pages = LZ4_MAX_DISTANCE_PAGES; >> void *kaddr = NULL; >> unsigned int i, j, top; >> >> + if (EROFS_SB(rq->sb)->compr_alg) >> + lz4_distance_pages = DIV_ROUND_UP(EROFS_SB(rq->sb)->compr_alg, >> + PAGE_SIZE) + 1; >> + > Thanks for your patch, I agree that will reduce runtime memory > footpoint. and keep max sliding window ondisk in bytes (rather > than in blocks) is better., but could we calculate lz4_distance_pages > ahead when reading super_block? Thanks for suggestion, i will update it soon. > Also, in the next cycle, I'd like to introduce a bitmap for available > algorithms (maximum 16-bit) for the next LZMA algorithm, and for each > available algorithm introduces an on-disk variable-array like below: > bitmap(16-bit) 2 1 0 > ... LZMA LZ4 > __le16 compr_opt_off; /* get the opt array start offset > (I think also in 4-byte) */ > > compr alg 0 (lz4) __le16 alg_opt_size; > /* next opt off = roundup(off + alg_opt_size, 4); */ > __le16 lz4_max_distance; > > /* 4-byte aligned */ > compr alg x (if available) u8 alg_opt_size; > ... > > ... > > When reading sb, first, it scans the whole bitmap, and get all the > available algorithms in the image at once. And then read such compr > opts one-by-one. > > Do you have some interest and extra time to implement it? :) That > makes me work less since I'm debugging mbpcluster compression now...
Sounds good, I will try to do this part of the work.
Thanks,
Jianan
> Thanks, > Gao Xiang >
|  |