Messages in this thread | | | Date | Mon, 19 Sep 2022 14:08:23 +0200 | Subject | Re: Correlation CMA size and FORCE_MAX_ZONEORDER | From | David Hildenbrand <> |
| |
On 19.09.22 13:59, Michael Nazzareno Trimarchi wrote: > Hi David > > > On Mon, Sep 19, 2022 at 1:28 PM David Hildenbrand <david@redhat.com> wrote: >> >> On 19.09.22 13:17, Michael Nazzareno Trimarchi wrote: >>> Hi >>> >>> On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@redhat.com> wrote: >>>> >>>> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote: >>>>> Hi >>>>> >>>>> On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote: >>>>>> >>>>>> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote: >>>>>>> Hi David >>>>>>> >>>>>>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote: >>>>>>>> >>>>>>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote: >>>>>>>>> Hi all >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>>> >>>>>>>>> Working on a small device with 128MB of memory and using imx_v6_v7 >>>>>>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE >>>>>>>>> are not respected. The calculation done does not allow the requested >>>>>>>>> size. I think that this should be somehow documented and described but >>>>>>>>> I did not >>>>>>>>> find the documentation. Does it work this way? >>>>>>>>> >>>>>>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have >>>>>>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb >>>>>>>> >>>>>>>> The underlying constraint is that CMA regions require a certain minimum >>>>>>>> alignment+size. They cannot be arbitrarily in size. >>>>>>>> >>>>>>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream >>>>>>>> kernels to the size of a single pageblock. >>>>>>>> >>>>>>>> In previous kernels, it used to be the size of the largest buddy >>>>>>>> allocation granularity (derived from MAX_ORDER, derived from >>>>>>>> FORCE_MAX_ZONEORDER). >>>>>>>> >>>>>>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer >>>>>>>> apply. On most archs, the minimum alignment+size should be 2 MiB >>>>>>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock. >>>>>>>> >>>>>>>> So far the theory. Are you still running into this limitation on >>>>>>>> upstream kernels? >>>>>>>> >>>>>>> >>>>>>> I can run 6-rc2 on my board. I test again but according to it, if I >>>>>>> put 4M as CMA in cma=4M in boot >>>>>>> parameters, the result is 32Mb of CMA. Apart of that seems that >>>>>>> process lime tiny membench can not even start >>>>>>> to mblock memory >>>>>>> >>>>>> >>>>>> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you > >>>>>> 4M, can you post /proc/meminfo and the early console output? >>>>>> >>>>> >>>>> cat /proc/cmdline >>>>> cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs) >>>>> root=ubi0:root rw ubi.mtd=ro >>>>> otfs rootfstype=ubifs rootwait=1 >>>>> # cat /proc/meminfo >>>>> MemTotal: 109560 kB >>>>> MemFree: 56084 kB >>>>> MemAvailable: 56820 kB >>>>> Buffers: 0 kB >>>>> Cached: 39680 kB >>>>> SwapCached: 0 kB >>>>> Active: 44 kB >>>>> Inactive: 644 kB >>>>> Active(anon): 44 kB >>>>> Inactive(anon): 644 kB >>>>> Active(file): 0 kB >>>>> Inactive(file): 0 kB >>>>> Unevictable: 39596 kB >>>>> Mlocked: 0 kB >>>>> HighTotal: 0 kB >>>>> HighFree: 0 kB >>>>> LowTotal: 109560 kB >>>>> LowFree: 56084 kB >>>>> SwapTotal: 0 kB >>>>> SwapFree: 0 kB >>>>> Dirty: 0 kB >>>>> Writeback: 0 kB >>>>> AnonPages: 628 kB >>>>> Mapped: 1480 kB >>>>> Shmem: 84 kB >>>>> KReclaimable: 4268 kB >>>>> Slab: 8456 kB >>>>> SReclaimable: 4268 kB >>>>> SUnreclaim: 4188 kB >>>>> KernelStack: 392 kB >>>>> PageTables: 88 kB >>>>> NFS_Unstable: 0 kB >>>>> Bounce: 0 kB >>>>> WritebackTmp: 0 kB >>>>> CommitLimit: 54780 kB >>>>> Committed_AS: 1876 kB >>>>> VmallocTotal: 901120 kB >>>>> VmallocUsed: 2776 kB >>>>> VmallocChunk: 0 kB >>>>> Percpu: 72 kB >>>>> CmaTotal: 32768 kB >>>>> CmaFree: 32484 kB >>>>> # uname -a >>>>> Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux >>>>> # >>>>> >>>>> Then here https://pastebin.com/6MUB2VBM dmesg >>>>> >>>>> CONFIG_ARM_MODULE_PLTS=y >>>>> CONFIG_FORCE_MAX_ZONEORDER=14 >>>>> CONFIG_ALIGNMENT_TRAP=y >>>>> ... >>>>> CONFIG_CMA >>>>> CONFIG_CMA_AREAS=7 >>>>> ... >>>>> >>>>> CONFIG_CMA_SIZE_MBYTES=8 >>>>> CONFIG_CMA_SIZE_SEL_MBYTES=y >>>>> >>>> >>>> Thanks! >>>> >>>> I assume that in your setup, the pageblock size depends on MAX_ORDER >>>> and, therefore, FORCE_MAX_ZONEORDER. >>>> >>>> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined >>>> (include/linux/pageblock-flags.h). >>>> >>>> In contrast to what I remember, the pageblock size does not seem to >>>> depend on the THP size (weird) as well. >>>> >>>> >>>> So, yes, that limitation is still in effect for some kernel configs. >>>> >>>> One could make the pageblock size configurable (similar to >>>> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller >>>> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE. >>>> >>>> I imagine something reasonable might be to set the pageblock size to >>>> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA. >>>> >>> >>> I don't think making more configuration options makes things clear. >> >> Yes, in an ideal case it should be automatic. >> >>> When we enable some configuration >>> we can force down the configuration. You need to explain clearly how >>> you envision it. FORCE_MAX_ZONEORDER >>> for me is the largest allocation that you can get from a zone (ex CMA >>> one). Any request allocation that is align to the >> >> FORCE_MAX_ZONEORDER is just a way to increase/decrease the maximum >> allocation size of the buddy in general. >> >>> CMA align and can fit inside a region should be allowed >>> >>> What am I missing? >> >> I think that the issue is that the CMA alignments nowadays depend on the >> pageblock size. And the pageblock size depends on *some* configurations >> on the maximum allocation size of the buddy. >> >> Documenting the interaction between FORCE_MAX_ZONEORDER and CMA size >> alignment is not trivial. >> >> For example, with CONFIG_HUGETLB_PAGE there might not be such an >> interaction. With CONFIG_HUGETLB_PAGE, there clearly is one. >> >> >> Let me phrase it this way: is it good enough in you setup to get 32mb vs >> 8mb or do you want/need to reduce it without adjusting >> FORCE_MAX_ZONEORDER ? > > Wait we have: > - CMA kconfig alignment that in most config we have not considered > natural dma alignment but is put to 1Mb in a lot of embedded > - We have CMA_SIZE, CMA_SIZE_PERCENTAGE etc. Those seems that are not > effect if ZONEORDER is not reasonable and without > HUGETLB_PAGE > - etc > > Changing MAX_ZONEORDER is ok and yes if you have an IOT device that > you know about your CMA allocation, it makes no sense to have > it 32MB for a 128Mb device. What I point out is that I need to figure > it out because in Kconfig there is no mention of it. Should it be > added there?
Something like
"Note that in some configurations, the CMA size must be aligned to the maximum allocation granularity of the buddy allocator, consequently depending on FORCE_MAX_ZONEORDER. The requested CMA size might get increased accordingly."
Maybe we'd want some kind of a warning in the kernel as well. If someone specifies "cma=2MB" but gets 32MB or more that might be a problem.
Does something like that make sense to you?
-- Thanks,
David / dhildenb
| |