Messages in this thread | | | Date | Wed, 10 Apr 2024 11:27:09 +0100 | Subject | Re: [PATCH v2 1/4] arm64: mm: Don't remap pgtables per-cont(pte|pmd) block | From | Ryan Roberts <> |
| |
On 10/04/2024 10:46, Mark Rutland wrote: > On Thu, Apr 04, 2024 at 03:33:05PM +0100, Ryan Roberts wrote: >> A large part of the kernel boot time is creating the kernel linear map >> page tables. When rodata=full, all memory is mapped by pte. And when >> there is lots of physical ram, there are lots of pte tables to populate. >> The primary cost associated with this is mapping and unmapping the pte >> table memory in the fixmap; at unmap time, the TLB entry must be >> invalidated and this is expensive. >> >> Previously, each pmd and pte table was fixmapped/fixunmapped for each >> cont(pte|pmd) block of mappings (16 entries with 4K granule). This means >> we ended up issuing 32 TLBIs per (pmd|pte) table during the population >> phase. >> >> Let's fix that, and fixmap/fixunmap each page once per population, for a >> saving of 31 TLBIs per (pmd|pte) table. This gives a significant boot >> speedup. >> >> Execution time of map_mem(), which creates the kernel linear map page >> tables, was measured on different machines with different RAM configs: >> >> | Apple M2 VM | Ampere Altra| Ampere Altra| Ampere Altra >> | VM, 16G | VM, 64G | VM, 256G | Metal, 512G >> ---------------|-------------|-------------|-------------|------------- >> | ms (%) | ms (%) | ms (%) | ms (%) >> ---------------|-------------|-------------|-------------|------------- >> before | 153 (0%) | 2227 (0%) | 8798 (0%) | 17442 (0%) >> after | 77 (-49%) | 431 (-81%) | 1727 (-80%) | 3796 (-78%) >> >> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> >> Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com> >> Tested-by: Eric Chanudet <echanude@redhat.com> >> --- >> arch/arm64/mm/mmu.c | 32 ++++++++++++++++++-------------- >> 1 file changed, 18 insertions(+), 14 deletions(-) >> >> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c >> index 495b732d5af3..fd91b5bdb514 100644 >> --- a/arch/arm64/mm/mmu.c >> +++ b/arch/arm64/mm/mmu.c >> @@ -172,12 +172,9 @@ bool pgattr_change_is_safe(u64 old, u64 new) >> return ((old ^ new) & ~mask) == 0; >> } >> >> -static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, >> - phys_addr_t phys, pgprot_t prot) >> +static pte_t *init_pte(pte_t *ptep, unsigned long addr, unsigned long end, >> + phys_addr_t phys, pgprot_t prot) >> { >> - pte_t *ptep; >> - >> - ptep = pte_set_fixmap_offset(pmdp, addr); >> do { >> pte_t old_pte = __ptep_get(ptep); >> >> @@ -193,7 +190,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, >> phys += PAGE_SIZE; >> } while (ptep++, addr += PAGE_SIZE, addr != end); >> >> - pte_clear_fixmap(); >> + return ptep; >> } >> >> static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, >> @@ -204,6 +201,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, >> { >> unsigned long next; >> pmd_t pmd = READ_ONCE(*pmdp); >> + pte_t *ptep; >> >> BUG_ON(pmd_sect(pmd)); >> if (pmd_none(pmd)) { >> @@ -219,6 +217,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, >> } >> BUG_ON(pmd_bad(pmd)); >> >> + ptep = pte_set_fixmap_offset(pmdp, addr); >> do { >> pgprot_t __prot = prot; >> >> @@ -229,20 +228,20 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, >> (flags & NO_CONT_MAPPINGS) == 0) >> __prot = __pgprot(pgprot_val(prot) | PTE_CONT); >> >> - init_pte(pmdp, addr, next, phys, __prot); >> + ptep = init_pte(ptep, addr, next, phys, __prot); >> >> phys += next - addr; >> } while (addr = next, addr != end); > > I reckon it might be better to leave init_pte() returning void, and move the > ptep along here, e.g. > > ptep = pte_set_fixmap_offset(pmdp, addr); > do { > ... > > init_pte(ptep, addr, next, phys, __prot); > > ptep += pte_index(next) - pte_index(addr); > phys += next - addr; > } while (addr = next, addr != end); > > > ... as that keeps the relationship between 'ptep' and 'phys' clear since > they're manipulated in the same way, adjacent to one another. > > Regardless this looks good, so with that change or as-is: > > Acked-by: Mark Rutland <mark.rutland@arm.com> > > ... though I would prefer with that change. ;)
Yep, will change. And I'll do the same for pmd_init() too.
> > Mark.
| |