lkml.org 
[lkml]   [2024]   [Mar]   [11]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: [RFC 00/14] Dynamic Kernel Stacks
    On Mon, Mar 11, 2024 at 1:09 PM Mateusz Guzik <mjguzik@gmail.com> wrote:
    >
    > On 3/11/24, Pasha Tatashin <pasha.tatashin@soleen.com> wrote:
    > > This is follow-up to the LSF/MM proposal [1]. Please provide your
    > > thoughts and comments about dynamic kernel stacks feature. This is a WIP
    > > has not been tested beside booting on some machines, and running LKDTM
    > > thread exhaust tests. The series also lacks selftests, and
    > > documentations.
    > >
    > > This feature allows to grow kernel stack dynamically, from 4KiB and up
    > > to the THREAD_SIZE. The intend is to save memory on fleet machines. From
    > > the initial experiments it shows to save on average 70-75% of the kernel
    > > stack memory.
    > >
    >

    Hi Mateusz,

    > Can you please elaborate how this works? I have trouble figuring it
    > out from cursory reading of the patchset and commit messages, that
    > aside I would argue this should have been explained in the cover
    > letter.

    Sure, I answered your questions below.

    > For example, say a thread takes a bunch of random locks (most notably
    > spinlocks) and/or disables preemption, then pushes some stuff onto the
    > stack which now faults. That is to say the fault can happen in rather
    > arbitrary context.
    >
    > If any of the conditions described below are prevented in the first
    > place it really needs to be described how.
    >
    > That said, from top of my head:
    > 1. what about faults when the thread holds a bunch of arbitrary locks
    > or has preemption disabled? is the allocation lockless?

    Each thread has a stack with 4 pages.
    Pre-allocated page: This page is always allocated and mapped at thread creation.
    Dynamic pages (3): These pages are mapped dynamically upon stack faults.

    A per-CPU data structure holds 3 dynamic pages for each CPU. These
    pages are used to handle stack faults occurring when a running thread
    faults (even within interrupt-disabled contexts). Typically, only one
    page is needed, but in the rare case where the thread accesses beyond
    that, we might use up to all three pages in a single fault. This
    structure allows for atomic handling of stack faults, preventing
    conflicts from other processes. Additionally, the thread's 16K-aligned
    virtual address (VA) and guaranteed pre-allocated page means no page
    table allocation is required during the fault.

    When a thread leaves the CPU in normal kernel mode, we check a flag to
    see if it has experienced stack faults. If so, we charge the thread
    for the new stack pages and refill the per-CPU data structure with any
    missing pages.

    > 2. what happens if there is no memory from which to map extra pages in
    > the first place? you may be in position where you can't go off cpu

    When the per-CPU data structure cannot be refilled, and a new thread
    faults, we issue a message indicating a critical stack fault. This
    triggers a system-wide panic similar to a guard page access violation

    Pasha

    \
     
     \ /
      Last update: 2024-05-27 15:44    [W:6.194 / U:0.012 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site