lkml.org 
[lkml]   [2008]   [Mar]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 01/30] swap over network documentation
    On Thu, 20 Mar 2008 21:10:43 +0100 Peter Zijlstra wrote:

    > Document describing the problem and proposed solution
    >
    > Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
    > ---
    > Documentation/network-swap.txt | 270 +++++++++++++++++++++++++++++++++++++++++
    > 1 file changed, 270 insertions(+)
    >
    > Index: linux-2.6/Documentation/network-swap.txt
    > ===================================================================
    > --- /dev/null
    > +++ linux-2.6/Documentation/network-swap.txt
    > @@ -0,0 +1,270 @@

    ...

    > +There are several major parts to this enhancement:
    > +
    > +1/ page->reserve, GFP_MEMALLOC

    ...

    > + For memory allocated using slab/slub: If a page that is added to a
    > + kmem_cache is found to have page->reserve set, then a s->reserve

    then an

    > + flag is set for the whole kmem_cache. Further allocations will only
    > + be returned from that page (or any other page in the cache) if they
    > + are emergency allocation (i.e. PF_MEMALLOC or GFP_MEMALLOC is set).

    allocations

    > + Non-emergency allocations will block in alloc_page until a
    > + non-reserve page is available. Once a non-reserve page has been
    > + added to the cache, the s->reserve flag on the cache is removed.
    > +
    > + Because slab objects have no individual state its hard to pass

    it's (or "it is")

    > + reserve state along, the current code relies on a regular alloc

    so the

    > + failing. There are various allocation wrappers help here.

    wrappers to help here. (?)

    > +
    > + This allows us to
    > + a/ request use of the emergency pool when allocating memory
    > + (GFP_MEMALLOC), and
    > + b/ to find out if the emergency pool was used.
    > +
    > +2/ SK_MEMALLOC, sk_buff->emergency.
    > +
    ...
    > +
    > + Similarly, if an skb is ever queued for delivery to user-space for

    user-space, for

    > + example by netfilter, the ->emergency flag is tested and the skb is
    > + released if ->emergency is set. (so obviously the storage route may
    > + not pass through a userspace helper, otherwise the packets will never
    > + arrive and we'll deadlock)
    > +
    > + This ensures that memory from the emergency reserve can be used to
    > + allow swapout to proceed, but will not get caught up in any other
    > + network queue.
    > +
    > +
    > +3/ pages_emergency
    > +
    ...
    > +
    > + So a new "watermark" is defined: pages_emergency. This is
    > + effectively added to the current low water marks, so that pages from
    > + this emergency pool can only be allocated if one of PF_MEMALLOC or
    > + GFP_MEMALLOC are set.

    is set.

    > +
    > + pages_emergency can be changed dynamically based on need. When
    > + swapout over the network is required, pages_emergency is increased
    > + to cover the maximum expected load. When network swapout is
    > + disabled, pages_emergency is decreased.
    > +
    > + To determine how much to increase it by, we introduce reservation
    > + groups....
    > +
    > +3a/ reservation groups
    > +
    > + The memory used transiently for swapout can be in a number of
    > + different places. e.g. the network route cache, the network

    places, e.g.,

    > + fragment cache, in transit between network card and socket, or (in
    > + the case of NFS) in sunrpc data structures awaiting a reply.
    > + We need to ensure each of these is limited in the amount of memory
    > + they use, and that the maximum is included in the reserve.
    > +

    ...

    > +
    > +4/ low-mem accounting
    > +
    > + Most places that might hold on to emergency memory (e.g. route
    > + cache, fragment cache etc) already place a limit on the amount of

    fragment cache, etc.)

    > + memory that they can use. This limit can simply be reserved using
    > + the above mechanism and no more needs to be done.
    > +
    > + However some memory usage might not be accounted with sufficient

    However,

    > + firmness to allow an appropriate emergency reservation. The
    > + in-flight skbs for incoming packets is on such example.

    one

    > +
    > + To support this, a low-overhead mechanism for accounting memory
    > + usage against the reserves is provided. This mechanism uses the
    > + same data structure that is used to store the emergency memory
    > + reservations through the addition of a 'usage' field.
    > +
    > + Before we attempt allocation from the memory reserves, we much check

    s/much/must/ ?

    > + if the resulting 'usage' is below the reservation. If so, we increase
    > + the usage and attempt the allocation (which should succeed). If
    > + the projected 'usage' exceeds the reservation we'll either fail the
    > + allocation, or wait for 'usage' to decrease enough so that it would
    > + succeed, depending on __GFP_WAIT.
    > +
    > + When memory that was allocated for that purpose is freed, the
    > + 'usage' field is checked again. If it is non-zero, then the size of
    > + the freed memory is subtracted from the usage, making sure the usage
    > + never becomes less than zero.
    > +
    > + This provides adequate accounting with minimal overheads when not in
    > + a low memory condition. When a low memory condition is encountered
    > + it does add the cost of a spin lock necessary to serialise updates
    > + to 'usage'.
    > +
    > +
    > +
    > +5/ swapon/swapoff/swap_out/swap_in
    > +
    > + So that a filesystem (e.g. NFS) can know when to set SK_MEMALLOC on
    > + any network socket that it uses, and can know when to account
    > + reserve memory carefully, new address_space_operations are
    > + available.
    > + "swapon" requests that an address space (i.e a file) be make ready

    (i.e.
    s/make/made/

    > + for swapout. swap_out and swap_in request the actual IO. They
    > + together must ensure that each swap_out request can succeed without
    > + allocating more emergency memory that was reserved by swapon. swapoff
    > + is used to reverse the state changes caused by swapon when we disable
    > + the swap file.
    > +
    > +
    > +Thanks for reading this far. I hope it made sense :-)
    > +
    > +Neil Brown (with updates from Peter Zijlstra)


    Thanks.

    ---
    ~Randy


    \
     
     \ /
      Last update: 2008-03-20 22:25    [W:0.045 / U:60.904 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site