lkml.org 
[lkml]   [2013]   [May]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 09/22] mm: page allocator: Allocate/free order-0 pages from a per-zone magazine
    On Wed, May 08, 2013 at 06:41:58PM +0000, Christoph Lameter wrote:
    > On Wed, 8 May 2013, Mel Gorman wrote:
    >
    > > 1. IRQs do not have to be disabled to access the lists reducing IRQs
    > > disabled times.
    >
    > The per cpu structure access also would not need to disable irq if the
    > fast path would be using this_cpu ops.
    >

    How does this_cpu protect against preemption due to interrupt? this_read()
    itself only disables preemption and it's explicitly documented that
    interrupt that modifies the per-cpu data will not be reliable so the use
    of the per-cpu lists is right out. It would require that a race-prone
    check be used with cmpxchg which in turn would require arrays, not lists.

    > > 2. As the list is protected by a spinlock, it is not necessary to
    > > send IPI to drain the list. As the lists are accessible by multiple CPUs,
    > > it is easier to tune.
    >
    > The lists are a problem since traversing list heads creates a lot of
    > pressure on the processor and TLB caches. Could we either move to an array
    > of pointers to page structs (like in SLAB)

    They would have large memory requirements. The magazine data structure in
    this series fits in a cache line. An array of 128 struct page pointers
    would require 1K on 64-bit and if that thing is per possible CPU and per
    zone then it could get really excessive.

    > or to a linked list that is
    > constrained within physical boundaries like within a PMD? (comparable
    > to the SLUB approach)?
    >

    I don't see how as the page allocator does not control the physical location
    of any pages freed to it and it's the struct pages it is linking together. On
    some systems at least with 1G pages, the struct pages will be backed by
    memory mapped with 1G entries so the TLB pressure should be reduced but
    the cache pressure from struct page modifications is certainly a problem.

    > > > 3. The magazine_lock is potentially hot but it can be split to have
    > > one lock per CPU socket to reduce contention. Draining the lists
    > > in this case would acquire multiple locks be acquired.
    >
    > IMHO the use of per cpu RMV operations would be lower latency than the use
    > of spinlocks. There is no "lock" prefix overhead with those. Page
    > allocation is a frequent operation that I would think needs to be as fast
    > as possible.

    The memory requirements may be large because those per-cpu areas sized are
    allocated depending on num_possible_cpus()s. Correct? Regardless of their
    size, it would still be required to deal with cpu hot-plug to avoid memory
    leaks and draining them would still require global IPIs so the overall
    code complexity would be similar to what exists today. Ultimately all that
    changes is that we use an array+cmpxchg instead of a list which will shave
    a small amount of latency but it will still be regularly falling back to
    the buddy lists and contend on the zone->lock due the limited size of the
    per-cpu magazines and hiding the advantage of using cmpxchg in the noise.

    --
    Mel Gorman
    SUSE Labs


    \
     
     \ /
      Last update: 2013-05-09 18:01    [W:4.061 / U:1.916 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site