lkml.org 
[lkml]   [2013]   [Jun]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    SubjectHandling NUMA page migration
    Date
    Hi,

    our memory management on Linux hosts conflicts with NUMA page migration.
    I assume this problem existed for a longer time but Linux 3.8 introduced
    automatic NUMA page balancing which makes the problem visible on
    multi-node hosts leading to kernel oopses.

    NUMA page migration means that the physical address of a page changes.
    This is fatal if the application assumes that this never happens for
    that page as it was supposed to be pinned.

    We have two kind of pinned memory:

    A) 1. allocate memory in userland with mmap()
    2. madvise(MADV_DONTFORK)
    3. pin with get_user_pages().
    4. flush dcache_page()
    5. vm_flags |= (VM_DONTCOPY | VM_LOCKED)
    (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND |
    VM_DONTCOPY | VM_LOCKED | 0xff)

    B) 1. allocate memory with alloc_pages()
    2. SetPageReserved()
    3. vm_mmap() to allocate a userspace mapping
    4. vm_insert_page()
    5. vm_flags |= (VM_DONTEXPAND | VM_DONTDUMP)
    (resulting flags are VM_MIXEDMAP | VM_DONTDUMP | VM_DONTEXPAND | 0xff)

    At least the memory allocated like B) is affected by automatic NUMA page
    migration. I'm not sure about A).

    1. How can I prevent automatic NUMA page migration on this memory?
    2. Can NUMA page migration also be handled on such kind of memory without
    preventing migration?

    Thanks,

    Frank
    --
    Dr.-Ing. Frank Mehnert | Software Development Director, VirtualBox
    ORACLE Deutschland B.V. & Co. KG | Werkstr. 24 | 71384 Weinstadt, Germany

    Hauptverwaltung: Riesstr. 25, D-80992 München
    Registergericht: Amtsgericht München, HRA 95603
    Geschäftsführer: Jürgen Kunz

    Komplementärin: ORACLE Deutschland Verwaltung B.V.
    Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
    Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
    Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2013-06-04 10:21    [W:3.034 / U:2.248 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site