[lkml]   [2010]   [Apr]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRE: Memory policy question for NUMA arch....
    On Fri, 2010-04-16 at 16:17 -0700, Chetan Loke wrote: 
    > Hello,
    > PS - Please 'CC' me on the emails.I have not subscribed to the list.
    > > Hi Andy,
    > >
    > > --- On Wed, 4/7/10, Andi Kleen <>
    > > wrote:
    > > > On Tue, Apr 06, 2010 at 01:46:44PM -0700, Rick Sherm
    > > wrote:
    > > > > On a NUMA host, if a driver calls
    > > __get_free_pages()
    > > > then
    > > > > it will eventually invoke
    > > > ->alloc_pages_current(..). The comment
    > > > > above/within alloc_pages_current() says
    > > > 'current->mempolicy' will be
    > > > > used.So what memory policy will kick-in if the
    > > driver
    > > > is trying to
    > > > > allocate some memory blocks during driver load
    > > > time(say from probe_one)? System-wide default
    > > > policy,correct?
    > > >
    > > > Actually the policy of the modprobe or the kernel boot
    > > up
    > > > if built in
    > > > (which is interleaving)
    > > >
    > I may be wrong but I think there's a difference. system-wide run-time default policy is M_PREFERRED | M_LOCAL and not Interleaving.
    > So, if current->mempolicy is set then default_policy will not be used.
    > And now if you don't want the default_policy mode then what?
    > I'm stuck in this confused state too. So we have two cases to take care off -
    > Case1) current->mempolicy is initialized and so we can just set it to
    > whatever we like and then reset it once we are done with
    > __get_free_pages(..) etc.

    Yes, as Andi mentioned. Also, see my response to Rick at:

    > Case2) current->mempolicy is not initialized. Then default_policy is
    > used. Now if we have to muck with the default_policy then we will need
    > to lock it down. Otherwise some other consumer will get affected by
    > it.

    If current->mempolicy is not initialized, you can create a new one and
    set it temporarily. You could probably call do_set_mempolicy() directly
    the way numa_policy_init() does and then call numa_default_policy() to
    restore it to default.

    You should never change the system default once the system is up and

    > But both the above solutions are twisted.Why not just create a
    > different wrapper? This way we can leave both current & default_policy
    > alone.
    > #ifdef CONFIG_NUMA
    > __get_free_policy_pages(policy,mask,order)??
    > endif

    As Andi mentioned in his response, you could certainly do this as long
    as it doesn't impact the normal allocation path.
    > For now I may end up hacking my kernel and implementing the above
    > mentioned quick and dirty solution. But if there's a cleaner approach
    > then please let me know.
    > PS - We should create some wrapper's that will automatically figure
    > out the MSIX-affinity(if present/set) and then default the allocation
    > to that node?

    Still not clear on what your requirements are but, if existing
    interfaces don't suffice, such a wrapper might make sense.
    __get_free_pages() is simply a wrapper around alloc_pages() that then
    returns page_address() of the resulting page. So, something like
    'get_free_pages_node()'--which should probably live in
    mm/page_alloc.c--would just be a wrapper around alloc_pages_node() that
    then returns the page_address() of the page.

    A device-centric interface--e.g., 'get_free_pages_dev()'--could get the
    device/bus node affinity via dev_to_node() and then do the
    allocation/conversion. I think this is close to what you're suggesting
    above. See dma_generic_alloc_coherent() [in arch/x86/kernel/pci-dma.c]
    for an example of a wrapper that does the device affinity lookup and
    allocation in one function.

    Of course, you could just do this in your driver, as well.

    > Also, is there a way to configure irqbalance and ask it to leave these
    > guys alone? Like a config file that says - leave these
    > irqs/pci-devices alone.For now I've shut down irqbalance.

    You can set the environment variable IRQBALANCE_BANNED_INTERRUPTS--when
    starting irqbalance--to list of interrupts that irqbalance should ignore
    if you're using a version that supports that. Check the init script
    that starts irqbalance on your distro of choice.


     \ /
      Last update: 2010-04-19 17:19    [W:0.024 / U:16.032 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site