[lkml]   [2016]   [Oct]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: [RFC 3/8] mm: Isolate coherent device memory nodes from HugeTLB allocation paths

On 25/10/16 15:15, Aneesh Kumar K.V wrote:
> Dave Hansen <> writes:
>> On 10/23/2016 09:31 PM, Anshuman Khandual wrote:
>>> This change is part of the isolation requiring coherent device memory nodes
>>> implementation.
>>> Isolation seeking coherent device memory node requires allocation isolation
>>> from implicit memory allocations from user space. Towards that effect, the
>>> memory should not be used for generic HugeTLB page pool allocations. This
>>> modifies relevant functions to skip all coherent memory nodes present on
>>> the system during allocation, freeing and auditing for HugeTLB pages.
>> This seems really fragile. You had to hit, what, 18 call sites? What
>> are the odds that this is going to stay working?
> I guess a better approach is to introduce new node_states entry such
> that we have one that excludes coherent device memory numa nodes. One
> possibility is to add N_SYSTEM_MEMORY and N_MEMORY.
> Current N_MEMORY becomes N_SYSTEM_MEMORY and N_MEMORY includes
> system and device/any other memory which is coherent.

I thought of this as well, but I would rather see N_COHERENT_MEMORY
as a flag. The idea being that some device memory is a part of
N_MEMORY, but N_COHERENT_MEMORY gives it additional attributes

> All the isolation can then be achieved based on the nodemask_t used for
> allocation. So for allocations we want to avoid from coherent device we
> use N_SYSTEM_MEMORY mask or a derivative of that and where we are ok to
> allocate from CDM with fallbacks we use N_MEMORY.

I suspect its going to be easier to exclude N_COHERENT_MEMORY.

> All nodes zonelist will have zones from the coherent device nodes but we
> will not end up allocating from coherent device node zone due to the
> node mask used.
> This will also make sure we end up allocating from the correct coherent
> device numa node in the presence of multiple of them based on the
> distance of the coherent device node from the current executing numa
> node.

The idea is good overall, but I think its going to be good to document
the exclusions with the flags

Balbir Singh.

 \ /
  Last update: 2016-10-25 09:18    [W:0.094 / U:7.348 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site