[lkml]   [2015]   [Apr]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: Interacting with coherent memory on external devices
On Tue, 21 Apr 2015, Paul E. McKenney wrote:

> Thoughts?

Use DAX for memory instead of the other approaches? That way it is
explicitly clear what information is put on the CAPI device.

> Although such a device will provide CPU's with cache-coherent

Maybe call this coprocessor like IBM does? It is like a processor after
all in terms of its participation in cache coherent?

> access to on-device memory, the resulting memory latency is
> expected to be slower than the normal memory that is tightly
> coupled to the CPUs. Nevertheless, data that is only occasionally
> accessed by CPUs should be stored in the device's memory.
> On the other hand, data that is accessed rarely by the device but
> frequently by the CPUs should be stored in normal system memory.

I would expect many devices to not have *normal memory* at all (those
that simply process some data or otherwise interface with external
hardware like f.e. a NIC). Other devices like GPUs have local memory but
what is in GPU memory is very specific and general OS structures should
not be allocated there.

What I mostly would like to see is that these devices will have the
ability to participate in the cpu cache coherency scheme. I.e. they
will have l1/l2/l3 caches that will allow fast data exchange between the
coprocessor and the regular processors in the system.

> a. It should be possible to migrate all data away
> from the device's memory at any time.

That would be device specific and only a special device driver for that
device could save the state of the device (if that is necessary. It would
not be for something like a NIC).

> b. Normal memory allocation should avoid using the
> device's memory, as this would interfere
> with the needed migration. It may nevertheless
> be desirable to use the device's memory
> if system memory is exhausted, however, in some
> cases, even this "emergency" use is best avoided.
> In fact, a good solution will provide some means
> for avoiding this for those cases where it is
> necessary to evacuate memory when offlining the
> device.

Ok that seems to mean that none of the approaches suggested later would
be useful.

> 3. The device's memory is treated like normal system
> memory by the Linux kernel, for example, each page has a
> "struct page" associate with it. (In contrast, the
> traditional approach has used special-purpose OS mechanisms
> to manage the device's memory, and this memory was treated
> as MMIO space by the kernel.)

Why do we need a struct page? If so then maybe equip DAX with a struct
page so that the contents of the device memory can be controlled via a
filesystem? (may be custom to the needs of the device).

 \ /
  Last update: 2015-04-22 02:01    [W:0.191 / U:4.464 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site