Messages in this thread | | | Date | Wed, 13 Feb 2008 17:01:03 -0700 | From | Jason Gunthorpe <> | Subject | Re: [ofa-general] Re: Demand paging for memory regions |
| |
On Wed, Feb 13, 2008 at 06:23:08PM -0500, Pete Wyckoff wrote: > christian.bell@qlogic.com wrote on Tue, 12 Feb 2008 20:09 -0800: > > One other area that has not been brought up yet (I think) is the > > applicability of notifiers in letting users know when pinned memory > > is reclaimed by the kernel. This is useful when a lower-level > > library employs lazy deregistration strategies on memory regions that > > are subsequently released to the kernel via the application's use of > > munmap or sbrk. Ohio Supercomputing Center has work in this area but > > a generalized approach in the kernel would certainly be welcome. > > The whole need for memory registration is a giant pain. There is no > motivating application need for it---it is simply a hack around > virtual memory and the lack of full VM support in current hardware. > There are real hardware issues that interact poorly with virtual > memory, as discussed previously in this thread.
Well, the registrations also exist to provide protection against rouge/faulty remotes, but for the purposes of MPI that is probably not important.
Here is a thought.. Some RDMA hardware can change the page tables on the fly. What if the kernel had a mechanism to dynamically maintain a full registration of the processes entire address space ('mlocked' but able to be migrated)? MPI would never need to register a buffer, and all the messy cases with munmap/sbrk/etc go away - the risk is that other MPI nodes can randomly scribble all over the process :)
Christoph: It seemed to me you were first talking about freeing/swapping/faulting RDMA'able pages - but would pure migration as a special hardware supported case be useful like Catilan suggested?
Regards, Jason
| |