Messages in this thread | | | Date | Tue, 21 Apr 2009 23:55:55 +0530 | From | Balbir Singh <> | Subject | Re: Large Pages - Linux Foundation HPC |
| |
[Fix my email address to balbir@linux.vnet.ibm.com]
* Dave Hansen <dave@linux.vnet.ibm.com> [2009-04-21 09:57:05]: > On Tue, 2009-04-21 at 09:32 -0700, Badari Pulavarty wrote: > > Hi Dave, > > > > On the Linux foundation HPC track summary, I saw: > > > > -- Memory and interface to it - mapping memory into apps > > - large pages important - current state not good enough > > I'm not sure exactly what this means. But, there was continuing concern > about large page interfaces. hugetlbfs is fine, but it still requires > special tools, planning, and requires some modification of the app. We > can modify it with linker tricks or with LD_PRELOAD, but those certainly > don't work everywhere. I was told over and over again that hugetlbfs > isn't a sufficient interface for large pages, no matter how much > userspace we try to stick in front of it. > > Some of their apps get a 6-7x speedup from large pages! > > Fragmentation also isn't an issue for a big chunk of the users since > they reboot between each job. > > > nodes going down due to memory exhaustion > > Virtually all the apps in an HPC environment start up try to use all the > memory they can get their hands on. With strict overcommit on, that > probably means brk() or mmap() until they fail. They also usually > mlock() anything they're able to allocate. Swapping is the devil to > them. :) > > Basically, what all the apps do is a recipe for stressing the VM and > triggering the OOM killer. Most of the users simply hack the kernel and > replace the OOM killer with one that fits their needs. Some have an > attitude that "the user's app should never die" and others "the user > caused this, so kill their app". Basically, there's no way to make > everyone happy since they have conflicting requirements. But, this is > true of the kernel in general... nothing special here.
OOM killer has been a hot topic. Have you seen Dan Malek's patches at http://lkml.org/lkml/2009/4/13/276.
> > The split LRU should help things. It will at least make our memory > scanning more efficient and ensure we're making more efficient reclaim > progress. I'm not sure that anyone there knew about the oom_adjust and > oom_score knobs in /proc. They do now. :)
:-)
> > One of my suggestions was to use the memory resource controller. They > could give each app 95% (or whatever) of the system. This should let > them keep their current "consume all memory" behavior, but stop them at > sane limits. >
Soft limits should help as well, basically we are trying to allow unrestricted memory access until there is contention. The patches are still under development.
> That leads into another issue, which is the "wedding cake" software > stack. There are a lot of software dependencies both in and out of the > kernel. It is hard to change individual components, especially in the > lower levels. This leads many of the users to use old (think 2.6.9) > kernels. Nobody runs mainline, of course. > > Then, there's Lustre. Everybody uses it, it's definitely a big hunk of > the "wedding cake". I haven't seen any LKML postings on it in years and > I really wonder how it interacts with the VM. No idea. > > There's a "Hyperion cluster" which is for testing new HPC software on a > decently sized cluster. One suggestion of ours was to try and get > mainline tested on this every so often to look for regressions since > we're not able to glean feedback from 2.6.9 kernel users. We'll see > where that goes. > > > checkpoint/restart > > Many of the MPI implementations have mechanisms in userspace for > checkpointing of user jobs. Most cluster administrators instruct their > users to use these mechanisms. Some do. Most don't. >
Good inputs and summary. Thanks!
-- Balbir
| |