Messages in this thread | | | Date | Wed, 22 Mar 2000 11:01:36 +0100 (CET) | From | Marco Colombo <> | Subject | RE: Overcommitable memory?? |
| |
On Tue, 21 Mar 2000, David Schwartz wrote:
> > We are not short of memory. We're short of swap. paging-out is > > the problem, > > not paging-in. > > This is why it's important to let the applications know about a potential > resource problem early enough. If you let them know too late, they may not > be able to do a thing about it without worsening it. > > Most of my applications can do lots of things if resource consumption is a > problem. For example, they can tell the load manager to divert connections > to other machines and let the connections to it close by themselves. Some of > them can free cached data. They can defer memory-intensive operations for a > later time. They can refuse 'expensive' commands from clients.
Handling faults instead of being victim of them is much a better approach, i agree.
> Well, I can tell you what my apps do. They have two memory allocation
[...] > If the hard allocator gets a NULL from malloc, it frees a portion of its ^^^^^^^^^^^^^^^^^^^^^ see (1) below.
> Sophisticated applications need the tools to allow them to deal with memory > (or memory+swap) exhaustion. A side issue is that applications can't tell > when they're hitting swap either, and so can't take steps to reduce memory > consumption to improve their own (and system) performance.
Exactly. malloc() is definitely NOT the tool a sophisticated application should use.
Libc just provides you a general-purpose memory allocator, which in turns relays on a general-porpose kernel mechanism. If you have "sophisticated" needs in allocating memory, do it another way. Think of the buffer cache. It's just a general purpose, most of the time effective, transparent buffering/caching system. Sophisticated applications which have a naive use of the FS (think of gcc) take full advantage of it; but applications that want a more complex and fine-grained control over disk storage may want to bypass it (think of a database system).
If your application is "sophisticated" in memory allocation, probably it does not like paging in the first place. It can do a better job in handling memory (i mean RAM) shortage than the kernel. Or it simply does not want its pages being paged out (if you're doing double-buffering towards a fast device, you don't want your buffers being swapped to disk).
Sophisticated kernel interfaces for memory allocation/management do exists. Just using brk() and relay on kernel paging (COW, swap, and so on) to magage your address space is just not being sophisticated. If you like the name, write your own malloc(). Use brk() or mmap() directly, and mlock() in order to get real RAM. If it fails, or if it hits some limit you choose, have a file handy where you swap-out part of your in-core data, and reuse the memory you freed. Not easy, but not impossible. And potentially much more effective than the standard kernel paging (you *know* what you need, the kernel can only make a wild guess).
[ Whether these "sophisticated kernel interfaces" can be improved, it a completely different matter (and thread). ]
> > DS >
(1) BTW, just wondering, if you have a hard allocator and an emergency pool, why you're freeing the pool and re-mallocing it? Overwrite it. B-)
.TM. -- ____/ ____/ / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _____/ _____/ _/ Colombo@ESI.it
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |