lkml.org 
[lkml]   [2000]   [Mar]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRE: Overcommitable memory??
On Tue, 21 Mar 2000, David Schwartz wrote:

> > We are not short of memory. We're short of swap. paging-out is
> > the problem,
> > not paging-in.
>
> This is why it's important to let the applications know about a potential
> resource problem early enough. If you let them know too late, they may not
> be able to do a thing about it without worsening it.
>
> Most of my applications can do lots of things if resource consumption is a
> problem. For example, they can tell the load manager to divert connections
> to other machines and let the connections to it close by themselves. Some of
> them can free cached data. They can defer memory-intensive operations for a
> later time. They can refuse 'expensive' commands from clients.

Handling faults instead of being victim of them is much a better approach,
i agree.

> Well, I can tell you what my apps do. They have two memory allocation

[...]
> If the hard allocator gets a NULL from malloc, it frees a portion of its
^^^^^^^^^^^^^^^^^^^^^
see (1) below.

> Sophisticated applications need the tools to allow them to deal with memory
> (or memory+swap) exhaustion. A side issue is that applications can't tell
> when they're hitting swap either, and so can't take steps to reduce memory
> consumption to improve their own (and system) performance.

Exactly. malloc() is definitely NOT the tool a sophisticated application
should use.

Libc just provides you a general-purpose memory allocator, which in turns
relays on a general-porpose kernel mechanism. If you have "sophisticated"
needs in allocating memory, do it another way. Think of the buffer cache.
It's just a general purpose, most of the time effective, transparent
buffering/caching system. Sophisticated applications which have a naive
use of the FS (think of gcc) take full advantage of it; but applications
that want a more complex and fine-grained control over disk storage may
want to bypass it (think of a database system).

If your application is "sophisticated" in memory allocation, probably it
does not like paging in the first place. It can do a better job in handling
memory (i mean RAM) shortage than the kernel. Or it simply does not want
its pages being paged out (if you're doing double-buffering towards a
fast device, you don't want your buffers being swapped to disk).

Sophisticated kernel interfaces for memory allocation/management do exists.
Just using brk() and relay on kernel paging (COW, swap, and so on) to
magage your address space is just not being sophisticated.
If you like the name, write your own malloc(). Use brk() or mmap()
directly, and mlock() in order to get real RAM. If it fails, or if it
hits some limit you choose, have a file handy where you swap-out part
of your in-core data, and reuse the memory you freed. Not easy, but
not impossible. And potentially much more effective than the standard kernel
paging (you *know* what you need, the kernel can only make a wild guess).

[ Whether these "sophisticated kernel interfaces" can be improved, it a
completely different matter (and thread). ]

>
> DS
>

(1) BTW, just wondering, if you have a hard allocator and an emergency pool,
why you're freeing the pool and re-mallocing it? Overwrite it. B-)

.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:57    [W:1.871 / U:0.248 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site