lkml.org 
[lkml]   [2000]   [Mar]   [17]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Overcommitable memory??
On Thu, 16 Mar 2000, Paul Jakma wrote:

> On Thu, 16 Mar 2000, James Sutherland wrote:
>
> > On Wed, 15 Mar 2000, Paul Jakma wrote:
>
> > That doesn't happen. malloc() ALLOCATES the memory to the process. It is
> > *NOT* overcommitted. It may be backed by swapspace rather than physical
> > memory, but that block of memory *IS* available to the process.
>
> so malloc() isn't overcommited? malloc()'ed memory is guaranteed to be
> available - ie the memory is reserved and accounted for at malloc()
> time?

malloc() CAN be overcommitted. If you set a VM flag via /proc, then
malloc() will *ALWAYS* succeed, even if there isn't any memory available
at all. With the flag clear, malloc() does some sanity checking before
granting the memory.

You CAN obtain an overcommit free malloc() by clearing the VM flag (it is
clear by default), then touching every page you allocate when you allocate
it.

> (i always thought malloc()'s were noted by the kernel ie in the page
> table for the process, but that the kernel didn't actually allocate a
> real page until the process tried to access that page - ie malloc() is
> overcommited)

That's the usual system, yes :-(

> If so, how do we get into trouble?

Erm - all the memory gets allocated, so there isn't any left. Nothing to
do with overcommit, COW, or any other allocation strategy - we just don't
have enough.

> > malloc() does this. fork() doesn't, because there is no memory to
> > allocate.
>
> i must be misunderstanding you, cause I'm pretty sure fork() is supposed
> to copy the complete VM of the parent. :) And on older Unix's like
> ultrix you really had to have lot's of swap if you wanted to allow big
> processes to do a fork().

Yes. This would appear to be the system Rask is advocating - if my 20Mb
Apache fork()s 100 times, I need to have 2Gb of swap space for it.
Otherwise, it fails due to "lack" of memory - even though I could fit 100
000 of them in the space I have...

> > The whole point of fork() is that you are *NOT* simply
> > duplicating the in-VM image of the process!
>
> no, the whole point of overcommiting is so that we don't have to copy
> the VM of a process at fork() time, eg COW. But fork() does specify that
> the child process will have a complete copy of the parents VM.
>
> (whether we copy it at fork() time like really ancient os's, COW with
> reserved backing store like ultrix or COW overcommit like linux is an
> implementation issue).

Genuinely copying the entire VM space of the process at fork() time would
be unbelievably inefficient...

> > > but that only applies to processes that try malloc() at the point of
> > > OOM. You still have a bunch of processes with memory they have already
> > > malloc()'ed but havn't allocated yet.
> >
> > Nope.
> >
>
> really really sure about this? cause it goes completely against my
> understanding of how linux works. but i'd be very glad to be corrected
> if i'm wrong. I've honestly thought that malloc() was overcommited up
> till now.

It is, to a limited extent; you can avoid this entirely by simply touching
each page you allocate at allocation time.



James.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:57    [W:0.290 / U:0.280 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site