[lkml]   [2000]   [Mar]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: Endless overcommit memory thread.
Hi Mike.

> I've been reading bits and pieces of this whole thread for
> a while now, and have finally come up with an opinion. I
> think that overcommit should be an option which defaults to
> off. Here's why:

> If something allocates memory and then later goes to use it
> and dies, that is just plain wrong. If malloc says the
> memory is avail, and allocates it, then it should be there.

> Define "allocate". I define it with respect to memory
> allocation as an app being allocated or GIVEN a requested
> amount of memory to use, and guaranteed that that memory
> exists and is available for usage.

> The reason for overcommit in the first place, and correct
> me if I'm wrong here - is because some applications
> allocate more memory than they really end up using.

Not completely, no - at least, not if I've followed the arguments
presented to date.

The basic problem as I understand it is not the memory allocated
by applications, but the memory allocated by the kernel AFTER the
various applications have grabbed theirs. In particular, this is
memory used for the buffer cache, the networking stack, the
system stack, and the like.

As I understand the arguments presented so far, they can be
summarised like this:

1. The anti-overcommit brigade:

No matter how much memory is used by applications when the
kernel is running, if the kernel requires more memory for
some essential task (such as the system stack) than is
available, we let the system crash by denying that memory
to the kernel.

2. The pro-overcommit brigade:

No matter whether a program really needs the 300M of memory
it's just asked for, we will tell the program we can give
it that memory, and then have the program crash when it
validly assumes we've told the truth.

Personally, I can't see any validity in either position as so far
expressed, so I'm waiting for one side to come up with some sort
of explanation of their position that actually makes sense.

> If an application allocates a large amount of memory and
> then does not use it, is the app not broken as designed?

There are some programming tasks where it is impossible to
predict in advance just how much memory one will require, and
where a requirement for a large memory buffer is rather more
common than for a small one.

> Can someone please explain to me how and why an app would
> allocate a large amount of memory and then not use it?
> Please give specific examples since I can't fathom the
> reason.

OK, here's one application that COMMONLY requires large amounts
of memory, but under certain rare situations needs only a small
amount of memory: Capturing video direct from TV to disk when one
can't store it in the format it arrives in, for whatever reason.

If the video one happens to be storing consists of lengthy
segments of virtually identical frames, very little memory will
be used to capture them to disk. However, in the far more common
case that consecutive frames differ considerably from each other,
it is common for one to have to buffer several megabytes of video
to allow the converter to keep up with the stream of video.

Since the video stream is basically real-time data that does not
repeat, the application HAS to decide how big a buffer to set up
prior to the first frame arriving - the conversion routine is
VERY time critical, and having to break out to request more ram
will almost inevitably mean that frames get lost.

> I think an app that requests a lot of memory and then
> doesn't use it is BAD - broken as designed.

With some applications, that can be a pre-requisite for the
application to actually work in the first place.

> I can see legitimate allocation of SMALL amounts of unused
> memory as speculation in time critical code, or for
> optimization, etc.. but not large amounts.

I have had reason to write a program that started off by
allocating the biggest buffer it can on the machine in question,
simply so that it stood a chance of actually doing the job it was
supposed to do. Its startup procedure literally consisted of a
loop that started by allocating 512 Megs of ram and halving the
allocation until it got one that succeeded. It then released
that, then tried allocating 50% more than the success level, and
used the largest of those two that actually succeeded.

Whilst I can't go into details, I can say that the application
involved capturing datasets from several hundred sensors several
thousand times a second, processing it to remove redundant data,
compressing the result, and saving that out to disk, and it was
important that not a single reading be missed over a period of

> Please correct me if I'm wrong. Either way it should be
> tuneable and disableable.

I hope these comments have helped.

> I felt that since I had to download 20Mb of this thread
> this week that I'd throw in my $0.02 as well. ;o)

I know the feeling...

Best wishes from Riley.

* Copyright (C) 2000, Memory Alpha Systems.
* All rights and wrongs reserved.

| There is something frustrating about the quality and speed of Linux |
| development, ie., the quality is too high and the speed is too high, |
| in other words, I can implement this XXXX feature, but I bet someone |
| else has already done so and is just about to release their patch. |

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
Please read the FAQ at

 \ /
  Last update: 2005-03-22 13:57    [W:0.116 / U:6.704 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site