Messages in this thread | | | Date | Fri, 24 Mar 2000 23:59:13 +0000 (GMT) | From | Riley Williams <> | Subject | Re: Endless overcommit memory thread. |
| |
Hi Mike.
> I've been reading bits and pieces of this whole thread for > a while now, and have finally come up with an opinion. I > think that overcommit should be an option which defaults to > off. Here's why:
> If something allocates memory and then later goes to use it > and dies, that is just plain wrong. If malloc says the > memory is avail, and allocates it, then it should be there.
> Define "allocate". I define it with respect to memory > allocation as an app being allocated or GIVEN a requested > amount of memory to use, and guaranteed that that memory > exists and is available for usage.
> The reason for overcommit in the first place, and correct > me if I'm wrong here - is because some applications > allocate more memory than they really end up using.
Not completely, no - at least, not if I've followed the arguments presented to date.
The basic problem as I understand it is not the memory allocated by applications, but the memory allocated by the kernel AFTER the various applications have grabbed theirs. In particular, this is memory used for the buffer cache, the networking stack, the system stack, and the like.
As I understand the arguments presented so far, they can be summarised like this:
1. The anti-overcommit brigade:
No matter how much memory is used by applications when the kernel is running, if the kernel requires more memory for some essential task (such as the system stack) than is available, we let the system crash by denying that memory to the kernel.
2. The pro-overcommit brigade:
No matter whether a program really needs the 300M of memory it's just asked for, we will tell the program we can give it that memory, and then have the program crash when it validly assumes we've told the truth.
Personally, I can't see any validity in either position as so far expressed, so I'm waiting for one side to come up with some sort of explanation of their position that actually makes sense.
> If an application allocates a large amount of memory and > then does not use it, is the app not broken as designed? > I say FIX THE APPLICATION!
There are some programming tasks where it is impossible to predict in advance just how much memory one will require, and where a requirement for a large memory buffer is rather more common than for a small one.
> Can someone please explain to me how and why an app would > allocate a large amount of memory and then not use it? > Please give specific examples since I can't fathom the > reason.
OK, here's one application that COMMONLY requires large amounts of memory, but under certain rare situations needs only a small amount of memory: Capturing video direct from TV to disk when one can't store it in the format it arrives in, for whatever reason.
If the video one happens to be storing consists of lengthy segments of virtually identical frames, very little memory will be used to capture them to disk. However, in the far more common case that consecutive frames differ considerably from each other, it is common for one to have to buffer several megabytes of video to allow the converter to keep up with the stream of video.
Since the video stream is basically real-time data that does not repeat, the application HAS to decide how big a buffer to set up prior to the first frame arriving - the conversion routine is VERY time critical, and having to break out to request more ram will almost inevitably mean that frames get lost.
> I think an app that requests a lot of memory and then > doesn't use it is BAD - broken as designed.
With some applications, that can be a pre-requisite for the application to actually work in the first place.
> I can see legitimate allocation of SMALL amounts of unused > memory as speculation in time critical code, or for > optimization, etc.. but not large amounts.
I have had reason to write a program that started off by allocating the biggest buffer it can on the machine in question, simply so that it stood a chance of actually doing the job it was supposed to do. Its startup procedure literally consisted of a loop that started by allocating 512 Megs of ram and halving the allocation until it got one that succeeded. It then released that, then tried allocating 50% more than the success level, and used the largest of those two that actually succeeded.
Whilst I can't go into details, I can say that the application involved capturing datasets from several hundred sensors several thousand times a second, processing it to remove redundant data, compressing the result, and saving that out to disk, and it was important that not a single reading be missed over a period of days...
> Please correct me if I'm wrong. Either way it should be > tuneable and disableable.
I hope these comments have helped.
> I felt that since I had to download 20Mb of this thread > this week that I'd throw in my $0.02 as well. ;o)
I know the feeling...
Best wishes from Riley.
* Copyright (C) 2000, Memory Alpha Systems. * All rights and wrongs reserved.
+----------------------------------------------------------------------+ | There is something frustrating about the quality and speed of Linux | | development, ie., the quality is too high and the speed is too high, | | in other words, I can implement this XXXX feature, but I bet someone | | else has already done so and is just about to release their patch. | +----------------------------------------------------------------------+ * http://www.memalpha.cx/Linux/Kernel/
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |