lkml.org 
[lkml]   [2000]   [Mar]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Endless overcommit memory thread.
    Hi Mike.

    > I've been reading bits and pieces of this whole thread for
    > a while now, and have finally come up with an opinion. I
    > think that overcommit should be an option which defaults to
    > off. Here's why:

    > If something allocates memory and then later goes to use it
    > and dies, that is just plain wrong. If malloc says the
    > memory is avail, and allocates it, then it should be there.

    > Define "allocate". I define it with respect to memory
    > allocation as an app being allocated or GIVEN a requested
    > amount of memory to use, and guaranteed that that memory
    > exists and is available for usage.

    > The reason for overcommit in the first place, and correct
    > me if I'm wrong here - is because some applications
    > allocate more memory than they really end up using.

    Not completely, no - at least, not if I've followed the arguments
    presented to date.

    The basic problem as I understand it is not the memory allocated
    by applications, but the memory allocated by the kernel AFTER the
    various applications have grabbed theirs. In particular, this is
    memory used for the buffer cache, the networking stack, the
    system stack, and the like.

    As I understand the arguments presented so far, they can be
    summarised like this:

    1. The anti-overcommit brigade:

    No matter how much memory is used by applications when the
    kernel is running, if the kernel requires more memory for
    some essential task (such as the system stack) than is
    available, we let the system crash by denying that memory
    to the kernel.

    2. The pro-overcommit brigade:

    No matter whether a program really needs the 300M of memory
    it's just asked for, we will tell the program we can give
    it that memory, and then have the program crash when it
    validly assumes we've told the truth.

    Personally, I can't see any validity in either position as so far
    expressed, so I'm waiting for one side to come up with some sort
    of explanation of their position that actually makes sense.

    > If an application allocates a large amount of memory and
    > then does not use it, is the app not broken as designed?
    > I say FIX THE APPLICATION!

    There are some programming tasks where it is impossible to
    predict in advance just how much memory one will require, and
    where a requirement for a large memory buffer is rather more
    common than for a small one.

    > Can someone please explain to me how and why an app would
    > allocate a large amount of memory and then not use it?
    > Please give specific examples since I can't fathom the
    > reason.

    OK, here's one application that COMMONLY requires large amounts
    of memory, but under certain rare situations needs only a small
    amount of memory: Capturing video direct from TV to disk when one
    can't store it in the format it arrives in, for whatever reason.

    If the video one happens to be storing consists of lengthy
    segments of virtually identical frames, very little memory will
    be used to capture them to disk. However, in the far more common
    case that consecutive frames differ considerably from each other,
    it is common for one to have to buffer several megabytes of video
    to allow the converter to keep up with the stream of video.

    Since the video stream is basically real-time data that does not
    repeat, the application HAS to decide how big a buffer to set up
    prior to the first frame arriving - the conversion routine is
    VERY time critical, and having to break out to request more ram
    will almost inevitably mean that frames get lost.

    > I think an app that requests a lot of memory and then
    > doesn't use it is BAD - broken as designed.

    With some applications, that can be a pre-requisite for the
    application to actually work in the first place.

    > I can see legitimate allocation of SMALL amounts of unused
    > memory as speculation in time critical code, or for
    > optimization, etc.. but not large amounts.

    I have had reason to write a program that started off by
    allocating the biggest buffer it can on the machine in question,
    simply so that it stood a chance of actually doing the job it was
    supposed to do. Its startup procedure literally consisted of a
    loop that started by allocating 512 Megs of ram and halving the
    allocation until it got one that succeeded. It then released
    that, then tried allocating 50% more than the success level, and
    used the largest of those two that actually succeeded.

    Whilst I can't go into details, I can say that the application
    involved capturing datasets from several hundred sensors several
    thousand times a second, processing it to remove redundant data,
    compressing the result, and saving that out to disk, and it was
    important that not a single reading be missed over a period of
    days...

    > Please correct me if I'm wrong. Either way it should be
    > tuneable and disableable.

    I hope these comments have helped.

    > I felt that since I had to download 20Mb of this thread
    > this week that I'd throw in my $0.02 as well. ;o)

    I know the feeling...

    Best wishes from Riley.

    * Copyright (C) 2000, Memory Alpha Systems.
    * All rights and wrongs reserved.

    +----------------------------------------------------------------------+
    | There is something frustrating about the quality and speed of Linux |
    | development, ie., the quality is too high and the speed is too high, |
    | in other words, I can implement this XXXX feature, but I bet someone |
    | else has already done so and is just about to release their patch. |
    +----------------------------------------------------------------------+
    * http://www.memalpha.cx/Linux/Kernel/


    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.rutgers.edu
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2005-03-22 13:57    [W:4.161 / U:0.084 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site