lkml.org 
[lkml]   [2000]   [Mar]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Avoiding OOM on overcommit...?
    On 18 Mar 2000, Mirian Crzig Lennox wrote:

    > In article <Pine.BSF.4.05.10003171855400.9271-100000@bsd1.nyct.net>,
    > Michael Bacarella <mbac@nyct.net> wrote:
    > >
    > >No overcommit of VM. Yes! This would be HORRIBLE for performance, but ONLY
    > >if you're actually allocating the memory to the process. If you don't
    > >actually allocate it, but in fact just ensure that it could allocate it
    > >in the future, you take no performance hit and gain that much stability.
    >
    > I agree with you Michael. This would make it possible to write
    > applications which are robust about memory allocation, since they will
    > know at malloc time whether they have the memory they requested and can do
    > intelligent things (e.g., purge buffers, alert the user, wait, etc.)
    > beyond merely catching SIGTERM and exiting gracefully.
    >
    > Overcommitting memory is the moral equivalent of writing bad checks and
    > praying there will be money to cover them before they are cashed. It's
    > completely irresponsible-- and when it fails, it really bites down hard.

    I must be missing something. How can you compare the two things?
    brk() is just extending your virtual address space. Your VA space
    has little to do with RAM or swap, just because is *virtual*. It just
    means that 0xe0000000 is not a *real* address, just a translated ones...
    As someone already pointed out, i may need it just to implement some
    sparse algorithm without having to handle a translation table and
    a cache myself. The kernel will give me RAM when i access it, and will
    reclaim it if it needs it back (copying my valuable data to disk, so i
    can get it back later). brk() does not grant you any *storage*, nor in RAM
    neither on disk. If you want to be sure to get real RAM, use another
    system call. You should lock pages into memory. If you need disk space,
    the FS is there for you.

    Kernel paging of process VM is there as a *general* purpose mean to make
    more efficent use of a limited resource. Being it *general*, yes you
    can make examples in which it fails. But it works well most of the time.
    OOM should not be a normal condition on your system. Nor should be
    the presence of one (or more) malicious user. Taking care of (or avoiding)
    them is not a kernel issue. It's an administrator task (and yes, you
    need a system administrator).

    Your critical application should not relay on general purpose mechanisms
    provided by the kernel. If your simulation needs 2GB of data, but uses
    only a small part of it as working set, and therefore can run in just 64MB
    of RAM, it should handle its backing storage directly, using mmap() or
    just plain lseek() & write(). This has been quite clear to database
    engines designers, e.g.

    On the other side, if you have a loaded web server, you simply don't
    need swap. You should limit the size of the whole web server (multi or
    single process it doesn't matter) so that it almost never needs to
    swap-in. Swap is there just for a few most-of-the-time-useless processes
    which, only sometimes, are needed (on a web server). On a web server
    you hit RAM OOM (the system is paging server data in and out) much
    before the RAM+swap OOM, unless you have very little swap.

    > >While, yes, a 40 meg process that fork()s will have to make sure that
    > >there are 40 megs of VM free, it doesn't mean that COW cannot be used,
    > >and it does not mean all 40 megs will have to be paged in.
    >
    > I know I'm going to get pelted with rotten tomatoes for saying so, but
    > if you're talking about fork+exec, vfork() was an obscenity but it did
    > handle the trivial case quite well.

    The only desiderable behaviour of vfork() is a hint to the kernel:
    i'll make no use on any new resource, unless after an exec(). The
    problem is that you may need some resources, and using your parent's ones
    is not always fair or possible.

    Anyway, fork() implementation is not the only reason for overcommitting.

    Being able to use large address spaces has nothing to do with large
    memory allocation. The same is true for files. Being able to use large
    file offsets has nothing to do with large files (= with many blocks
    allocated to). Just try to write 1 block at offset 1GB on an otherwise
    empty file. ls will shown you a 1GB file "size", while du will show 1 block
    "usage". No one call this "overcommitting". The kernel has made no promises
    that you'll be able to fill the whole file with 1GB of data just because
    you succeded to write 1KB at 1GB offset. The same is true for malloc().
    You may malloc()ate 1GB of VA space, successfully write 4 bytes at
    (1Gb - 4) offset, still no promises are made that the space in between
    is avalable to you in any way.

    The fact that in legacy UNIX implementations the kernel preallocated
    enough swap space for the whole VA space of each process, well,
    I've always seen that as a limit not a feature. The so called
    "overcommitting" greatly increases system throughput, on the average
    case. Yes, it loses something in the worst case, but we're not going
    back to the trivial sort algorithms for the qsort worse case (in general,
    of course: sometimes bubblesort is just what you're looking for).

    And if you feel you are in some "special" case, use "special" tecniques,
    not just plain-old-malloc().

    [...]
    >
    > --Mirian

    .TM.
    --
    ____/ ____/ /
    / / / Marco Colombo
    ___/ ___ / / Technical Manager
    / / / ESI s.r.l.
    _____/ _____/ _/ Colombo@ESI.it


    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.rutgers.edu
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2005-03-22 13:57    [W:4.345 / U:0.008 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site