lkml.org 
[lkml]   [2000]   [Feb]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: maximum memory limit
Sorry for taking too much of your time on this, but I really appreciate your
help.

PS: If this is not a discussion for the kernel mailing list, please let me
know and I will take it off line :)

> >lots of trivial-sized allocations are a really bad idea in any case.
> >doing it in 1M chunks will give you more like 3G.
>
> But what if I have an application that doesent know how much memory it
will
> need?
> then allocating lots is a horrendous design flaw.

> My application is reading in a file, parsing it and constructing a
> huge in-memory database.

why not just mmap the file?
or at least malloc big chunks, which is an obviously smart thing
to do in the first place.

** I cant do this since the format of the file is different from the
structure I need to put in memory... This is a flat file that I need to put
into a tree like datastructure. I need to add one node per entry in the
file at a time. So the algorithm blindly mallocs for every record it finds.
To understand you correctly, you are suggesting that I have my own memory
management code that mallocs from the OS even bigger chunks and gives
smaller chunks to the tree algorithm, correct? In any event, its clear you
are suggesting I go away from malloc and use my own memory management code
+ mmap. Correct?

>> Should I write my own library that will first mmap
>> a large (say 2 GB) memory and then dish out smaller chunks to the
>> application from there on an as need babsis?

> sure. though 2G makes no sense, since malloc is page-granular,
> and you'll see no advantage over sensible chunk sizes (say, a > few MB
> at a time.)

> This is what malloc should
> really do.

> no, it's not. malloc is not really intended to be used for this
> kind of long-term allocation. sure, it usually works, but it can't
> please all possible uses, and the most time critical uses are small
> blocks that get used and freed, not large chunks that get used throughout
the
> life of the app.

>> What is the general approach taken for these kind of
>> applications?

> malloc your own big chunks. or better yet, mmap the database.
>all serious DB's do the latter, afaik.

> > the issue is
> > actually that the kernel normally permits 3G of VM to each task,
> > but starts mmaped allocations at the 1G mark. that limits the sbrk
>
> It seems complicated for an application that wants to use a large amount
of
> memory to know that it first has to use brk then switch to mmap after a
> 1GB... or am I understanding this incorrectly? It also might break
> portability, right?

> no. first, an app which uses a large amount of memory normally
> does so in nontrivial allocations. those will get put into > mmaps.
> next, malloc cannot possibly serve all applications perfectly.
> finally, >GB apps are still rare.

> > heap, of course. but it's easy and not harmfull to shift the
> > starting point for mmap if you must use tiny mallocs. a better
> How do I do this? Is this a kernel change?

> the simplest tweak is to change TASK_UNMAPPED_BASE in >asm/processor.h
>from TASK_SIZE/3 to 2*TASK_SIZE/3 (for 2G heap). actually, I >found
>that you can change it to TASK_SIZE-512M if you don't have any >big mmaps.

______________________________________________
FREE Personalized Email at Mail.com
Sign up at http://www.mail.com?sr=mc.mk.mcm.tag001


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:56    [W:0.194 / U:0.016 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site