[lkml]   [1997]   [Feb]   [11]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subjectre: Big mallocs, mmap sorrows and double buffering.

On Tue, 11 Feb 1997, John Carter wrote:

> > use mmap! [...]

> Sounds like just what I need.
> Hmm. I remember using a facility like this on the old vaxen. The
> glitch was if I mapped an entire satellite image (+-400megabytes) the
> page tables used up so much physical memory that the thing started
> thrashing again. (There we just settled down to using two 32k buffers
> and doing NO_WAIT read/writes.)

400M mmap()-ed file takes ~400K page tables (for every 4K page there is a
4 byte descriptor). So this shouldnt be an issue even on low-end machines.

> Side Question : On Vaxen they had fancy IO hardware so it paid to do double
> buffering. (Ie Read first buffer, Read second buffer NO_WAIT, process
> first while second buffer is reading, Write NO_WAIT first output
> buffer, Read NO_WAIT into first input buffer, process second input
> buffer, etc. etc.) Does it pay to do this on a PC under Linux?

Linux does asyncron readaheads, so usually you will use max IO bandwith
even when not parallelizing the code. If you want to parallelize
explicitly, you might want to look into LinuxThreads (which now is
integrated into glibc 2.0), this works on SMP Linux too.

> I have just tried mmap() & memcpy() to copy a 19Mb file on a 24Mb ram
> system. It started thrashing quite heavily but completed without
> hanging.

you code maps the file 'private'. This means no other process should see
the changes made to that 'file' ---> it gets swapped out to the swap

if you want to use the zero-copy central page cache for doing fast stuff,
you should map your file 'shared' ---> these files get swapped out into
themselves. (written back).

> Sorrow! Woe! Misery! I tried it on a 38Mb file and it said
> "output memory map : out of memory."
> Is that a mmem() bug perhaps? Or has too many page descriptors
> been created to fit into physical memory?

it warns you that 'you wont be able to swap that thing out' [and you dont
want to swap it out into the swap partition anyways]

for shared mappings no such checks are done.

> So back to my original question. How do I ask linux kernel how much
> physical memory I can use without causing thrashing?

it's rather a question of 'working set', and 'how fast the working set

admittedly, the current situation for the 'simulatneous large serial
access' isnt too good, but that can be helped fast, the Linux VM manager
is very flexible, so just mail to this list if you need something ;)

the current logic is optimized for the access 'process usage based swapout
(LRU+priorities) + random swapin' (desktop usage).

but IMHO sattelite image processing shouldnt be a problem. [maybe some
large sequential access detection logic should be added ... dont know]

-- mingo

 \ /
  Last update: 2005-03-22 13:38    [W:0.090 / U:0.064 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site