lkml.org 
[lkml]   [1997]   [Feb]   [11]   [last100]   RSS Feed
Views: [more markup]  [less markup]  [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: Big mallocs, mmap sorrows and double buffering.
DateTue, 11 Feb 1997 18:24:34 +0200 (EET)
FromMatti Aarnio <>
John Carter <john@dwaf-hri.pwv.gov.za> writes:
> On Mon, 10 Feb 1997, Mark Hahn wrote:
> 
> > use mmap!  it's _easier_ than malloc/read, and better yet, is also
> > _optimal_ when confronted with changing memory loads.  and of course
> > it also happens to be significantly more efficient, since you get
> > buffer-cache blocks mapped directly into your address space:
> > your disk controller reads them into memory, and you get to see
> > them, no silly copying from kernel to user-space.
> > 
> > let me know if you need some code on mmap.  I use it extensively.
> 
> Sounds like just what I need.

	But don't overdo it!

> Hmm. I remember using a facility like this on the old vaxen. The
> glitch was if I mapped an entire satellite image (+-400megabytes) the
> page tables used up so much physical memory that the thing started
> thrashing again. (There we just settled down to using two 32k buffers
> and doing NO_WAIT read/writes.)
> 
> Side Question : On Vaxen they had fancy IO hardware so it paid to do double
> buffering. (Ie Read first buffer, Read second buffer NO_WAIT, process
> first while second buffer is reading, Write NO_WAIT first output
> buffer, Read NO_WAIT into first input buffer, process second input
> buffer, etc. etc.) Does it pay to do this on a PC under Linux?

	The general UNIX file-io paradigms don't do it easily.
	You could try aioread()/aiowrite(), though.

> I have just tried mmap() & memcpy() to copy a 19Mb file on a 24Mb ram
> system. It started thrashing quite heavily but completed without
> hanging.
> 
> Sorrow! Woe! Misery! I tried it on a 38Mb file and it said 
> "output memory map : out of memory." 
> Is that a mmem() bug perhaps? Or has too many page descriptors
> been created to fit into physical memory?

	Lets see your source...

> So back to my original question. How do I ask linux kernel how much
> physical memory I can use without causing thrashing?
> 
> ======================================================================
> #include <sys/mman.h>
> #include <fcntl.h>
> #include <unistd.h>
> #include <sys/stat.h>
> #include <stdio.h>
> 
> main()
> {
>   int ifd, ofd;
>   struct stat st;
>   char * icp, * ocp;
> 
>   ifd = open( "/home/john/pgm/tryout/temp", 0, 0);
>   if( ifd < 0)
>     {
>       perror( "input file");
>       return 1;
>     }
> 
>   ofd = open( "/home/john/pgm/tryout/temp1", O_CREAT | O_TRUNC, O_WRONLY);
>   if( ofd < 0)
>     {
>       perror( "output file");
>       return 1;
>     }
> 
>   fstat( ifd, &st);
> 
>   icp = (char *)mmap( 0, st.st_size, 
> 		      PROT_READ, MAP_FILE | MAP_PRIVATE, ifd, 0);

	Ouch!  You need:  MAP_FILE|MAP_SHARED
	You don't want a private copy of it!
>   if( icp == (char *)-1)
>     {
>       perror( "input memory map");
>       return 1;
>     }
> 
>   ocp = (char *)mmap( 0, st.st_size, 
> 		      PROT_WRITE, MAP_FILE | MAP_PRIVATE, ofd, 0);

	No, you can't mmap() for writing the file.
	Writing you must do by write() call.
	Doing a MAP_PRIVATE creates you a private block of memory of the
	size of the file, however it will not go to the destination file!

>   if( ocp == (char *)-1)
>     {
>       perror( "output memory map");
>       return 1;
>     }
> 
>   memcpy( ocp, icp, st.st_size); // OK, so I don't actually change
>                                  // anything here, but in a real
> 				 // program I would.

	write(ofd, icp, st.st_size);

>   munmap( icp, st.st_size);
>   munmap( ocp, st.st_size);
> 
>   close( ifd);
>   close( ofd);
> }
> ======================================================================
> 
> John Carter                    EMail: ece@dwaf-hri.pwv.gov.za
> Telephone : 27-12-808-0374x194 Fax:- 27-12-808-0338
> 
> Founder of the Council for Unnatural Scientists.

	/Matti Aarnio <matti.aarnio@tele.fi>

\
 
 \ /
  Last update: 2005-03-22 12:38    [W:0.360 / U:0.010 seconds]
©2003-2008 Jasper Spaans