Messages in this thread |  | | Subject | Re: Big mallocs, mmap sorrows and double buffering. | Date | Tue, 11 Feb 1997 18:24:34 +0200 (EET) | From | Matti Aarnio <> |
| |
John Carter <john@dwaf-hri.pwv.gov.za> writes: > On Mon, 10 Feb 1997, Mark Hahn wrote: > > > use mmap! it's _easier_ than malloc/read, and better yet, is also > > _optimal_ when confronted with changing memory loads. and of course > > it also happens to be significantly more efficient, since you get > > buffer-cache blocks mapped directly into your address space: > > your disk controller reads them into memory, and you get to see > > them, no silly copying from kernel to user-space. > > > > let me know if you need some code on mmap. I use it extensively. > > Sounds like just what I need.
But don't overdo it!
> Hmm. I remember using a facility like this on the old vaxen. The > glitch was if I mapped an entire satellite image (+-400megabytes) the > page tables used up so much physical memory that the thing started > thrashing again. (There we just settled down to using two 32k buffers > and doing NO_WAIT read/writes.) > > Side Question : On Vaxen they had fancy IO hardware so it paid to do double > buffering. (Ie Read first buffer, Read second buffer NO_WAIT, process > first while second buffer is reading, Write NO_WAIT first output > buffer, Read NO_WAIT into first input buffer, process second input > buffer, etc. etc.) Does it pay to do this on a PC under Linux?
The general UNIX file-io paradigms don't do it easily. You could try aioread()/aiowrite(), though.
> I have just tried mmap() & memcpy() to copy a 19Mb file on a 24Mb ram > system. It started thrashing quite heavily but completed without > hanging. > > Sorrow! Woe! Misery! I tried it on a 38Mb file and it said > "output memory map : out of memory." > Is that a mmem() bug perhaps? Or has too many page descriptors > been created to fit into physical memory?
Lets see your source...
> So back to my original question. How do I ask linux kernel how much > physical memory I can use without causing thrashing? > > ====================================================================== > #include <sys/mman.h> > #include <fcntl.h> > #include <unistd.h> > #include <sys/stat.h> > #include <stdio.h> > > main() > { > int ifd, ofd; > struct stat st; > char * icp, * ocp; > > ifd = open( "/home/john/pgm/tryout/temp", 0, 0); > if( ifd < 0) > { > perror( "input file"); > return 1; > } > > ofd = open( "/home/john/pgm/tryout/temp1", O_CREAT | O_TRUNC, O_WRONLY); > if( ofd < 0) > { > perror( "output file"); > return 1; > } > > fstat( ifd, &st); > > icp = (char *)mmap( 0, st.st_size, > PROT_READ, MAP_FILE | MAP_PRIVATE, ifd, 0);
Ouch! You need: MAP_FILE|MAP_SHARED You don't want a private copy of it!
> if( icp == (char *)-1) > { > perror( "input memory map"); > return 1; > } > > ocp = (char *)mmap( 0, st.st_size, > PROT_WRITE, MAP_FILE | MAP_PRIVATE, ofd, 0);
No, you can't mmap() for writing the file. Writing you must do by write() call. Doing a MAP_PRIVATE creates you a private block of memory of the size of the file, however it will not go to the destination file!
> if( ocp == (char *)-1) > { > perror( "output memory map"); > return 1; > } > > memcpy( ocp, icp, st.st_size); // OK, so I don't actually change > // anything here, but in a real > // program I would.
write(ofd, icp, st.st_size);
> munmap( icp, st.st_size); > munmap( ocp, st.st_size); > > close( ifd); > close( ofd); > } > ====================================================================== > > John Carter EMail: ece@dwaf-hri.pwv.gov.za > Telephone : 27-12-808-0374x194 Fax:- 27-12-808-0338 > > Founder of the Council for Unnatural Scientists.
/Matti Aarnio <matti.aarnio@tele.fi>
|  |