Messages in this thread |  | | Date | Tue, 11 Feb 1997 11:23:27 +0200 (SAT) | From | John Carter <> | Subject | re: Big mallocs, mmap sorrows and double buffering. |
| |
On Mon, 10 Feb 1997, Mark Hahn wrote:
> use mmap! it's _easier_ than malloc/read, and better yet, is also > _optimal_ when confronted with changing memory loads. and of course > it also happens to be significantly more efficient, since you get > buffer-cache blocks mapped directly into your address space: > your disk controller reads them into memory, and you get to see > them, no silly copying from kernel to user-space. > > let me know if you need some code on mmap. I use it extensively.
Sounds like just what I need.
Hmm. I remember using a facility like this on the old vaxen. The glitch was if I mapped an entire satellite image (+-400megabytes) the page tables used up so much physical memory that the thing started thrashing again. (There we just settled down to using two 32k buffers and doing NO_WAIT read/writes.)
Side Question : On Vaxen they had fancy IO hardware so it paid to do double buffering. (Ie Read first buffer, Read second buffer NO_WAIT, process first while second buffer is reading, Write NO_WAIT first output buffer, Read NO_WAIT into first input buffer, process second input buffer, etc. etc.) Does it pay to do this on a PC under Linux?
I have just tried mmap() & memcpy() to copy a 19Mb file on a 24Mb ram system. It started thrashing quite heavily but completed without hanging.
Sorrow! Woe! Misery! I tried it on a 38Mb file and it said "output memory map : out of memory." Is that a mmem() bug perhaps? Or has too many page descriptors been created to fit into physical memory?
So back to my original question. How do I ask linux kernel how much physical memory I can use without causing thrashing?
====================================================================== #include <sys/mman.h> #include <fcntl.h> #include <unistd.h> #include <sys/stat.h> #include <stdio.h>
main() { int ifd, ofd; struct stat st; char * icp, * ocp;
ifd = open( "/home/john/pgm/tryout/temp", 0, 0); if( ifd < 0) { perror( "input file"); return 1; } ofd = open( "/home/john/pgm/tryout/temp1", O_CREAT | O_TRUNC, O_WRONLY); if( ofd < 0) { perror( "output file"); return 1; }
fstat( ifd, &st); icp = (char *)mmap( 0, st.st_size, PROT_READ, MAP_FILE | MAP_PRIVATE, ifd, 0);
if( icp == (char *)-1) { perror( "input memory map"); return 1; }
ocp = (char *)mmap( 0, st.st_size, PROT_WRITE, MAP_FILE | MAP_PRIVATE, ofd, 0);
if( ocp == (char *)-1) { perror( "output memory map"); return 1; }
memcpy( ocp, icp, st.st_size); // OK, so I don't actually change // anything here, but in a real // program I would.
munmap( icp, st.st_size); munmap( ocp, st.st_size); close( ifd); close( ofd); } ======================================================================
John Carter EMail: ece@dwaf-hri.pwv.gov.za Telephone : 27-12-808-0374x194 Fax:- 27-12-808-0338
Founder of the Council for Unnatural Scientists.
|  |