Messages in this thread | | | Date | Tue, 2 Jun 1998 21:39:44 +0100 (BST) | From | Peter Horton <> | Subject | Re: Q: memory management |
| |
On Tue, 2 Jun 1998, Richard Gooch wrote:
> Peter Horton writes: > > > > I have a machine with 72MB of RAM and 128MB of swap. > > > > If I try to do an allocation of 512MB using sbrk(), it fails. However, I > > can successfully do 512 sbrk()'s of 1MB. This causes a problem if I store > > the pointers to the 1MB blocks, and then try and write to the 512MB of > > memory I have just allocated ... the kernel seems to lock up solid. If I > > write to each 1MB as it is allocated, the allocation fails around 180MB > > (amount of free memory). > > > > If I understand correctly I can allocate more memory than I have because > > the physical memory pages are not allocated until you access a block > > (allowing sparse data structures). > > Yep. > > > So, my first question is why does the single sbrk(512MB) fail whilst the > > 512 x sbrk(1MB) succeeds ? > > It's a similar dumb-arse bug that you'll find with mips4_IRIX6.2 > machines. On my Linux box I can allocate up to 2046 MBytes, but the > largest single block is only 128 MBytes. This box has 128 MBytes of > RAM and 128 MBytes of swap. > > On a certain mips4_IRIX6.2 machine (with 1 GByte RAM and 10 GBytes of > swap) I can malloc 6454 MBytes but the single largest block I can > allocate is only 2048 MBytes. This is if I allocate ever increasing > blocks. If the first thing my programme does is allocate 800 MBytes, > that will fail. > Allocating 800 MBytes by mmapping /dev/zero works straight away. > > Clearly, the Linux and IRIX implementations of sbrk(2) are flawed. I > note that Solaris 2 doesn't have these problems. > > > Secondly, does the kernel lock up, or did I just not wait long enough for > > it to come back ? ... I'm not keen to try it again, as it resulted in an > > hour of fsck'ing ... 8-) > > You could easily spend hours waiting for your machine to stop > thrashing. Heavy paging also wears down your discs. The solution is: > Don't Do That[tm]. > > If you don't have sparse data, then don't use swap, just buy more > RAM. This prevents thrashing, since you can't over-commit VM. > > If you are stuck with large sparse arrays larger than RAM, be careful > not to touch more pages than you have available. > You can improve things by putting your swap parition on a separate > disc. Swap should *never* be on the same disc as your data and should > preferably not be on the disc with your OS and programmes. >
The machine didn't thrash at all. It quickly filled up all memory, and then all the swap. Then it locked. What does the kernel do when it needs a page, and all of the pages (including swap) are used ? I kinda assumed that it would just kill the process ...
P.
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu
| |