Messages in this thread | | | Date | Tue, 8 Oct 2002 13:37:38 -0700 | From | Larry McVoy <> | Subject | Re: [PATCH] O_STREAMING - flag for optimal streaming I/O |
| |
> What we need is to detect the situation where someone is linearly > walking through a file which is preposterously too large to cache, > and just start dropping it.
...
> The tricky part is designing the algorithm which decides when to > pull the trigger.
I did a variation of this in SunOS years ago. It did not have the "big file" wrinkle you are suggesting, it worked like
if (we are sequential AND we are running low on memory AND file size > 256K) { invalidate the pages behind me }
You use the same data structures which turn on read ahead to mean sequential access, that's obvious.
What this didn't fix was when you read a monstor file into memory and then didn't do anything with it. That would fill the page cache and the above alg would keep you from thrashing the machine but didn't flush the stale memory.
If I were to do it again, I'd maintain stats in the inode about access pattern, # of pages in ram for the inode, time of last I/O, time of last page fault or read. Then when memory is getting tight you do a
foreach i (ALL INODES) { unless (i.accesspat == SEQ) continue; unless ((now() - i.pagefault) > STALE_TIME) continue; unless ((now() - i.io) > STALE_TIME) continue; flush_pages(); }
You want to be a lot more clever than that because you'd like to have fudging in favor of the clean pages vs dirty pages, you more or less end up wanting to go through the loop more than once, getting more and more eager as you are more and more desparate for ram. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |