lkml.org 
[lkml]   [2004]   [Feb]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectImproved file handling mechanisms for 64-bit architectures
Date
I would like to make a suggestion how to best use the new AMD64 architecture
in the new kernel version for 64-bit architectures. You are making new sets
of functions and optimizing new compilers anyway so I think that now is also
the time to introduce new file handling mechanisms that can be fully
implemented only in 64-bit environment.



Currently, the AMD64 architecture defines a mechanism for translating 48-bit
virtual addresses to 52-bit physical addresses.



I will make a suggestion on how to improve file handling performance:



Currently if you want to work with files, you have to:

1) assign file a handle; 2) read wanted portions of file into memory 3) if
there is not enough physical memory, save the contents of the file to disk



With AMD64 you are able to make 48-bit addresses, which amount to 256000
gigabytes of virtual memory. When working with large (eg. 10GB) video or
database files, Linux kernel could map the whole file into the virtual
memory using processor's Page Translation Mechanism.

Those 10GB would then be mapped to a certain memory range. If a portion of
file that is currently requested is in physical RAM the processor would
handle it without OS intervention; if not, then a page fault (#PF, 14)
exception would occur and read a missing page (a missing portion of file).



The application would see the file as a 10GB large array (or a string), or
perhaps could map its own data structures into this memory space.



When writing to file, i.e. modifying the data in the array, a #PF would
occur and mark this page as dirty and write modified data to the disk (or
schedule writing to a later time)



There is also one addition benefit to this: when inserting data into the
middle of the 10GB file - a new video frame or enlarging a table in SQL only
a remapping of virtual memory and the cooperation of file system is needed.



Let's say that AAAAAAAAAAAAAAAAAAAAAAAA is the original file. You want to
insert B somewhere in the middle of the file. AAAAAAAAABAAAAAAAAAAAAAAA is
done by calling a function increasefile(startoffset,length) that moves a few
hundred bytes in Page Table or in Page directory table and by adding an
additional segment to the file chain (eg. FAT chain). The length must just
be a multiplier of 4KB.



Adding some 'prediction' functions about intended file usage in the
application also wouldn't hurt. Application could tell the kernel that a
portion of file that has been read before isn't going to be needed anymore
(eg. movie player after playing a frame), or that it would need certain
segments of the file in the near future (parameter passed as an array of
ranges; 0-10000, 300000-600000 etc). Something similar as Itanium's branch
predictions (.sptk, .spnt,.dptk, or .dpnt), just that they are meant for
file (and not memory) usage.



I just had this idea in my mind since I've been to high school, but now I
don't do kernel programming anymore.



If you find this idea useful, you can use it. Best wishes,

Janez Zagar



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 14:00    [W:0.045 / U:0.320 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site