Messages in this thread | | | Date | Thu, 16 Sep 1999 09:06:04 -0400 (EDT) | From | "Richard B. Johnson" <> | Subject | Re: ext2 file sizes |
| |
On Thu, 16 Sep 1999, Clem Taylor wrote:
> "Richard B. Johnson" wrote: > > On Wed, 15 Sep 1999, Blankenship, Keith wrote: > > > I am having some difficulty with the ext2 file systems. I need to generate a > > > file that will be > 5 Gigabytes, and there appears to be a file size cap at > > > approximately 2 Gig. I am running what appears to be a version 2.0.36 > > > Kernel. Is there anything I can adjust, or do to increase the maximum file > > > size? Or is there a newer kernel that may work? > > > > Note that on a 32-bit machine, toff_t, used as fpos_t, for file offsets, > > i.e., lseek, is unsigned 32 bits. You will not be able to access such a > > file on a 32-bit machine. There has been some work on changing this > > to 'long long' (__kernel_loff_t, in ../asm/posix_types.h although you may > > need a new 'C' runtime library to take advantage of this. > > > > What on earth are you doing with a single _file_ of that size? If you > > need a gob of storage for data acquisition, you might be better off > > with a dedicated raw device. Since you access it in blocks, rather than > > bytes, you won't have a size limitation for a few more years. > > I work with compressed video and my average file size is 3-6 gigs, > increasing > all the time (higher bitrates). A while back I looked into using Linux for > a > video application and gave up and went with Solaris and Irix because of > the lack > of large file support. I was hoping that 2.4 would have large file support > (and > not just on 64 bit platforms), but that isn't looking promising based on > comments here. Going to a 64 bit off_t (on 32 bit platforms) is a major > change > and will have a serious impact in user space. Just look at all the pain > Solaris > and Irix had in making the move. > > --Clem >
Here, we acquire great gobs of data (CAT Scan Data), typically 4 gb per run. I run it off to a dedicated SCSI. After it's converted into useful images, the images are saved in a conventional database on a file-system.
This seems to be a real good approach because, even with smaller files the ext2 file-system has a problem keeping up with the data-rate when buffers get filled and must be flushed to disk. Our data rate is 2880*1440 = 4,147,200 longwords/second. We throw away the high byte so we have a sustained data rate of 12,441,600 bytes/second. Good SCSI controllers (BusLogic) keep up with this fine. The ext2 file-system keeps up with this until its buffers get full. Then it's too bad.
Writes to a dedicated SCSI solved all our problems. I can sustain writes at these speeds forever (until the drive is full).
Cheers, Dick Johnson **** FILE SYSTEM WAS MODIFIED **** Penguin : Linux version 2.2.4 on an i686 machine (400.59 BogoMips). Warning : It's hard to remain at the trailing edge of technology.
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |