lkml.org 
[lkml]   [1999]   [Sep]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: ext2 file sizes
On Thu, 16 Sep 1999, Clem Taylor wrote:

> "Richard B. Johnson" wrote:
> > On Wed, 15 Sep 1999, Blankenship, Keith wrote:
> > > I am having some difficulty with the ext2 file systems. I need to generate a
> > > file that will be > 5 Gigabytes, and there appears to be a file size cap at
> > > approximately 2 Gig. I am running what appears to be a version 2.0.36
> > > Kernel. Is there anything I can adjust, or do to increase the maximum file
> > > size? Or is there a newer kernel that may work?
> >
> > Note that on a 32-bit machine, toff_t, used as fpos_t, for file offsets,
> > i.e., lseek, is unsigned 32 bits. You will not be able to access such a
> > file on a 32-bit machine. There has been some work on changing this
> > to 'long long' (__kernel_loff_t, in ../asm/posix_types.h although you may
> > need a new 'C' runtime library to take advantage of this.
> >
> > What on earth are you doing with a single _file_ of that size? If you
> > need a gob of storage for data acquisition, you might be better off
> > with a dedicated raw device. Since you access it in blocks, rather than
> > bytes, you won't have a size limitation for a few more years.
>
> I work with compressed video and my average file size is 3-6 gigs,
> increasing
> all the time (higher bitrates). A while back I looked into using Linux for
> a
> video application and gave up and went with Solaris and Irix because of
> the lack
> of large file support. I was hoping that 2.4 would have large file support
> (and
> not just on 64 bit platforms), but that isn't looking promising based on
> comments here. Going to a 64 bit off_t (on 32 bit platforms) is a major
> change
> and will have a serious impact in user space. Just look at all the pain
> Solaris
> and Irix had in making the move.
>
> --Clem
>

Here, we acquire great gobs of data (CAT Scan Data), typically 4 gb per
run. I run it off to a dedicated SCSI. After it's converted into useful
images, the images are saved in a conventional database on a file-system.

This seems to be a real good approach because, even with smaller files
the ext2 file-system has a problem keeping up with the data-rate when
buffers get filled and must be flushed to disk. Our data rate is
2880*1440 = 4,147,200 longwords/second. We throw away the high byte
so we have a sustained data rate of 12,441,600 bytes/second. Good SCSI
controllers (BusLogic) keep up with this fine. The ext2 file-system keeps
up with this until its buffers get full. Then it's too bad.

Writes to a dedicated SCSI solved all our problems. I can sustain writes
at these speeds forever (until the drive is full).


Cheers,
Dick Johnson
**** FILE SYSTEM WAS MODIFIED ****
Penguin : Linux version 2.2.4 on an i686 machine (400.59 BogoMips).
Warning : It's hard to remain at the trailing edge of technology.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:54    [W:0.048 / U:0.184 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site