lkml.org 
[lkml]   [2001]   [Sep]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: Editing-in-place of a large file
    Ingo Oeser [ingo.oeser@informatik.tu-chemnitz.de] wrote:
    > On Sun, Sep 02, 2001 at 05:59:38PM -0700, Larry McVoy wrote:
    > > > What's needed is a generalisation of sparse files and truncate().
    > > > They both handle similar problems.
    > >
    > > how about
    > >
    > > fzero(int fd, off_t off, size_t len)
    > > fdelete(int fd, off_t off, size_t len)
    >
    > and
    >
    > finsert(int fd, off_t off, size_t len, void *buf, size_t buflen)
    >
    > > The main problem with this is if the off/len are not block aligned. If they
    > > are, then this is just block twiddling, if they aren't, then this is a file
    > > rewrite anyway.

    *exactly* I don't know enough about ext2fs to know if this is possible
    (i.e. a partially filled block in the middle of a file) so that's why I
    asked.

    > Another solution for the original problem is to rewrite the file
    > in-place by coping from the end of the gap to the beginning of
    > the gap until the gap is shifted to the end of the file and thus
    > can be left to ftruncate().

    For editing commercials, you'd still have to copy 90% of the data. In
    the US, there's roughly 5 minutes of commercials for every 15 of the
    show, so that would only save copying the first 15 minutes...

    > This will at least not require more space on disk, but will take
    > quite a while and risk data corruption for this file in case of
    > abortion.

    Yep. I should mention that the Linux/mjpeg tools
    (http://mjpeg.sourceforge.net) already have an elegant way of "marking"
    a portion of a video and skipping it when playing it, through the use of
    "edit lists". (use xlav/glav to mark it, and then you can lavplay the
    edit list, which just contains the start/end of skipped sections) They
    also have a program to apply the edit list and create a new video
    (lavtrans). But this requires copying the desired sections of video to
    a new file, which requires 75% more disk space than the original file,
    and takes a looong time.

    The idea behind my first message should be obvious here...an almost
    atomic operation modifying at most 2 blocks (and marking a bunch as
    free) wouldn't require nearly as much disk-thrashing, and would be
    nearly instantaneous from the user's perspective.

    Disk fragmentation is unimportant when the contiguous chunks are 300MB
    long.

    > But fzero, fdelete and finsert might be worth considering, since
    > some file systems, which pack tails could also pack these kind of
    > partial used blocks and handle them properly.

    Do the journaling filesystems use blocks in a similar manner to ext2fs?
    Anyone know if any of them can handle partially filled blocks in the
    middle of a file?

    Are there any media-filesystems out there that have these kinds of
    extensions? I'm not sure these extensions would be useful for anything
    but editing media...

    > We already handle partial pages, so why not handle them with
    > offset/size pairs and enable this mechanisms? Multi media streams
    > would love these kind of APIs ;-)

    Yep yep yep. What do multimedia people use? Custom multi-thousand
    dollar programs with their own filesystem layer? What about TiVo?
    Didn't they contribute some fs-layer modifications a while back?

    Cheers,
    -- Bob

    Bob McElrath (rsmcelrath@students.wisc.edu)
    Univ. of Wisconsin at Madison, Department of Physics
    [unhandled content-type:application/pgp-signature]
    \
     
     \ /
      Last update: 2005-03-22 13:01    [W:0.028 / U:0.372 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site