[lkml]   [2007]   [Jun]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [ANNOUNCE] Btrfs: a copy on write, snapshotting FS
    On Mittwoch, 20. Juni 2007, Vladislav Bolkhovitin wrote:
    > Philipp Matthias Hahn wrote:
    > >>>>I would also suggest one more feature: support for block level
    > >>>>de-duplication. I mean:
    > So, seems ever for file based de-duplication some support from the FS,
    > including some kind of ability for different inodes point to the same
    > data blocks to store the meta-data, would be needed anyway.
    The easy way is to have the inode point to a (shared, reference counted) data
    storage, which lists the data - then inodes can share the data, but have
    different meta-data.

    Ever since I read about filesystems using the files' hash as addressing
    mechanism (per some Linus mail on LKML, about 10 years ago) and manber hashes
    ( I'm thinking about various
    ways to use both in a filesystem.

    Manber-Hashes allow to split data into similar chunks, which could be
    addresses per some cryptographic checksum, and used by several files.

    *But*: The boundaries are not at power-of-2 addresses, so the data for
    read()/mmap() would have to be rebuild somehow. (Would eg. be necessary
    anyway if the data block itself is stored compressed).

    The other question I'm still pondering ... File sizes vary very much. If I
    have a large project, eg. the kernel, with many thousand files of some 10 kB,
    some data could be shared - GPL licenses in files, #include lists, and some
    If I have some other data, with files of several hundred megabytes, sharing
    gets more interesting ... but what should the right block size (for manber
    hashes) be? If it's small, we have to concatenate a lot of blocks to
    reconstruct data - if it's big, we might lose many chances for sharing.

    And getting good performance, when blocks in the middle of a file have to be
    re-splitted, might be a bit of a problem, too ...



    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2007-06-20 11:21    [W:0.025 / U:1.836 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site