Messages in this thread | | | Date | Wed, 20 Jun 2007 12:44:58 +0400 | From | Vladislav Bolkhovitin <> | Subject | Re: [ANNOUNCE] Btrfs: a copy on write, snapshotting FS |
| |
Philipp Matthias Hahn wrote: >>>>I would also suggest one more feature: support for block level >>>>de-duplication. I mean: > > ... > >>>>That would be very usable feature, which in most cases would allow to >>>>shrink occupied disk space on 50-90%. >>> >>>Have you references for this number? >>>In my experience one gets a lot of benefit from >>>the much simpler process of "de-duplication" of files. >> >>Yes, I would expect simple hard links to be a better solution for this, >>but the feature request is not that out of line. I actually had plans >>on implementing auto duplicate block reuse earlier in btrfs. > > > One problem with hard-links for me is, they also share the meta-data, > especially file permissions and owners. > > Take a Subversion checkout for example: For each file "$A" Subversion > saves a backup under ".svm/text-base/$A.svn-base" for file comparison > and diff generation. The user controls the file permissions of "$A", > Subversion protects its backup with 0444. You can't hard-link them, > because than "svn diff" doesn't work anymore if your editor doesn't > break the hard-link, or worse, your permissions can get wrong. > > If previous versions Subversion also had an extra file for file > attributes (mime-type, permissions, to-be-ignored, etc.) Since most > files had no special attributes, each had a file only containing "END". > Those you could hard-link by hand to save space. > > If somebody want to research this further: > > There is this nice little package called "perforate", which contains > "finddup" to find duplicate files. Run it two times, once with "-i" to > ignore permissions while comparing file contents > finddup -i -d / > and once without "-i" for "content and permissions must match" > finddup -d / > This will give you a hint on how many files you could hard-link or how > many files share their content.
So, seems ever for file based de-duplication some support from the FS, including some kind of ability for different inodes point to the same data blocks to store the meta-data, would be needed anyway.
Vlad - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |