[lkml]   [2003]   [Oct]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: Transparent compression in the FS
Andrea Arcangeli wrote:
> Hi Jeff,
> On Wed, Oct 15, 2003 at 11:13:27AM -0400, Jeff Garzik wrote:
>>Josh and others should take a look at Plan9's venti file storage method
>>-- archival storage is a series of unordered blocks, all of which are
>>indexed by the sha1 hash of their contents. This magically coalesces
>>all duplicate blocks by its very nature, including the loooooong runs of
>>zeroes that you'll find in many filesystems. I bet savings on "all
>>bytes in this block are zero" are worth a bunch right there.
> I had a few ideas on the above.
> if the zero blocks are the problem, there's a tool called zum that nukes
> them and replaces them with holes. I use it sometime, example:
> andrea@velociraptor:~> dd if=/dev/zero of=zero bs=1M count=100
> 100+0 records in
> 100+0 records out
> andrea@velociraptor:~> ls -ls zero
> 102504 -rw-r--r-- 1 andrea andrea 104857600 2003-10-16 18:24 zero
> andrea@velociraptor:~> ~/bin/i686/zum zero
> zero [820032K] [1 link]
> andrea@velociraptor:~> ls -ls zero
> 0 -rw-r--r-- 1 andrea andrea 104857600 2003-10-16 18:24 zero
> andrea@velociraptor:~>


> the hash to the data is interesting, but 1) you lose the zerocopy
> behaviour for the I/O, it's like doing a checksum for all the data going to
> disk that you normally would never do (except for the tiny files in reiserfs
> with tail packing enabled, but that's not bulk I/O), 2) I wonder how much data
> is really duplicate besides the "zero" holes trivially fixable in userspace
> (modulo bzImage or similar where I'm unsure if the fs code in the bootloader
> can handle holes ;).

FWIW archival storage doesn't really care... Since all data written to
disk is hashed with SHA1 (sha1 hash == block's unique id), you gain (a)
duplicate block coalescing and (b) _real_ data integrity guaranteed, but
OTOH, you lose performance and possibly lose zero-copy.

I _really_ like the checksum aspect of Plan9's archival storage (venti).

As Andre H and Larry McVoy love to point out, data isn't _really_ secure
until it's been checksummed, and that checksum data is verified on
reads. LM has an anecdote (doesn't he always? <g>) about how BitKeeper
-- which checksums its data inside the app -- has found data-corrupting
kernel bugs, in days long past.


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
More majordomo info at
Please read the FAQ at

 \ /
  Last update: 2005-03-22 13:58    [W:0.312 / U:3.928 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site