[lkml]   [2003]   [Oct]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: Transparent compression in the FS
On Thu, 16 Oct 2003, Jeff Garzik wrote:

> Val Henson wrote:
> > Abstract:
> >
> > "Recent research has produced a new and perhaps dangerous technique
> > for uniquely identifying blocks that I will call
> > compare-by-hash. Using this technique, we decide whether two blocks
> > are identical to each other by comparing their hash values, using a
> > collision-resistant hash such as SHA-1. If the hash values match,
> > we assume the blocks are identical without further ado. Users of
> > compare-by-hash argue that this assumption is warranted because the
> > chance of a hash collision between any two randomly generated blocks
> > is estimated to be many orders of magnitude smaller than the chance
> > of many kinds of hardware errors. Further analysis shows that this
> > approach is not as risk-free as it seems at first glance."
> I'm curious if anyone has done any work on using multiple different
> checksums? For example, the cost of checksumming a single block with
> multiple algorithms (sha1+md5+crc32 for a crazy example), and storing
> each checksum (instead of just one sha1 sum), may be faster than reading
> the block off of disk to compare it with the incoming block. OTOH,
> there is still a mathematical possibility (however-more-remote) of a
> collission...

At that point it is better to extend the fingerprint size, since the SHA1
algorithm has a better distribution compared to md5 and crc32. Probability
estimates are pretty low though. If you consider a 2^32 blocks FS, that
with a 4Kb block size makes a 4 tera FS, the collision probability is in
the orders of 2^(-95) (with a 160 bit fingerprint). That's a pretty low
number. Yes, it is true that the input is not completely random, but a
good property of SHA1 is the one of spreading output result of very
similar input patterns.

- Davide

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
More majordomo info at
Please read the FAQ at

 \ /
  Last update: 2005-03-22 13:58    [W:0.123 / U:2.016 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site