[lkml]   [2011]   [Sep]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: [PATCH v1 00/16] ext4: Add metadata checksumming
On Sun, Sep 04, 2011 at 07:41:03AM -0400, Martin K. Petersen wrote:
> >>>>> "Darrick" == Darrick J Wong <> writes:
> Darrick,
> Darrick> Furthermore, the nice thing about the in-filesystem checksum is
> Darrick> that we bake in other things like the FS UUID and the inode
> Darrick> number, which gives you a somewhat better assurance that the
> Darrick> data block belongs to the fs and the file that the code think
> Darrick> it belongs to.
> Yeah, I view DIF/DIX mostly as in-flight protection for writes. Whereas
> FS metadata checksumming is great for problem detection at read time.
> Another problem with using the DIF app tag to store filesystem metadata
> is that many array vendors use it internally and thus only disk drives
> are likely to provide the app tag space.
> Darrick> The DIX interface allows for a 32-bit block number and a 16-bit
> Darrick> application tag ... which is unfortunately small given 64-bit
> Darrick> block numbers and 32-bit inode numbers.
> I never understood the 32-bit ref tag. Seems silly to have a check that
> wraps at the exact boundary where problems are most likely to occur.
> I advocated for a DIF Type with 16-bit guard tag and 48-bit ref tag but
> that never went anywhere. Too bad - would have been easy for the storage
> vendors to implement.

> Darrick> As a side note, the crc-t10dif implementation is quite slow --
> Darrick> the hardware accelerated crc32c is 15x faster, and the sw
> Darrick> implementation is usually 3-6x faster. I suspect somebody will
> Darrick> want to fix that before DIF becomes more widespread...
> The CRC32C op on Nehalem and beyond is really, really fast. It's
> essentially free except for pulling the data through the cache. So it's
> not entirely fair to use that as baseline for a pure software
> implementation. What is the faster sw implementation are you referring
> to, btw.?

I have some benchmarking data for various crc algorithms here:

The "faster sw implementation" that I was talking about is the slice-by-8
algorithm that I sent to the crypto list a few days ago that's based off of Bob
Pearson's slice-by-8 crc32 patch.

In the huge table, "crc32c-by8-le" is crc32c slice-by-8.

> lib/crc-t10dif is a regular 256-entry table-based CRC implementation. It
> is done pretty much like all our other software CRCs. I seem to recall
> attempting a bigger table but that yielded worse real life results due
> to cache pollution.

Yes, the only downside to the slice-by-8 method is that it eats 8K of data
cache for the table. Not a huge issue on recent Intel and POWER where the L1D
is 32K, but I imagine it could be painful elsewhere.

Do you know of any faster crc16 algorithms? I guess it wouldn't be hard to
make a family of crcs, each with different cache/speed characteristics.

> On Westmere and beyond it is possible to accelerate generic CRC
> calculation using the PCLMULQDQ operation. There are many of our CRC
> functions that could benefit from this. However, so far intel have not
> been willing to contribute the relevant code to Linux.
> Darrick> The good news is that if you're really worried about integrity,
> Darrick> metadata_csum and DIF/DIX aren't mutually exclusive features.
> Darrick> Rejecting corrupted write commands at write time seems like a
> Darrick> useful feature. :)
> Yup!
> --
> Martin K. Petersen Oracle Linux Engineering
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to
> More majordomo info at

 \ /
  Last update: 2011-09-05 20:47    [W:0.102 / U:0.872 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site