lkml.org 
[lkml]   [1999]   [Oct]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectLinux Buffer Cache Does Not Support Mirroring

Linus,

The Netware file system for linux is running mirroring now (up to 8
mirrors per logical partition), but unfortunately, we had to write our
own buffer cache and integrate it in a modular fashion with all the
Linux versions -- not as simple an undertaking as one might think given
all the versions of Linux. We are still calling getblk() underneath so
both buffer caches are in sync, but are now looking at completely
by-passing your cache and going to disk with make_request(). We are
close on balancing memory consumption between the caches, but sure could
use some help.

All of this work delayed us by about 4 months with the project and was
something we did not anticipate. The problem is that the buffer cache
is based on physical disk LBA offsets rather than supporting a tiered
semantic for logical disk objects. Ideally, if the buffer cache
supported a logical object manager (partition based) that would
translate logical partition blocks to LBA's, with a configurable "list"
of devices to write to, we would be there.

Here's the problem with what's there in a nutshell. If we use your
buffer cache to perform the mirroring, we end up double, triple, and
quadruple caching data. Since you store blocks based on LBA, each time
we write a mirror, the data gets buffered once for each device.
Ideally, since we are dealing with "mirrors" we only need one cached
copy of the disk block and the ability to force it to write to several
physical devices at once. There are also two other cases that are
biting us in the ass. We load balance reads by striping them across the
mirrors and reading them round robin -- again we get double caching for
each mirror present.

There is also another case where we perform an artificial hotfix to
mirrors on read HOTFIX errors (if the sectors were not recovered on the
primary, and could not be located on a valid mirror, then mirrors must
be artificially HOTFIXED so users will get a disk read error on any
files that could not be recovered.

This will require a semantic similiar to what's in NT's VM Cache today
(they already ran across this with their FTDISK.SYS stuff, and learned
that this semantic is needed -- a cache object handle that is relative
to each partition on a device that the buffer cache maps underneath to
LBA). For now, we've got our own buffer cache that is customized for
what we need, and I don't expect you to backrev older Linux versions,
but we do need this moving forward, otherwise, Linux will be limited in
terms of fault tolerance available.

We've got all the tools for creating Netware volumes, mirrors, disk
repair, etc. finished and will be putting them up soon. Steve Spicer is
enabling NWFS for SPARC and ALPHA machines. We want to get EVERYTHING
done at once and put up at one time.

Please advise on these cache issues.

Jeff Merkey

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:54    [W:1.309 / U:0.004 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site