[lkml]   [2008]   [Dec]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: Pull request for FS-Cache, including NFS patches
On Thu, Dec 18, 2008 at 07:45:57PM -0800, Muntz, Daniel wrote:
> Local disk cache was great for AFS back around 1992. Typical networks
> were 10 or 100Mbps (slower than disk access at the time),

Would a disk cache on SSD make any sense? Seems like it'd change the
latency tradeoff.

> and memories
> were small (typical 16MB). FS-Cache appears to help only with read
> traffic--one reason why the web loves caching--and only for reads that
> would miss the buffer/page cache (memories are now "large"). Solaris
> has had CacheFS since ~1995, HPUX had a port of it since ~1997. I'd be
> interested in evidence of even a small fraction of Solaris and/or HPUX
> shops using CacheFS. I am aware of customers who thought it sounded
> like a good idea, but ended up ditching it for various reasons (e.g.,
> CacheFS just adds overhead if you almost always hit your local mem
> cache).

More details on the experiences of RHEL/Fedora users might be
interesting. My (vague, mostly uniformed) impression is that the group
of people who think they need it is indeed a lot larger than the group
who really do need it--but that the latter group still exists.


> One argument in favor that I don't see here is that local disk cache is
> persistent (I'm assuming it is in your implementation).
> Addressing 1 and 2 in your list, I'd be curious how often a miss in core
> is a hit on disk.
> Number 3 scares me. How does this play with the expected semantics of
> NFS?
> Number 5 is hard, if not provably requiring human intervention to do
> syncs when writes are involved (see Disconnected AFS work by
> UM/CITI/Huston, and work at CMU).
> Add persistence as number 6. This may be the best reason to have it,
> imho.
> -Dan
> -----Original Message-----
> From: David Howells []
> Sent: Thursday, December 18, 2008 6:27 PM
> To: Andrew Morton
> Cc:;;
> Subject: Re: Pull request for FS-Cache, including NFS patches
> Andrew Morton <> wrote:
> > I don't believe that it has yet been convincingly demonstrated that we
> > want to merge it at all.
> >
> > It's a huuuuuuuuge lump of new code, so it really needs to provide
> > decent value. Can we revisit this? Yet again? What do we get from
> > all this?
> I should tell you to go and reread LKML at this point.
> But... What can FS-Cache do for you? Well, in your specific case,
> probably nothing. If all your computers are local to your normal
> desktop box and are connected by sufficiently fast network and you have
> sufficiently few of them, or you don't use any of NFS, AFS, CIFS,
> Lustre, CRFS, CD-ROMs then it is likely that won't gain you anything.
> Even if you do use some of those "netfs's", it won't get you anything
> yet because I haven't included patches to support anything other than
> NFS and the in-kernel AFS client yet.
> However, if you do use NFS (or my AFS client), and you are accessing
> computers via slow networks, or you have lots of machines spamming your
> NFS server, then it might avail you.
> It's a compromise: a trade-off between the loading and latencies of your
> network vs the loading and latencies of your disk; you sacrifice disk
> space to make up for the deficiencies of your network. The worst bit is
> that the latencies are additive under some circumstances (when doing a
> page read you may have to check disk and then go to the network).
> So, FS-Cache can do the following for you:
> (1) Allow you to reduce network loading by diverting repeat reads to
> local
> storage.
> (2) Allow you to reduce the latency of slow network links by diverting
> repeat
> reads to local storage.
> (3) Allow you to reduce the effect of uncontactable servers by serving
> data
> out of local storage.
> (4) Allows you to reduce the latency of slow rotating media (such as
> and CD-changers).
> (5) Allow you to implement disconnected operation, partly by (3), but
> also by
> caching changes for later syncing.
> Now, (1) and (2) are readily demonstrable. I have posted benchmarks to
> do this. (3) to (5) are not yet implemented; these have to be mostly
> implemented in the filesystems that use FS-Cache rather than FS-Cache
> itself. FS-Cache currently has sufficient functionality to do (3) and
> (4), but needs some extra bits to do (5).
> I've tried to implement just the minimal useful functionality for
> persistent caching. There is more to be added, but it's not immediately
> necessary.
> FS-Cache tries to make its facilities as simple and as general as
> possible so that _any_ filesystem or blockdev can use it. I have
> patches here to make NFS and AFS use it. I know someone is working on
> getting Lustre to use it, and other filesystem maintainers have
> expressed interest, subject to the code getting upstream.
> Furthermore, FS-Cache hides the implementation of the cache from the
> netfs.
> Not only that, it hides I/O errors in the cache from the netfs. Why
> should the netfs have to deal with such things?
> Another way to look at things is to look at other cases of cached
> netfs's.
> OpenAFS for example. It has a local cache of its own. Solaris has
> local NFS caching. Windows has local caching for NFS and CIFS, I think.
> Even web browsers have local caching.
> > 303 files changed, 21049 insertions(+), 3726 deletions(-)
> A big chunk of that, particularly the deletions, is the creds patches.
> Excluding the stuff pulled from the security and NFS trees, the combined
> FS-Cache, CacheFiles and AFS+ and NFS+FS-Cache patches are, in fact:
> 86 files changed, 15385 insertions(+), 413 deletions(-)
> and over 19% of the insertions is documentation. Most of the deletions
> (373) are in AFS.
> > Are any distros pushing for this? Or shipping it? If so, are they
> > able to weigh in and help us with this quite difficult decision?
> We (Red Hat) have shipped it in RHEL-5 and some Fedora releases. Doing
> so is quite an effort, though, precisely because the code is not yet
> upstream. We have customers using it and are gaining more customers who
> want it. There even appear to be CentOS users using it (or at least
> complaining when it breaks).
> I don't know what will convince you. I've given you theoretical reasons
> why caching ought to be useful; I've backed up the ones I've implemented
> with benchmarks; I've given you examples of what our customers are doing
> with it or want to do with it. Please help me understand what else you
> want.
> Do you perhaps want the netfs maintainers (such as Trond) to say that
> it's necessary?
> David
> _______________________________________________
> NFSv4 mailing list
> _______________________________________________
> NFSv4 mailing list

 \ /
  Last update: 2008-12-19 05:13    [W:0.111 / U:2.124 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site