[lkml]   [2008]   [Feb]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [PATCH 00/37] Permit filesystem local caching
    Daniel Phillips <> wrote:

    > On Monday 25 February 2008 15:19, David Howells wrote:
    > > So I guess there's a problem in cachefiles's efficiency - possibly due
    > > to the fact that it tries to be fully asynchronous.
    > OK, not just my imagination, and it makes me feel better about the patch
    > set because efficiency bugs are fixable while fundamental limitations
    > are not.

    One can hope:-)

    > How much of a hurry are you in to merge this feature? You have bits
    > like this:

    I'd like to get it upstream sooner rather than later. As it's not upstream,
    but it's prerequisite patches touch a lot of code, I have to spend time
    regularly making my patches work again. Merge windows are completely not fun.

    > "Add a function to install a monitor on the page lock waitqueue for a
    > particular page, thus allowing the page being unlocked to be detected.
    > This is used by CacheFiles to detect read completion on a page in the
    > backing filesystem so that it can then copy the data to the waiting
    > netfs page."
    > We already have that hook, it is called bio_endio.

    Except that isn't accessible. CacheFiles currently has no access to the
    notification from the blockdev to the backing fs, if indeed there is one. All
    we can do it trap the backing fs page becoming available.

    > My strong intuition is that your whole mechanism should sit directly on the
    > block device, no matter how attractive it seems to be able to piggyback on
    > the namespace and layout management code of existing filesystems.

    There's a place for both.

    Consider a laptop with a small disk, possibly subdivided between Linux and
    Windows. Linux then subdivides its bit further to get a swap space. What you
    then propose is to break off yet another chunk to provide the cache. You
    can't then use this other chunk for anything else, even if it's, say, 1% used
    by the cache.

    The way CacheFiles works is that you tell it that it can use up to a certain
    percentage of the otherwise free disk space on an otherwise existing
    filesystem. In the laptop case, you may just have a single big partition. The
    cache will fill up as much of it can, and as the other contents of the
    partition consume space, the cache will be culled to make room.

    On the other hand, a system like my desktop, where I can slap in extra disks
    with mound of extra disk space, it might very well make sense to commit block
    devices to caching, as this can be used to gain performance.

    I have another cache backend (CacheFS) which takes the form of a filesystem,
    thus allowing you to mount a blockdev as a cache. It's much faster than Ext3
    at storing and retrieving files... at first. The problem is that I've mucked
    up the free space retrieval such that performance degrades by 20x over time for
    files of any size.

    Basically any cache on a raw blockdev _is_ a filesystem, just one in which
    you're randomly allowed to discard data to make life easier.

    > I see your current effort as the moral equivalent of FUSE: you are able to
    > demonstrate certain desirable behavioral properties, but you are unable to
    > reach full theoretical efficiency because there are layers and layers of
    > interface gunk interposed between the netfs user and the cache device.

    The interface gunk is meant to be as thin as possible, but there are
    constraints (see the documentation in the main FS-Cache patch for more

    (1) It's a requirement that it only be tied to, say, AFS. We might have
    several netfs's that want caching: AFS, CIFS, ISOFS (okay, that last isn't
    really a netfs, but it might still want caching).

    (2) I want to be able to change the backing cache. Under some circumstances I
    might want to use an existing filesystem, under others I might want to
    commit a blockdev. I've even been asked about using battery-backed RAM -
    which has different design constraints.

    (3) The constraint has been imposed by the NFS team that the cache be
    completely asynchronous. I haven't quite met this: readpages() will wait
    until the cache knows whether or not the pages are available on the
    principle that read operations done through the cache can be considered
    synchronous. This is an attempt to reduce the context switchage involved.

    Unfortunately, the asynchronicity requirement has caused the middle layer to
    bloat. Fortunately, the backing cache needn't bloat as it can use the middle
    layer's bloat.

    > That said, I also see you have put a huge amount of work into this over
    > the years, it is nicely broken out, you are responsive and easy to work
    > with, all arguments for an early merge. Against that, you invade core
    > kernel for reasons that are not necessarily justified:
    > * two new page flags

    I need to keep track of two bits of per-cached-page information:

    (1) This page is known by the cache, and that the cache must be informed if
    the page is going to go away.

    (2) This page is being written to disk by the cache, and that it cannot be
    released until completion. Ideally it shouldn't be changed until
    completion either so as to maintain the known state of the cache.

    I could set up a radix tree per data storage object to keep track of both these
    points, however this would mean that the netfs would have to do a call,
    spinlock, conditional jumps, etc to find out either state.

    On the other hand, if we can spare two page flags, those are sufficient.

    Note that the cache doesn't necessarily need to be able to find the netfs
    pages, but may have to pin resources for backing them.

    Further note that PG_private may not be used as I want to be able to use
    caching with ISOFS eventually.

    > * a new fileops method

    Do you mean a new address space ops method? Yes. I have to be able to write
    from a kernel page without the use of a struct file. The struct file isn't
    actually necessary to do the write, and so is a waste of space. What's worse
    is that the struct file plays havoc with resource limits and ENFILE production.

    Ideally I want a couple of hooks: one to do O_DIRECT writing to a file from
    kernel pages, one to do O_DIRECT|O_NOHOLE reading from a file to kernel pages
    (holes in cache files represent blocks not retrieved from the server, so I want
    to see ENODATA not a block of zeros).

    > * many changes to LSM including new object class and new hooks
    > * separate fs*id from task struct

    It has been required that I call vfs_mkdir() and suchlike rather than bypassing
    security and calling inode ops directly. Therefore the VFS and LSM get to deny
    the caching kernel modules access to the cache data because under some
    circumstances the caching code is running in the security context of whatever
    process issued the original syscall on the netfs.

    Furthermore, the security parameters with which a file is created (UID, GID,
    security label) would be derived from that process that issued the system call,
    thus potentially preventing other processes from accessing the cache, in
    particular cachefilesd.

    So, what is required is to temporarily override the security of the process
    that issued the system call. We can't, however, just do an in-place change of
    the security data as that affects the process as an object, not just as a
    subject. This means it may lose signals or ptrace events for example, and
    affect what the task looks like in /proc.

    So what I've done is to make a logical slit in the security between the
    objective security (task->sec) and the subjective security (task->act_as). The
    objective security holds the intrinsic security properties of a process and is
    never overridden. This is what appears in /proc, and is used when a process is
    the target of an operation by some other process (SIGKILL for example).

    The subjective security holds the active security properties of a process, and
    may be overridden. This is not seen externally, and is used whan a process
    acts upon another object, for example SIGKILLing another process or opening a

    The new hooks allow SELinux (or Smack or whatever) to reject a request for a
    kernel service (such as cachefiles) to run in a context of a specific security
    label or to create files and directories with another security label. These
    hooks may also be useful for NFSd.

    > * new page-private destructor hook

    The cache may attach state to pages before read_cache_pages() is called.
    Therefore read_cache_pages() may need to arrange for it to be cleaned up. The
    only way it can know to do this is by examining the page flags. PG_private may
    not be overloaded because it is owned by fs/buffer.c and friends on things like

    > * probably other bits I missed

    Note that most of these things have been muchly argued over already.

    > Would it be correct to say that some of these changes are to support
    > disconnected operation?


    > You have some short snappers that look generally useful:
    > * add_wait_queue_tail (cool)

    Which you complained about above.

    > * write to a file without a struct file (includes ->mapping cleanup,
    > probably good)


    > * export fsync_super

    > Why not hunt around for existing in-kernel users that would benefit so
    > these can be submitted as standalone patches, shortening the remaining
    > patch set and partially overcoming objections due to core kernel
    > changes?

    The only ones that really fall into that category are the security patches,
    which admittedly affect a lot of places. That might be acceptable, and the
    thought has occurred to me, because of NFSd.

    > One thing I don't see is users coming on to lkml and saying "please
    > merge this, it works great for me". Since you probably have such
    > users, why not give them a poke?

    The problem is that I'm stuck on waiting for the NFS guys to okay the NFS

    > Your cachefilesd is going to need anti-deadlock medicine like ddsnap
    > has.

    You mean the userspace daemon? Why should it deadlock?

    > Since you don't seem at all worried about that right now, I suspect you have
    > not hammered this code really heavily, correct?

    I had run iozone on cached NFS prior to asynchronising it. However, I've found
    a bug in my thread pool code that I'm currently chasing, so I need to do more
    parallelisation testing

    > Without preventative measures, any memory-using daemon sitting in the block
    > IO path will deadlock if you hit it hard enough.

    cachefilesd doesn't actually seem to consume that much memory, and it's
    unlikely to deadlock as it only does one thing at once and has no locking.

    There is a potential race though between cachefilesd's cull scanner and someone
    scanning through all the files that are cached in the same order over and over
    again. The problem is that we cannot keep the view of old stuff in the cache
    up to date, no matter how hard we try. I haven't thought of a good way around

    > A couple of years ago you explained the purpose of the new page flags to
    > me and there is no way I can find that email again. Could you explain
    > it again please? Meanwhile I am doing my duty and reading your OLS
    > slides etc.

    See above.


     \ /
      Last update: 2008-02-26 03:05    [W:0.092 / U:71.464 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site