lkml.org 
[lkml]   [2008]   [Feb]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    SubjectRe: [PATCH 00/37] Permit filesystem local caching
    Date

    Daniel Phillips <phillips@phunq.net> wrote:

    > This factor of four (even worse on XFS, not quite as bad on Ext3) is
    > worth ruminating upon. Is all of the difference explained by avoiding
    > seeks on the server, which has the files in memory?

    Here are some more stats for you to consider:

    (1) Copy the data across the network to a fresh Ext3 fs on the same partition
    I was using for the cache:

    [root@andromeda ~]# time cp -a /warthog/aaa /var/fscache
    real 0m39.052s
    user 0m0.368s
    sys 0m15.229s

    (2) Reboot and read back the files just written into Ext3 on the local disk:

    [root@andromeda ~]# time tar cf - /var/fscache/aaa >/dev/zero
    real 0m40.574s
    user 0m0.164s
    sys 0m3.512s

    (3) Run through the cache population process, and then run a tar directly on
    cachefiles's cache directly after a reboot:

    [root@andromeda ~]# time tar cf - /var/fscache/cache >/dev/zero
    real 4m53.104s
    user 0m0.192s
    sys 0m4.240s

    So I guess there's a problem in cachefiles's efficiency - possibly due to the
    fact that it tries to be fully asynchronous.

    In case (1) this is very similar to the time for a read through a completely
    cold cache (37.497s).

    In case (2) this is comparable to cachefiles with a cache warmed prior to a
    reboot (1m54.350s); in this case, however, cachefiles is doing some extra work:

    (a) It's doing a lookup on the server for each file, in addition to the
    lookups on the disk. However, just doing a tar from plain NFS, the
    command completes in 22.330s.

    (b) It's reading an xattr per object for cache coherency management.

    (c) As the cache knows nothing of directories, files, etc., it lays its
    directory subtree out in a way that suits it. File lookup keys are
    turned into filenames. This may result in a less efficient arrangement
    in the cache than the original data, especially as directories may become
    very large, so Ext3 may be doing some extra work.

    In case (3), this perhaps suggests that cachefiles's directory layout may be
    part of the problem. Running the following:

    ls -ldSr `find . -type d`

    in /var/fscache/cache shows that the directories are either 4096 bytes in size
    (158 instances) or 12288 bytes in size (105 instances), for a total of 263
    directories. There are 19255 files.

    Running that ls command in /warthog/aaa shows 1185 directories, all but three
    of them 4096 bytes in size; two are 12288 bytes and one is 20480 bytes in size
    (include/linux/ unsurprisingly). There are 19258 files, three of which are
    hardlinks to other files in the tree.

    > This could be easily tested by running a test against a server that is the
    > same as the client, and does not have the files in memory. If local access
    > is still slower than network then there is a real issue with cache
    > efficiency.

    My server is also my desktop machine. The only way to guarantee that the
    memory is scrubbed is to reboot it:-( I'll look at setting up one of my other
    machines as an NFS server.

    David


    \
     
     \ /
      Last update: 2008-02-26 00:23    [from the cache]
    ©2003-2014 Jasper Spaans. Advertise on this site