[lkml]   [2010]   [Jun]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [PATCH v2] d_ino considered harmful
    On Wed, 2010-06-16 at 15:54 -0400, Valerie Aurora wrote:
    > On Wed, Jun 16, 2010 at 02:59:13PM -0400, Valerie Aurora wrote:
    > > Who needs d_ino anyway? I am running a kernel with this patch -
    > > Gnome, a browser, IRC, kernel compile, etc. and everything works.

    For large-scale Lustre users, not having a valid d_ino in the dirent is
    a problem when we need to put a name (or names) to an inode number.

    >From time to time, we have a problem reported in the logs against an
    inode and need to figure out the name of that file. stat() is very slow
    on these filesystems due to the need to talk to multiple servers to get
    file size information. When we purge the filesystem, it is faster to do
    a scan of the inode table looking for old files than it is to walk the
    tree. Then we have to match those inodes to names for unlink().

    For example, our main Lustre scratch space has over 285 million files in
    it, and using find -inum takes over 72 hours to walk the tree using
    stat(). Using a scanner that takes advantage of d_ino allows us to cover
    that ground in 13 hours. It's not perfectly apples-to-apples, as the
    custom scanner has some parallelism, but it gives you an idea of the
    problem's scale. Certainly, using d_ino reduces the number of RPCs we
    have to send.

    Using ne2scan -- which uses libext2fs and combines the inode scan and
    the name lookup -- takes over 48 hours to generate a list of candidate
    files for the purge example. With an optimized inode scan and the custom
    scanner above, it should take 18 hours, which is a considerable time
    saver for us.

    I can see that there are pitfalls in the presence of mountpoint and
    such, but we control the environment on these machines, and using d_ino
    is a huge win for us.
    Dave Dillow
    National Center for Computational Science
    Oak Ridge National Laboratory
    (865) 241-6602 office

     \ /
      Last update: 2010-06-16 23:03    [W:0.020 / U:140.336 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site