lkml.org 
[lkml]   [2008]   [Apr]   [28]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
SubjectRe: stat benchmark
From
On Mon, Apr 28, 2008 at 07:53:22AM -0400, Theodore Tso wrote:
> On Sun, Apr 27, 2008 at 09:43:05PM -0700, Ulrich Drepper wrote:
> > On Thu, Apr 24, 2008 at 1:59 PM, Soeren Sandmann <sandmann@daimi.au.dk> wrote:
> > > So I am looking for ways to improve this.
> >
> > Aside from what has already been proposed there is also the
> > readdirplus() route. Unfortunately the people behind this and related
> > proposals vanished after the last discussions. I was hoping they
> > come back with a revised proposal but perhaps not. Maybe it's time to
> > pick up the ball myself.
> >
> > As a reminder, readdirplus() is an extended readdir() which also
> > returns (a subset of) the stat information for the file at the same
> > time. The subset part is needed to account for the different
> > information contained in the inodes. For most applications the subset
> > should be sufficient and therefore all that's needed is a single
> > iteration over the directory.
>
> I'm not sure this would help in the cold cache case, which is what
> Soeren originally complained about.[1] The problem is whaever
> information the user might need won't be store in the directory, so
> the filesystem would end having to stat the file anyway, incurring a
> disk seek, which was what the user was complaining about. A
> readdirplus() would save a whole bunch of system calls if the inode
> was already cached, yes, but I'm not sure that's it would be worth the
> effort given how small Linux's system call overhead would be. But in
> the cold cache case, you end up seeking all over the disk, and the
> only thing you can do is to try to keep the inodes close to each
> other, and to have either readdir() or the caller of readdir() sort
> all of the returned directory entries by inode number to avoid seeking
> all over the disk.

The other reason for something like a readdirplus or a bulk stat is to
provide an opportunity for parallelism.

As my favorite example: cold-cache "git diff" of a linux tree on my
desktop (with an nfs-mounted /home) takes about 12 seconds. That's
mainly just a sequential stat of about about 24000 files. Patching git
to issue the stats in parallel, I could get that down to about 3.5
seconds. (Still not great. I don't know if it's disk seeks on the
server or what that are the limiting factor.)

In the case of git, it's looking just for files that it tracks--it's not
reading whole directories--so I don't know if readdirplus() specifically
would help.

--b.


\
 
 \ /
  Last update: 2008-04-28 18:21    [W:0.200 / U:0.036 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site