lkml.org 
[lkml]   [2007]   [Apr]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: If not readdir() then what?
On Thursday April 12, joern@lazybastard.org wrote:
> On Thu, 12 April 2007 11:46:41 +1000, Neil Brown wrote:
> >
> > I could argue that nfs came before ext3+dirindex, so ext3 should have
> > been designed to work properly with NFS. You could argue that fixing
> > it in nfsd fixes it for all filesystems. But I'm not sure either of
> > those arguments are likely to be at all convincing...
>
> Caring about a non-ext3 filesystem, I sure would like an nfs solution as
> well. :)

I have a non-ext3 filesystem I care about too.....

But my perspective is that a solution in nfsd at-best a work-around.
Caching the whole 'struct file' when there is just a small bit that we
might want seems like a heavy hammer. The filesystem is in the best
place to know what needs to be cached, and it should be the one doing
the caching.

>
> > Hmmm. I wonder. Which is more likely?
> > - That two 64bit hashes from some set are the same
> > - or that 65536 48bit hashes from a set of equal size are the same.
>
> The former. Each bit going from hash strength to collision chain length
> reduces the likelihood of an overflow. In the extreme case of a 0bit
> hash and 64bit collision chain, you need 2^64 entries compared to 2^32
> for the other extreme.
>
> However, the collision chain gives me quite a bit of headache. One
> would have to store each entry's position on the chain, deal with older
> entries getting deleted, newer entries getting removed, etc. All this
> requires a lot of complicated code that basically never gets tested in
> the wild.

This is a simple consequence of the design decision to use hashes as
the search key. They aren't dense and they will collide. So the
solution will be a bit fuzzy around the edges. And maybe that is an
acceptable tradeoff. But the filesystem should take full
responsibility for it, whether in performance or correctness :-)

>
> Just settling for a 64bit hash and returning -EEXIST when someone causes
> a collision an creat() sounds more appealing. Directories with 4
> billion entries will cause problems, but that is hardly news to anyone.
>

I think you want -EFBIG or -ENOSPC. -EEXIST sounds just wrong.

But there are alternatives. e.g. internal chaining.
Insist on a unique 64bit hash for every file. If the hash is in use,
increment and try again. On lookup, if the hash leads you to a file
with the wrong name, increment and try again until you find a hole
(hash value that is not stored). When you delete an entry, leave a
place holder if the next hash is in use. Conversely if the next hash
is not in use, delete the entry and delete the previous one if it is a
place holder.

Then you get 100% correct semantics and a performance hit in the face
of hash collisions that is probably no worse than that which ext3
currently gets. It probably does cost you a bit of storage to store
those 64bit hashes, though I suspect some clever compression can help
out there (You only need one bit more than the filename when there is
no chaining).

You have to require 64bit cookies/fpos, but I think that today, that
is a reasonable thing to require (5 years ago it might not have been).

NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2007-04-12 08:43    [W:0.134 / U:0.084 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site