lkml.org 
[lkml]   [2008]   [Jul]   [2]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: nfs client readdir caching issue?
    From
    Date
    On Wed, 2008-07-02 at 12:03 +0100, Andy Chittenden wrote:
    > Very rarely, we're seeing various problems on a linux kernel client
    > (seen on various versions) with ls on directories from an NFS server
    > that haven't changed:
    >
    > * looping ls (strace -v shows getdents returning the same names over
    > again).
    > * duplicate directory entries.
    > * missing directory entries.
    >
    > I've hunted google but can only see problems where NFS servers have
    > returned duplicate cookies. I've packet captured the readdirplus on one
    > of the directories and see no duplicate cookies. The problems remain
    > until the directory is touched, the NFS server is unmounted or some
    > other event happens (the data is flushed from the cache?).
    >
    > I think we then got lucky and got two packet captures from different
    > clients running the same linux kernel. On these clients, the ls output
    > was ok - no loops, no duplicates, no missing entries. Both captures
    > showed two readdirplus requests returning the same entries in the same
    > order but the amount of data in the responses was different. One capture
    > showed the server returned 1724 bytes, 10 entries, last cookie of 12,
    > followed by the next readdirplus returning a length of 948 bytes, 5
    > entries, a first cookie value of 13. In the other capture, the responses
    > returned 2204 bytes, 13 entries, a last cookie of 17 and 468 bytes, 2
    > entries, a first cookie of 19.
    >
    > In the past we've found that ls has returned duplicate entries on this
    > directory (but didn't have a capture at the time) and those duplicate
    > entries are the ones that are returned as the last 3 entries in the
    > first response of the second capture and the first 3 entries in the
    > second response of the first capture.
    >
    > So what I think has happened in this particular case, is that at some
    > point in the past, the directory was read OK with packets similar to the
    > first capture. Next, the client decided to get rid of the first page of
    > cached readdir responses from memory for some reason (running low on
    > memory?) but kept the second page. Subsequently, the readdir cache needs
    > repopulating so the client sends a readdirplus specifying cookie of 0
    > and this time it gets a response which is similar to the first packet of
    > the second capture and thus we now have in cache duplicate names and
    > cookie values.
    >
    > So is this possible? Is there some easy way to provoke it? Does this
    > mean the client's readdir cache is broken?

    If so, then invalidate_inode_pages2_range() would have to be broken: we
    always clear the readdir cache immediately after reading in the page
    with index 0 (i.e. the first readdir page).

    It shouldn't be possible for another thread to race with that cache
    invalidation either since the entire readdir() call is protected by the
    parent directory's inode->i_mutex.

    Cheers
    Trond



    \
     
     \ /
      Last update: 2008-07-02 19:39    [W:0.037 / U:0.276 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site