lkml.org 
[lkml]   [2010]   [Apr]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: endless sync on bdi_sched_wait()? 2.6.33.1
    On Thu, Apr 08, 2010 at 11:28:50AM +0200, Jan Kara wrote:
    > On Wed 31-03-10 19:07:31, Denys Fedorysychenko wrote:
    > > I have a proxy server with "loaded" squid. On some moment i did sync, and
    > > expecting it to finish in reasonable time. Waited more than 30 minutes, still
    > > "sync". Can be reproduced easily.
    ....
    > >
    > > SUPERPROXY ~ # cat /proc/1753/stack
    > > [<c019a93c>] bdi_sched_wait+0x8/0xc
    > > [<c019a807>] wait_on_bit+0x20/0x2c
    > > [<c019a9af>] sync_inodes_sb+0x6f/0x10a
    > > [<c019dd53>] __sync_filesystem+0x28/0x49
    > > [<c019ddf3>] sync_filesystems+0x7f/0xc0
    > > [<c019de7a>] sys_sync+0x1b/0x2d
    > > [<c02f7a25>] syscall_call+0x7/0xb
    > > [<ffffffff>] 0xffffffff
    > Hmm, I guess you are observing the problem reported in
    > https://bugzilla.kernel.org/show_bug.cgi?id=14830
    > There seem to be several issues in the per-bdi writeback code that
    > cause sync on a busy filesystem to last almost forever. To that bug are
    > attached two patches that fix two issues but apparently it's not all.
    > I'm still looking into it...

    Jan, just another data point that i haven't had a chance to look
    into yet - I noticed that 2.6.34-rc1 writeback patterns have changed
    on XFS from looking at blocktrace.

    The bdi-flush background write threadi almost never completes - it
    blocks in get_request() and it is doing 1-2 page IOs. If I do a
    large dd write, the writeback thread starts with 512k IOs for a
    short while, then suddenly degrades to 1-2 page IOs that get merged
    in the elevator to 512k IOs.

    My theory is that the inode is getting dirtied by the concurrent
    write() and the inode is never moving back to the dirty list and
    having it's dirtied_when time reset - it's being moved to the
    b_more_io list in writeback_single_inode(), wbc->more_io is being
    set, and then we re-enter writeback_inodes_wb() which splices the
    b_more_io list back onto the b_io list and we try to write it out
    again.

    Because I have so many dirty pages in memory, nr_pages is quite high
    and this pattern continues for some time until it is exhausted, at
    which time throttling triggers background sync to run again and the
    1-2 page IO pattern continues.

    And for sync(), nr_pages is set to LONG_MAX, so regardless of how
    many pages were dirty, if we keep dirtying pages it will stay in
    this loop until LONG_MAX pages are written....

    Anyway, that's my theory - if we had trace points in the writeback
    code, I could confirm/deny this straight away. First thing I need to
    do, though, is to forward port the original writeback tracng code
    Jens posted a while back....

    Cheers,

    Dave.
    --
    Dave Chinner
    david@fromorbit.com


    \
     
     \ /
      Last update: 2010-04-13 03:25    [W:0.023 / U:32.036 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site