lkml.org 
[lkml]   [2009]   [Oct]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 00/45] some writeback experiments
    On Wed, Oct 07, 2009 at 11:18:22PM +0800, Wu Fengguang wrote:
    > On Wed, Oct 07, 2009 at 09:47:14PM +0800, Peter Staubach wrote:
    > >
    > > > # vmmon -d 1 nr_writeback nr_dirty nr_unstable # (per 1-second samples)
    > > > nr_writeback nr_dirty nr_unstable
    > > > 11227 41463 38044
    > > > 11227 41463 38044
    > > > 11227 41463 38044
    > > > 11227 41463 38044
    >
    > I guess in the above 4 seconds, either client or (more likely) server
    > is blocked. A blocked server cannot send ACKs to knock down both

    Yeah the server side is blocked. The nfsd are mostly blocked in
    generic_file_aio_write(), in particular, the i_mutex lock! I'm copying
    one or two big files over NFS, so the i_mutex lock is heavily contented.

    I'm using the default wsize=4096 for NFS-root..

    wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
    329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod
    4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
    4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
    4693 4693 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
    4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
    4696 4696 TS - -5 24 1 0.0 D< log_wait_commit nfsd
    4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
    wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
    329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod
    4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4691 4691 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
    4692 4692 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd
    4693 4693 TS - -5 24 0 0.0 D< sync_buffer nfsd
    4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4695 4695 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
    4696 4696 TS - -5 24 1 0.0 D< generic_file_aio_write nfsd
    4697 4697 TS - -5 24 0 0.0 D< generic_file_aio_write nfsd

    wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
    329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod
    4690 4690 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4691 4691 TS - -5 24 0 0.1 D< get_request_wait nfsd
    4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4693 4693 TS - -5 24 0 0.1 S< svc_recv nfsd
    4694 4694 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4696 4696 TS - -5 24 0 0.1 S< svc_recv nfsd
    4697 4697 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd

    wfg ~% ps -o pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:24,comm ax|g nfs
    329 329 TS - -5 24 1 0.0 S< worker_thread nfsiod
    4690 4690 TS - -5 24 1 0.1 D< get_write_access nfsd
    4691 4691 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4692 4692 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4693 4693 TS - -5 24 1 0.1 D< generic_file_aio_write nfsd
    4694 4694 TS - -5 24 1 0.1 D< get_write_access nfsd
    4695 4695 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4696 4696 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd
    4697 4697 TS - -5 24 0 0.1 D< generic_file_aio_write nfsd

    Thanks,
    Fengguang

    > nr_writeback/nr_unstable. And the stuck nr_writeback will freeze
    > nr_dirty as well, because the dirtying process is throttled until
    > it receives enough "PG_writeback cleared" event, however the bdi-flush
    > thread is also blocked when trying to clear more PG_writeback, because
    > the client side nr_writeback limit has been reached. In summary,
    >
    > server blocked => nr_writeback stuck => nr_writeback limit reached
    > => bdi-flush blocked => no end_page_writeback() => dirtier blocked
    > => nr_dirty stuck
    >
    > Thanks,
    > Fengguang
    >
    > > > 11045 53987 6490
    > > > 11033 53120 8145
    > > > 11195 52143 10886
    > > > 11211 52144 10913
    > > > 11211 52144 10913
    > > > 11211 52144 10913
    > > >
    > > > btrfs seems to maintain a private pool of writeback pages, which can go out of
    > > > control:
    > > >
    > > > nr_writeback nr_dirty
    > > > 261075 132
    > > > 252891 195
    > > > 244795 187
    > > > 236851 187
    > > > 228830 187
    > > > 221040 218
    > > > 212674 237
    > > > 204981 237
    > > >
    > > > XFS has very interesting "bumpy writeback" behavior: it tends to wait
    > > > collect enough pages and then write the whole world.
    > > >
    > > > nr_writeback nr_dirty
    > > > 80781 0
    > > > 37117 37703
    > > > 37117 43933
    > > > 81044 6
    > > > 81050 0
    > > > 43943 10199
    > > > 43930 36355
    > > > 43930 36355
    > > > 80293 0
    > > > 80285 0
    > > > 80285 0
    > > >
    > > > Thanks,
    > > > Fengguang
    > > >
    > > > --
    > > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > > > the body of a message to majordomo@vger.kernel.org
    > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
    > > > Please read the FAQ at http://www.tux.org/lkml/


    \
     
     \ /
      Last update: 2009-10-08 07:41    [W:0.029 / U:0.380 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site