lkml.org 
[lkml]   [2012]   [Dec]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRead starvation by sync writes

    Hi,

    I was looking into IO starvation problems where streaming sync writes (in
    my case from kjournald but DIO would look the same) starve reads. This is
    because reads happen in small chunks and until a request completes we don't
    start reading further (reader reads lots of small files) while writers have
    plenty of big requests to submit. Both processes end up fighting for IO
    requests and writer writes nr_batching 512 KB requests while reader reads
    just one 4 KB request or so. Here the effect is magnified by the fact that
    the drive has relatively big queue depth so it usually takes longer than
    BLK_BATCH_TIME to complete the read request. The net result is it takes
    close to two minutes to read files that can be read under a second without
    writer load. Without the big drive's queue depth, results are not ideal but
    they are bearable - it takes about 20 seconds to do the reading. And for
    comparison, when writer and reader are not competing for IO requests (as it
    happens when writes are submitted as async), it takes about 2 seconds to
    complete reading.

    Simple reproducer is:

    echo 3 >/proc/sys/vm/drop_caches
    dd if=/dev/zero of=/tmp/f bs=1M count=10000 &
    sleep 30
    time cat /etc/* 2>&1 >/dev/null
    killall dd
    rm /tmp/f

    The question is how can we fix this? Two quick hacks that come to my mind
    are remove timeout from the batching logic (is it that important?) or
    further separate request allocation logic so that reads have their own
    request pool. More systematic fix would be to change request allocation
    logic to always allow at least a fixed number of requests per IOC. What do
    people think about this?

    Honza
    --
    Jan Kara <jack@suse.cz>
    SUSE Labs, CR


    \
     
     \ /
      Last update: 2012-12-10 23:41    [W:3.480 / U:0.040 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site