lkml.org 
[lkml]   [2014]   [May]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: hanging aio process
    On Tue, May 20, 2014 at 03:16:47PM +0200, Sebastian Ott wrote:
    > On Tue, 20 May 2014, Sebastian Ott wrote:
    > > On Mon, 19 May 2014, Benjamin LaHaise wrote:
    > > > It is entirely possible the bug isn't
    > > > caused by the referenced commit, as the commit you're pointing to merely
    > > > makes io_destroy() syscall wait for all aio outstanding to complete
    > > > before returning.
    > >
    > > I cannot reproduce this when I revert said commit (on top of 14186fe). If
    > > that matters - the arch is s390.

    > Hm, ok - maybe that commit is really just highlighting a refcounting bug.
    > I just compared traces for a good and a few bad cases. The good case:
    ...

    > (4 fio workers, free_ioctx_reqs is called 4 times)


    > One of the bad cases:
    ....
    > (1 fio worker in D state, free_ioctx_reqs is called 3 times)

    This would seem to indicate that the problem is not with Anatol's change,
    and the hang is a consequence of the AIO not completing. Can you trace
    calls to aio_complete() in addition to free_ioctx_reqs() to see if a
    completion is happening in the failed case? If aio_complete() is only
    getting called 3 times, the problem is not in the aio layer.

    -ben


    > Regards,
    > Sebastian
    >
    > > >
    > > > > git bisect points to:
    > > > > commit e02ba72aabfade4c9cd6e3263e9b57bf890ad25c
    > > > > Author: Anatol Pomozov <anatol.pomozov@gmail.com>
    > > > > Date: Tue Apr 15 11:31:33 2014 -0700
    > > > >
    > > > > aio: block io_destroy() until all context requests are completed
    > > > >
    > > > >
    > > > > The fio workers are on the wait_for_completion in sys_io_destroy.
    > > > >
    > > > > Regards,
    > > > > Sebastian
    > > > > [global]
    > > > > blocksize=4K
    > > > > size=256M
    > > > > rw=randrw
    > > > > verify=md5
    > > > > iodepth=32
    > > > > ioengine=libaio
    > > > > direct=1
    > > > > end_fsync=1
    > > > >
    > > > > [file1]
    > > > > filename=/dev/scma
    > > > >
    > > > > [file2]
    > > > > filename=/dev/scmbw
    > > > >
    > > > > [file3]
    > > > > filename=/dev/scmc
    > > > >
    > > > > [file4]
    > > > > filename=/dev/scmx
    > > >
    > > >
    > > > --
    > > > "Thought is the essence of where you are now."
    > > >
    > > >
    > >

    --
    "Thought is the essence of where you are now."


    \
     
     \ /
      Last update: 2014-05-20 17:01    [W:4.270 / U:0.016 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site