Messages in this thread | | | Date | Tue, 20 May 2014 10:26:46 -0400 | From | Benjamin LaHaise <> | Subject | Re: hanging aio process |
| |
On Tue, May 20, 2014 at 03:16:47PM +0200, Sebastian Ott wrote: > On Tue, 20 May 2014, Sebastian Ott wrote: > > On Mon, 19 May 2014, Benjamin LaHaise wrote: > > > It is entirely possible the bug isn't > > > caused by the referenced commit, as the commit you're pointing to merely > > > makes io_destroy() syscall wait for all aio outstanding to complete > > > before returning. > > > > I cannot reproduce this when I revert said commit (on top of 14186fe). If > > that matters - the arch is s390.
> Hm, ok - maybe that commit is really just highlighting a refcounting bug. > I just compared traces for a good and a few bad cases. The good case: ...
> (4 fio workers, free_ioctx_reqs is called 4 times)
> One of the bad cases: .... > (1 fio worker in D state, free_ioctx_reqs is called 3 times)
This would seem to indicate that the problem is not with Anatol's change, and the hang is a consequence of the AIO not completing. Can you trace calls to aio_complete() in addition to free_ioctx_reqs() to see if a completion is happening in the failed case? If aio_complete() is only getting called 3 times, the problem is not in the aio layer.
-ben
> Regards, > Sebastian > > > > > > > > git bisect points to: > > > > commit e02ba72aabfade4c9cd6e3263e9b57bf890ad25c > > > > Author: Anatol Pomozov <anatol.pomozov@gmail.com> > > > > Date: Tue Apr 15 11:31:33 2014 -0700 > > > > > > > > aio: block io_destroy() until all context requests are completed > > > > > > > > > > > > The fio workers are on the wait_for_completion in sys_io_destroy. > > > > > > > > Regards, > > > > Sebastian > > > > [global] > > > > blocksize=4K > > > > size=256M > > > > rw=randrw > > > > verify=md5 > > > > iodepth=32 > > > > ioengine=libaio > > > > direct=1 > > > > end_fsync=1 > > > > > > > > [file1] > > > > filename=/dev/scma > > > > > > > > [file2] > > > > filename=/dev/scmbw > > > > > > > > [file3] > > > > filename=/dev/scmc > > > > > > > > [file4] > > > > filename=/dev/scmx > > > > > > > > > -- > > > "Thought is the essence of where you are now." > > > > > > > >
-- "Thought is the essence of where you are now."
| |