lkml.org 
[lkml]   [2011]   [Jul]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [RFC PATCH]vhost-blk: In-kernel accelerator for virtio block device
Hi
On 07/29/2011 12:48 PM, Stefan Hajnoczi wrote:
> On Thu, Jul 28, 2011 at 4:44 PM, Stefan Hajnoczi<stefanha@gmail.com> wrote:
>> On Thu, Jul 28, 2011 at 3:29 PM, Liu Yuan<namei.unix@gmail.com> wrote:
>>
>> Did you investigate userspace virtio-blk performance? If so, what
>> issues did you find?
>>
>> I have a hacked up world here that basically implements vhost-blk in userspace:
>> http://repo.or.cz/w/qemu/stefanha.git/blob/refs/heads/virtio-blk-data-plane:/hw/virtio-blk.c
>>
>> * A dedicated virtqueue thread sleeps on ioeventfd
>> * Guest memory is pre-mapped and accessed directly (not using QEMU's
>> usually memory access functions)
>> * Linux AIO is used, the QEMU block layer is bypassed
>> * Completion interrupts are injected from the virtqueue thread using ioctl
>>
>> I will try to rebase onto qemu-kvm.git/master (this work is several
>> months old). Then we can compare to see how much of the benefit can
>> be gotten in userspace.
> Here is the rebased virtio-blk-data-plane tree:
> http://repo.or.cz/w/qemu-kvm/stefanha.git/shortlog/refs/heads/virtio-blk-data-plane
>
> When I run it on my laptop with an Intel X-25M G2 SSD I see a latency
> reduction compared to mainline userspace virtio-blk. I'm not posting
> results because I did quick fio runs without ensuring a quiet
> benchmarking environment.
>
> There are a couple of things that could be modified:
> * I/O request merging is done to mimic bdrv_aio_multiwrite() - but
> vhost-blk does not do this. Try turning it off?

I noted bdrv_aio_multiwrite() do the murging job, but I am not sure if
this trick is really needed since we have an io scheduler down the path
that is in a much more better position to murge requests. I think the
duplicate *pre-mature* merging of bdrv_aio_multiwrite is the result of
laio_submit()'s lack of submitting the requests in a batch mode.
io_submit() in the fs/aio.c says that every time we call laio_submit(),
it will submit the very request into the driver's request queue, which
would be run when we blk_finish_plug(). IMHO, you can simply batch
io_submit() requests instead of this tricks if you already bypass the
QEMU block layer.

> * epoll(2) is used but perhaps select(2)/poll(2) have lower latency
> for this use case. Try another event mechanism.
>
> Let's see how it compares to vhost-blk first. I can tweak it if we
> want to investigate further.
>
> Yuan: Do you want to try the virtio-blk-data-plane tree? You don't
> need to change the qemu-kvm command-line options.
>
> Stefan
Yes, please, sounds interesting. BTW, I think the user space would
achieve the same performance gain if you bypass qemu io layer all the
way down to the system calls in a request handling cycle, compared to
the current vhost-blk implementation that uses linux AIO. But hey, I
would go further to optimise it with block layer and other resources in
the mind. ;) and I don't add complexity to the current qemu io layer.

Yuan


\
 
 \ /
  Last update: 2011-07-29 10:03    [from the cache]
©2003-2011 Jasper Spaans