lkml.org 
[lkml]   [2015]   [Jun]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
SubjectRe: Unexpected slow block device write IO performance compared to uncached, unsynced direct IO using stock kernels
From
On Tue, Jun 16, 2015 at 3:56 PM, Erik Cumps <erik.cumps@esaturnus.com> wrote:
>
> we are noticing some strange block device IO performance and our
> investigations are leading us away from the hardware and towards the
> kernel. This could be a simple tuning problem or a known issue, so
> before taking a deep dive down kernel sources and debian kernel patches
> we would like to rule out the simple things first.
>
> The context is a 16 GB 32-bit intel debian workstation, using an ext4
> filesystem with journalling, on a lvm SATA3 SSD disk, with relatively
> recent stock kernels from 3.2 onwards to 4.0, running some KVM virtual
> machines. The host system (so not the virual machines) shows sporadic
> extremely slow write performance (around 4 megabytes per second).
> However, if we use the debian 3.2.0 kernel this problem does not
> manifest itself.
>
> We've created a simple IO performance test script to investigate this.
> It is basically a smart wrapper around dd, copying data between the
> block device under test and a ramdisk filesystem, using flags to select
> the usage of cache, sync and direct io, clearing caches before the test
> and running the write and read tests three times to account for
> transient performance.
>
> The results of these tests are unexpected: we see the expected normal
> write performance when using uncached, unsynced, direct IO and very slow
> write performance using the regular cached IO. The difference is huge:
> it is sometimes two orders of magnitude!

Actually, it is the *synchronous*, direct IO that matches the expected
raw write performance of the device.

The "regular IO" test is doing roughly this:

echo 3 > /proc/sys/vm/drop_caches
dd if=ramdisk_file of=test_file bs=1M count=100
dd if=ramdisk_file of=test_file bs=1M count=100
dd if=ramdisk_file of=test_file bs=1M count=100

The direct IO test is doing roughly this:

echo 3 > /proc/sys/vm/drop_caches
dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100
dd if=ramdisk_file of=test_file oflag=sync,direct bs=1M count=100

> This seems to rule out the block device and IO controller being at
> fault. In fact, other tests showed the same performance discrepancy with
> an NFS mounted filesystem and a platter disk.
>
> We also noticed that when the performance is slow, if we shut down the
> KVM virtual machines the performance returns to normal.
>
> Maybe there is something going wrong with cache/buffer handling?
> Thanks for your insights.
>
> I've kept this mail intentionally free from too many technical details
> but I'll be happy to provide additional relevant info as required.

Regards,
Erik Cumps


\
 
 \ /
  Last update: 2015-06-18 10:01    [W:0.066 / U:0.088 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site