lkml.org 
[lkml]   [2011]   [Jun]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Subject[RFC PATCH 0/8] blk-throttle: Throttle buffered WRITE in balance_dirty_pages()
Date
Hi,

I have been trying to find ways to solve two problems with block IO controller
cgroups.

- Current throttling logic in IO controller does not throttle buffered WRITES.
Well it does throttle all the WRITEs at device and by that time buffered
WRITE have lost the submitter's context and most of the IO comes in
flusher thread's context at device. Hence currently buffered write
throttling is not supported.

- All WRITEs are throttled at device level and this can easily lead to
filesystem serialization.

One simple example is that if a process writes some pages to cache and
then does fsync(), and process gets throttled then it locks up the
filesystem. With ext4, I noticed that even a simple "ls" does not make
progress. The reason boils down to the fact that filesystems are not
aware of cgroups and one of the things which get serialized is journalling
in ordered mode.

So even if we do something to carry submitter's cgroup information
to device and do throttling there, it will lead to serialization of
filesystems and is not a good idea.

So how to go about fixing it. There seem to be two options.

- Throttling should still be done at device level. Make filesystems aware
of cgroups so that multiple transactions can make progress in parallel
(per cgroup) and there are no shared resources across cgroups in
filesystems which can lead to serialization.

- Throttle WRITEs while they are entering the cache and not after that.
Something like balance_dirty_pages(). Direct IO is still throttled
at device level. That way, we can avoid these journalling related
serialization issues w.r.t trottling.

But the big issue with this approach is that we control the IO rate
entering into the cache and not IO rate at the device. That way it
can happen that flusher later submits lots of WRITEs to device and
we will see a periodic IO spike on end node.

So this mechanism helps a bit but is not the complete solution. It
can primarily help those folks which have the system resources and
plenty of IO bandwidth available but they don't want to give it to
customer because it is not a premium customer etc.

Option 1 seem to be really hard to fix. Filesystems have not been written
keeping cgroups in mind. So I am really skeptical that I can convince file
system designers to make fundamental changes in filesystems and journalling
code to make them cgroup aware.

Hence with this patch series I have implemented option 2. Option 2 is not
the best solution but atleast it gives us some control then not having any
control on buffered writes. Andrea Righi did similar patches in the past
here.

https://lkml.org/lkml/2011/2/28/115

This patch series had issues w.r.t to interaction between bio and task
throttling, so I redid it.

Design
------
IO controller already has the capability to keep track of IO rates of
a group and enqueue the bio in internal queues if group exceeds the
rate and dispatch these bios later.

This patch series also introduce the capability to throttle a dirtying
task in balance_dirty_pages_ratelimited_nr(). Now no WRITES except
direct WRITES will be throttled at device level. If a dirtying task
exceeds its configured IO rate, it is put on a group wait queue and
woken up when it can dirty more pages.

No new interface has been introduced and both direct IO as well as buffered
IO make use of common IO rate limit.

How To
=====
- Create a cgroup and limit it to 1MB/s for writes.
echo "8:16 1024000" > /cgroup/blk/test1/blkio.throttle.write_bps_device
- Launch dd thread in the cgroup
dd if=/dev/zero of=zerofile bs=4K count=1K

1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 4.00428 s, 1.0 MB/s

Any feedback is welcome.

Thanks
Vivek

Vivek Goyal (8):
blk-throttle: convert wait routines to return jiffies to wait
blk-throttle: do not enforce first queued bio check in
tg_wait_dispatch
blk-throttle: use IO size and direction as parameters to wait
routines
blk-throttle: Specify number of IOs during dispatch update
blk-throttle: Get rid of extend slice trace message
blk-throttle: core logic to throttle task while dirtying pages
blk-throttle: Do not throttle WRITEs at device level except direct IO
blk-throttle: enable throttling of task while dirtying pages

block/blk-cgroup.c | 6 +-
block/blk-cgroup.h | 2 +-
block/blk-throttle.c | 506 +++++++++++++++++++++++++++++++++++---------
block/cfq-iosched.c | 2 +-
block/cfq.h | 6 +-
fs/direct-io.c | 1 +
include/linux/blk_types.h | 2 +
include/linux/blkdev.h | 5 +
mm/page-writeback.c | 3 +
9 files changed, 421 insertions(+), 112 deletions(-)
--
1.7.4.4



\
 
 \ /
  Last update: 2011-06-03 23:11    [from the cache]
©2003-2011 Jasper Spaans