Messages in this thread | | | From | "Stephen C. Tweedie" <> | Date | Tue, 23 Mar 1999 00:25:43 +0000 (GMT) | Subject | Re: disk head scheduling |
| |
Hi,
On Mon, 22 Mar 1999 21:39:23 +0100 (MET), Gerard Roudier <groudier@club-internet.fr> said:
> Linus,
> I agree completely with all your proposals below. I just will try to > detail what I understood and would be happy to have:
> The per major mapping function (kdev_t), as you suggest, seems indeed a > excellent approach.
You want the driver to be able to define its own granularity. In particular, you want to let it define multiple queues per major if that happens to make sense to the driver (eg, one queue per controller for ide or scsi). The existing ll_rw_block lets you do this.
There is another issue which has not been brought up yet, and which I've suddenly become aware of because it is causing deadlocks in the nbd code (primarily when using loopback, but also possible if you have two hosts each nbd-exporting to the other). The problem is that we not only have one request pool, we also have only one thread which feeds dirty buffers to the request layer.
The result of these two limits combined is that bdflush can progress no faster than the slowest currently-active writer device. As soon as the request queue is full, we can only write as fast as requests become free. That's just a performance problem, but it leads to a deadlock too, any time satisfying a request causes a new write request to come in from user space.
This is not the old loopback deadlock on the request queue. What happens is that when you start generating a lot of dirty data against the nbd device, you eventually trigger the wakeup_bdflush(1) call in buffer.c and stall waiting for bdflush to complete. At this point, _all_ writes will also stall at the same point, since only after bdflush has started working will the too-many-dirty-buffers condition be made untrue.
In particular, the nbd server blocks. There are enough free requests to allow the underlying writes to succeed, but there is no worker thread able to actually schedule those writes to disk, since bdflush is busy stalled in get_request() satisfying the nbd_server requests.
Ultimately, I think the right way to fix the deadlock issue is for the nbd server to perform its writes synchronously, and preferably from a kernel server thread. As long as it doesn't rely on bdflush, there is no deadlock. However, we still have bdflush as a bottleneck, and any multi-queue system for device IOs will have to consider how to perform per-device write throttling in a way which does not allow slow devices to penalise faster ones.
--Stephen
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |