Messages in this thread | | | Subject | Problem with I/O counts in ll_rw_blk.c? | Date | Mon, 25 Oct 1999 14:53:22 +0100 (BST) | From | Malcolm Beattie <> |
| |
While working on a block device hot migration utility, I've come across what looks like a problem in the way block requests are handled/merged/counted/something in ll_rw_blk.c. I have added some code to add_request which gets device, sector, nr_sectors, and cmd (READ or WRITE) for each request added and records it somewhere that a userland program can find it (via bufflink, which I released on this list last week).
After reading Rubini's book and from looking at the code myself, it looks as though req.nr_sectors should be the total number of sectors in the request (the sum of the sizes of linked adjacent-on-disk buffer heads within the request ). req.current_nr_sectors is the number of sectors in the first buffer head of each request, bascially a hoisted copy of bh->b_size >> 9 for drivers that don't want to implement clustering. The code for updating the kstat.dk_driver_{r,w}{io,blk}[] stats appears to confirm that understanding.
The problem is that the only requests logged for activity on an ext2 filesystem (1K block size) are ones with nr_sectors of 2 and the number of spacing of those requests show that each request is actually carrying much more data (typically 128 sectors) which is somehow merged into the request but not reflected in the nr_sectors count. Reading the merging code in add_request and make_request shows that nr_sectors is updated in most obvious places where necessary but there must be something I'm missing that isn't updating it or else I'm misunderstanding something (another likely possibility). Looking at various block drivers, I can't find anything relying on nr_sectors so maybe driver authors already implicitly "know" this. Indeed, floppy.c has a buffer_chain_size function that walks the buffer head list of the current request to work out the number of adjacent sectors.
As evidence, here are the disk writes queued by "cp 64k 64k.again" (where 64k is a 64KB file and cp uses a 4KB buffer for syswrite which is irrelevant here because it's left to sync to dribble out dirty buffers from sync_buffers or some such, I assume):
Before: disk_wio 28884, disk_wblk 57768 Request log: requests, all with nr_sectors = 2, for sectors: 32794 32792 4 33306 98326 107936 108064 32792 4 33306 98326 108066 108194 2 After: disk_wio 28891, disk_wblk 57782
So the 64KB (plus metadata and superblock updates) was apparently written with 7 I/Os and 14 sectors (disk_{r,w}blk counts the total nr_sectors from each request). As a further data point, writing to a file opened with O_SYNC (which makes ext2 use ll_rw_block with a long buffer list instead of the single buffers that sync_buffers passes to ll_rw_block each time) also shows the oddity. Creating a 64KB file by using 16 writes of 4KB each (the file opened with O_SYNC) gives 85 I/Os and 170 sectors. This sounds more reasonable until you see the sector numbers that each 1KB (nr_sectors = 2) request is issued for:
32794 32794 32794 32794 108196 32794 32794 32794 32794 108204 32794 32794 32794 32794 108212 32794 108220 108220 108220 108220 108222 108220 108220 108220 108220 108230 108220 108220 108220 108220 108220 108220 108220 108220 108246 108220 108220 108220 108220 108254 108220 108220 108220 108220 108262 108220 108220 108220 108220 108270 108220 108220 108220 108220 108278 108220 108220 108220 108220 108286 108220 108220 108220 108220 108294 108220 108220 108220 108220 108302 108220 108220 108220 108220 108310 108220 108220 108220 108220 108318
What's going on, please? This block device hot migration will let you do a backup and/or migration of any live block device to a remote system and, in particular, will let you backup or move a live database from one system to another without any performance issues and only a few seconds unavailability. This will work even for databases that use disk partitions directly with no filesystem involved (and I intend to persuade it to work with the recent raw I/O stuff too once I read it to find out how it interacts with add_request: if it calls it already, there's nothing further needed). You can do the same for NFS servers and such like if you play the appropriate games with IP addresses. Since it's at the block device level and not at the filesystem level, you can do the same thing whatever filesystem the thing has on it (but you can't backup/restore individual files of course :-).
--Malcolm
-- Malcolm Beattie <mbeattie@sable.ox.ac.uk> Unix Systems Programmer Oxford University Computing Services
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |