lkml.org 
[lkml]   [1999]   [Oct]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectProblem with I/O counts in ll_rw_blk.c?
Date
From
While working on a block device hot migration utility, I've come
across what looks like a problem in the way block requests are
handled/merged/counted/something in ll_rw_blk.c. I have added some
code to add_request which gets device, sector, nr_sectors, and cmd
(READ or WRITE) for each request added and records it somewhere that
a userland program can find it (via bufflink, which I released on
this list last week).

After reading Rubini's book and from looking at the code myself, it
looks as though req.nr_sectors should be the total number of sectors
in the request (the sum of the sizes of linked adjacent-on-disk buffer
heads within the request ). req.current_nr_sectors is the number of
sectors in the first buffer head of each request, bascially a hoisted
copy of bh->b_size >> 9 for drivers that don't want to implement
clustering. The code for updating the kstat.dk_driver_{r,w}{io,blk}[]
stats appears to confirm that understanding.

The problem is that the only requests logged for activity on an ext2
filesystem (1K block size) are ones with nr_sectors of 2 and the
number of spacing of those requests show that each request is actually
carrying much more data (typically 128 sectors) which is somehow
merged into the request but not reflected in the nr_sectors count.
Reading the merging code in add_request and make_request shows that
nr_sectors is updated in most obvious places where necessary but there
must be something I'm missing that isn't updating it or else I'm
misunderstanding something (another likely possibility). Looking at
various block drivers, I can't find anything relying on nr_sectors so
maybe driver authors already implicitly "know" this. Indeed, floppy.c
has a buffer_chain_size function that walks the buffer head list of
the current request to work out the number of adjacent sectors.

As evidence, here are the disk writes queued by "cp 64k 64k.again"
(where 64k is a 64KB file and cp uses a 4KB buffer for syswrite which
is irrelevant here because it's left to sync to dribble out dirty
buffers from sync_buffers or some such, I assume):

Before: disk_wio 28884, disk_wblk 57768
Request log: requests, all with nr_sectors = 2, for sectors:
32794 32792 4 33306 98326 107936 108064 32792 4 33306 98326 108066
108194 2
After: disk_wio 28891, disk_wblk 57782

So the 64KB (plus metadata and superblock updates) was apparently
written with 7 I/Os and 14 sectors (disk_{r,w}blk counts the total
nr_sectors from each request). As a further data point, writing to
a file opened with O_SYNC (which makes ext2 use ll_rw_block with a
long buffer list instead of the single buffers that sync_buffers
passes to ll_rw_block each time) also shows the oddity. Creating a
64KB file by using 16 writes of 4KB each (the file opened with
O_SYNC) gives 85 I/Os and 170 sectors. This sounds more reasonable
until you see the sector numbers that each 1KB (nr_sectors = 2)
request is issued for:

32794 32794 32794 32794 108196 32794 32794 32794 32794 108204 32794
32794 32794 32794 108212 32794 108220 108220 108220 108220 108222
108220 108220 108220 108220 108230 108220 108220 108220 108220 108220
108220 108220 108220 108246 108220 108220 108220 108220 108254 108220
108220 108220 108220 108262 108220 108220 108220 108220 108270 108220
108220 108220 108220 108278 108220 108220 108220 108220 108286 108220
108220 108220 108220 108294 108220 108220 108220 108220 108302 108220
108220 108220 108220 108310 108220 108220 108220 108220 108318

What's going on, please? This block device hot migration will let you
do a backup and/or migration of any live block device to a remote
system and, in particular, will let you backup or move a live database
from one system to another without any performance issues and only a
few seconds unavailability. This will work even for databases that use
disk partitions directly with no filesystem involved (and I intend to
persuade it to work with the recent raw I/O stuff too once I read it
to find out how it interacts with add_request: if it calls it already,
there's nothing further needed). You can do the same for NFS servers
and such like if you play the appropriate games with IP addresses.
Since it's at the block device level and not at the filesystem level,
you can do the same thing whatever filesystem the thing has on it (but
you can't backup/restore individual files of course :-).

--Malcolm

--
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Unix Systems Programmer
Oxford University Computing Services

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:54    [W:0.047 / U:0.016 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site