[lkml]   [2011]   [May]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: REQ_FLUSH, REQ_FUA and open/close of block devices

    --On 22 May 2011 06:44:49 -0400 Christoph Hellwig <> wrote:

    > On Sat, May 21, 2011 at 09:42:45AM +0100, Alex Bligh wrote:
    >> What I am concerned about is that relatively normal actions (e.g. unmount
    >> a filing system) do not appear to be flushing all data, even though I
    >> did "sync" then "umount". I suspect the sync is generating the FLUSH
    >> here, and nothing is flushing the umount writes. How can I know as a
    >> block device that I have to write out a (long lasting) writeback cache if
    >> I don't receive anything beyond the last WRITE?
    > In your case it seems like ext3 is doing something wrong. If you
    > run the same on XFS, you should not only see the last real write
    > having FUA and FLUSH as it's a transaction commit, but also an
    > explicit cache flush when devices are closed from the filesystem
    > to work around issues like that.

    OK. Sounds like an ext3 bug then. I will test with xfs, ext4 and btrfs
    and see if they exhibit the same symptoms, and come back with a more
    appropriate subject line.

    > But the raw block device node
    > really doesn't behave different from a file and shouldn't cause
    > any fsync on close.

    Fair enough. I will check whether the hypervisor concerned is doing
    an fsync() or equivalent in the right place.

    > Btw, using sync_file_range is a really bad idea. It will not actually
    > flush the disk cache on the server, nor make sure metadata is commited in
    > case of a sparse or preallocated file, and thus does not implement
    > the FLUSH or FUA semantics correctly.
    > And btw, I'd like to know what makes sync_file_range so tempting,
    > even after I added documentation explaining why it's almost always
    > wrong to use it to the man page.

    I think you are referring to this (which in my defence wasn't in my
    local copy of the manpage).

    > This system call is extremely dangerous and should not be used in
    > portable programs. None of these operations writes out the file's
    > metadata. Therefore, unless the application is strictly performing
    > overwrites of already- instantiated disk blocks, there are no
    > guarantees that the data will be available after a crash. There is no
    > user interface to know if a write is purely an overwrite. On file
    > systems using copy-on-write semantics (e.g., btrfs) an overwrite of
    > existing allocated blocks is impossible. When writing into preallocated
    > space, many file systems also require calls into the block allocator,
    > which this system call does not sync out to disk. This system call
    > does not flush disk write caches and thus does not provide any data
    > integrity on systems with volatile disk write caches.

    So, the file in question is not mmap'd (it's an nbd disk). fsync() /
    fdatasync() is too expensive as it will sync everything. As far as I can
    tell, this is no more dangerous re metadata than fdatasync() which also
    does not sync metadata. I had read the last sentence as "this system
    call does not *necessarily* flush disk write caches" (meaning "if you
    haven't mounted e.g. ext3 with barriers=1, then you can't ensure write
    caches write through"), as opposed to "will not ever flush disk write
    caches", and given mounting ext3 without barriers=1 produces no FUA or
    FLUSH commands in normal operation anyway (as far as light debugging
    can see) that's not much of a loss.

    But rather than trying to justify myself: what is the best way to
    emulate FUA, i.e. ensure a specific portion of a file is synced before
    returning, without ensuring the whole lot is synced (which is far too
    slow)? The only other option I can see is to open the file with a second
    fd, mmap the chunk of the file (it may be larger than the available
    virtual address space), mysnc it with MS_SYNC, then fsync, then munmap
    and close, and hope the fsync doesn't spit anything else out. This
    seems a little excessive, and I don't even know whether it would work.

    I guess given NBD currently does nothing at all to support barriers,
    I thought this was an improvement!

    Alex Bligh

     \ /
      Last update: 2011-05-22 13:19    [W:0.030 / U:8.500 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site