[lkml]   [2010]   [Apr]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    Patch in this message
    Subject[PATCH 16/35] union-mount: Writable overlays/union mounts documentation
    Document design and implementation of writable overlays (a.k.a. union

    XXX - out of date

    Signed-off-by: Valerie Aurora <>
    Documentation/filesystems/union-mounts.txt | 708 ++++++++++++++++++++++++++++
    1 files changed, 708 insertions(+), 0 deletions(-)
    create mode 100644 Documentation/filesystems/union-mounts.txt

    diff --git a/Documentation/filesystems/union-mounts.txt b/Documentation/filesystems/union-mounts.txt
    new file mode 100644
    index 0000000..5f47296
    --- /dev/null
    +++ b/Documentation/filesystems/union-mounts.txt
    @@ -0,0 +1,708 @@
    +State of writable overlays (formerly union mounts)
    +This version of union mounts is renamed "writable overlays." The goal
    +of this patch set is to support a single read-write file system
    +overlaid on a single read-only file system. "Union mounts" suggests
    +that we support unions of arbitrary numbers and types of file systems,
    +which is not the goal of this patch set.
    +The most recent version of writable overlays can boot to multi-user
    +mode with a writable overlay root file system. open(), truncate(),
    +creat(), unlink(), mkdir(), rmdir(), and rename() work. link(),
    +chmod(), chown(), and chattr() don't work yet.
    +This document describes the architecture and current status of
    +writable overlays, including an item-by-item todo list.
    +Writable overlays (formerly union mounts)
    +In this document:
    + - Overview of writable overlays
    + - Terminology
    + - VFS implementation
    + - Locking strategy
    + - VFS/file system interface
    + - Userland interface
    + - NFS interaction
    + - Status
    + - Contributing to writable overlays
    +Writable overlays (formerly known as union mounts) are used to layer a
    +single writable file system over a single read-only file system, with
    +all writes going to the writable file system. The namespace of both
    +file systems appears as a combined whole to userland, with those on
    +the writable file system covering up any matching pathnames on the
    +read-only file system. A few use cases:
    +- Root file system on CD with writes saved to hard drive (LiveCD)
    +- Multiple virtual machines with the same starting root file system
    +- Cluster with NFS mounted root on clients
    +Most if not all of these problems could be solved with a COW block
    +device; however, sharing at the file system level has higher
    +performance and uses less disk space.
    +What writable overlays are not
    +Writable overlays are not a general-purpose unioning file system.
    +They do not provide a generic "union of namespaces" operation for an
    +arbitrary number of file systems. Many interesting features can be
    +implemented with a generic unioning facility: unioning of more than
    +two file systems, dynamic insertion and removal of branches, online
    +upgrade, etc. Some unioning file systems that do this are UnionFS and
    +AUFS. Unfortunately, the complexity of these feature sets lead to
    +difficult corner cases which so far have been unsolvable in the
    +context of the Linux VFS.
    +Writable overlays avoid these corner cases by reducing the feature set
    +to the bare minimum most requested features: one writable file system
    +layered over one read-only file system. Despite the limitations of
    +writable overlays, the VFS infrastructure it uses are generic enough
    +to be reused by more full-featured unioning file systems.
    +The main analogy for writable overlays is that a writable file system
    +is mounted "on top" of a read-only file system. Lookups start at the
    +"top" read-write file system and travel "down" to the "bottom"
    +read-only file system only if no blocking entry exists on the top
    +Top layer: The read-write file system. Lookups begin here.
    +Bottom layer: The read-only file system. Lookups end here.
    +Path: Combination of the vfsmount and dentry structure.
    +Follow down: Given a path from the top layer, find the corresponding
    +path on the bottom layer.
    +Follow up: Given a path from the bottom layer, find the corresponding
    +path on the top layer.
    +Whiteout: A directory entry in the top layer that prevents lookups
    +from travelling down to the bottom layer. Created on unlink()/rmdir()
    +if a corresponding directory entry exists in the bottom layer.
    +Opaque: A flag on a directory in the top layer that prevents lookups
    +of entries in this directory from travelling down to the bottom
    +layer (unless there is an explicit fallthru entry allowing that for a
    +particular entry). Set on creation of a directory that replaces a
    +whiteout, and after a directory copyup.
    +Fallthru: A directory entry which allows lookups to "fall through" to
    +the bottom layer for that exact directory entry. This serves as a
    +placeholder for directory entries from the bottom layer during
    +readdir(). Fallthrus override opaque flags.
    +File copyup: Create a file on the top layer that has the same properties
    +and contents as the file with the same pathname on the bottom layer.
    +Directory copyup: Copy up the visible directory entries from the
    +bottom layer as fallthrus in the matching top layer directory. Mark
    +the directory opaque to avoid unnecessary negative lookups on the
    +bottom layer.
    +What happens when I...
    +- creat() /newfile -> creates on top layer
    +- unlink() /oldfile -> creates a whiteout on top layer
    +- Edit /existingfile -> copies up to top layer at open(O_WR) time
    +- truncate /existingfile -> copies up to top layer + N bytes if specified
    +- touch()/chmod()/chown()/etc. -> copies up to top layer
    +- mkdir() /newdir -> creates on top layer
    +- rmdir() /olddir -> creates a whiteout on top layer
    +- mkdir() /olddir after above -> creates on top layer w/ opaque flag
    +- readdir() /shareddir -> copies up entries from bottom layer as fallthrus
    +- link() /oldfile /newlink -> copies up /oldfile, creates /newlink on top layer
    +- symlink() /oldfile /symlink -> nothing special
    +- rename() /oldfile /newfile -> copies up /oldfile to /newfile on top layer
    +- rename() dir -> EXDEV
    +Getting to a root file system with a writable overlay:
    +- Mount the base read-only file system as the root file system
    +- Mount the read-only file system again on /newroot
    +- Mount the writable overlay on /newroot:
    + # mount -o union /dev/sda /newroot
    +- pivot_root to /newroot
    +- Start init
    +See scripts/ in the UML devkit linked to from:
    +VFS implementation
    +Writable overlays are implemented as an integral part of the VFS,
    +rather than as a VFS client file system (i.e., a stacked file system
    +like unionfs or ecryptfs). Implementing writable overlays inside the
    +VFS eliminates the need for duplicate copies of VFS data structures,
    +unnecessary indirection, and code duplication, but requires very
    +maintainable, low-to-zero overhead code. Writable overlays require no
    +change to file systems serving as the read-only layer, and requires
    +some minor support from file systems serving as the read-write layer.
    +File systems that want to be the writable layer must implement the new
    +->whiteout() and ->fallthru() inode operations, which create special
    +dummy directory entries.
    +union_mount structure
    +The primary data structure for writable overlays is the union_mount
    +structure, which connects overlapping directory dentries into a "union
    +struct union_mount {
    + atomic_t u_count; /* reference count */
    + struct mutex u_mutex;
    + struct list_head u_unions; /* list head for d_unions */
    + struct list_head u_list; /* list head for mnt_unions */
    + struct hlist_node u_hash; /* list head for searching */
    + struct hlist_node u_rhash; /* list head for reverse searching */
    + struct path u_this; /* this is me */
    + struct path u_next; /* this is what I overlay */
    +The union_mount is referenced from the corresponding directory's
    +struct dentry {
    + /*
    + * The following fields are used by the VFS based union mount
    + * implementation. Both are protected by union_lock!
    + */
    + struct list_head d_unions; /* list of union_mounts */
    + unsigned int d_unionized; /* unions referencing this dentry */
    +Each top layer directory with the potential for a lookup to fall
    +through to the bottom layer has a union_mount structure stored in a
    +union_mount hash table. The union_mount's can be looked up both by the
    +top layer's path (via union_lookup()) and the bottom layer's path (via
    +union_rlookup()). Once you have the path (vfsmount and dentry pair)
    +of a file, the union stack can be followed down, layer by layer, with
    +follow_union_down(), and up with follow_union_mount().
    +All union_mount's are allocated from a kmem cache when the
    +corresponding dentries are created. union_mount's are allocated when
    +the first referencing dentry is allocated and freed when all of the
    +referencing dentries are freed - that is, the dcache drives the union
    +cache. While writable overlays only use two layers, the union stack
    +infrastructure is capable of supporting an arbitrary number of file
    +system layers (leaving aside locking issues).
    +- Rename union_mount structure - it's per directory, not per mount
    +Code paths
    +Writable overlays modify the following key code paths in the VFS:
    +- mount()/umount()
    +- Path lookup
    +- Any path that modifies an existing file
    +Writable overlays are created in two steps:
    +1. Mount the bottom layer file system read-only in the usual manner.
    +2. Mount the top layer with the "-o union" option at the same mountpoint.
    +The bottom layer must be read-only and the top layer must be
    +read-write and support whiteouts and fallthrus (indicated by setting
    +the MS_WHITEOUT flag). Currently, the top layer is forced to
    +"noatime" to avoid a copyup on every access of a file. Supporting
    +atime with the current infrastructure would require a copyup on every
    +Currently, the top layer covers all submounts on the read-only file
    +system. This can be inconvenient; e.g., mounting a writable overlay
    +on the root file system after procfs has been mounted. It's not clear
    +what the right behavior is. Also, it may be smarter to mount both
    +read-only and read-write layers in one step, but the mount options get
    +pretty ugly.
    +pivot_root() is supported and is the recommended way to get to a root
    +file system with a writable overlay.
    +- Rename "-o union" mount option - "overlay"?
    +- Don't permit mounting over read-write submounts
    +- Choose submount covering behavior
    +- Allow atime?
    +Really really read-only file systems: In Linux, any individual file
    +system may be mounted at multiple places in the namespace. The file
    +system may change from read-only to read-write while still mounted.
    +Thus, simply checking that the bottom layer is read-only at the time
    +the writable overlay is mounted over it is pointless, since at any
    +time the bottom layer may become read-write.
    +We need to guarantee that a file system will be read-only for as long
    +as it is the bottom layer of a writable overlay. To do this, we track
    +the number of "read-only users" of a file system in its VFS superblock
    +structure. When we mount a writable overlay over a file system, we
    +increment its read-only user count. The file system can only be
    +mounted read-write if its read-only users count is zero.
    +- Support really really read-only NFS mounts. See discussion here:
    +Path lookup
    +Much of the action in writable overlasy happens during lookup().
    +First, if we lookup a directory on the bottom layer that doesn't yet
    +exist on the top layer, __link_path_walk() always create a matching
    +directory on the top layer. This way, we never have to walk back up a
    +path, creating directories as we go, before we can copyup a file.
    +Second, if we need to copy up a file, we first (re)look it up with the
    +LOOKUP_TOPMOST flag, which instructs __link_path_walk() to create it
    +on the top layer. Neither directory entries nor file data are copied
    +up in __link_path_walk() - that happens after the lookup, in the
    +The main cut-out to writable overlay code is in do_lookup():
    +static int do_lookup(struct nameidata *nd, struct qstr *name,
    + struct path *path)
    + int err;
    + if (IS_MNT_UNION(nd->path.mnt))
    + goto need_union_lookup;
    + err = cache_lookup_union(nd, name, path);
    + if (!err && path->dentry)
    + goto done;
    + err = real_lookup_union(nd, name, path);
    + if (err)
    + goto fail;
    + goto done;
    +cache_lookup_union() looks for the dentry in the dcache, starting at
    +the top layer and following down. If it finds nothing, it returns a
    +negative dentry from the top layer. If it finds a directory, it looks
    +for the same directory in the bottom layer; if that exists, it
    +allocates a union_mount struct and hangs the bottom layer dentry off
    +of it. real_lookup_union() does the same for uncached entries.
    +- Reorganize cache/hash/real lookup code - lots of code duplication
    +- Turn create-on-topmost test into #ifdef'able function
    +- Rewrite with assumption that topmost directory always exists
    +- Remove duplicated tests and other duplicated code
    +File copyup
    +Any system call that alters an existing file on the bottom layer
    +(including creating or moving a hard link to it) will trigger a copyup
    +of the target file to the top layer (via union_copyup() or
    +__union_copyup()). This includes:
    + - open(O_WRITE | O_RDWR | O_APPEND | O_DIRECT)
    + - truncate()/ftruncate()/open(O_TRUNC)
    + - link()
    + - rename()
    + - chmod()
    + - chattr()
    +Copyup of a file DOES NOT occur on:
    + - open(O_RDONLY) if noatime
    + - stat() if no atime
    + - creat()/mkdir()/mknod()
    + - symlink()
    + - unlink()/rmdir()
    +From an application's point of view, the result of an in-kernel file
    +copyup is the logical equivalent of another application updating the
    +file via the rename() pattern: creat() a new file, copy the data over,
    +make changes the copy, and rename() over the old version. Any
    +existing open file descriptors for that file (including those in the
    +same application) refer to a now invisible and unreferenced object
    +that used to have the same pathname. Only opens that occur after the
    +copyup will see updates to the file.
    +- copyup on chown()/chmod()/chattr()
    +- copyup if atime is enabled?
    +Permission checks
    +We want to be sure we have the correct permissions to actually succeed
    +in a system call before copying a file up to avoid unnecessary IO. At
    +present, the permission check for a single system call may be spread
    +out over many hundreds of lines of code (e.g., open()). In order to
    +check permissions, we occasionally need to determine if there is a
    +writable overlay on top of this inode. This requires a full path, but
    +often we only have the inode at this point. In particular,
    +inode_permission() returns EROFS if the inode is on a read-only file
    +system, which is the wrong answer if there is a writable overlay
    +mounted on top of it.
    +Another trouble-maker is may_open(), which both checks permissions for
    +open AND truncates the file if O_TRUNC is specified. It doesn't make
    +any sense to copy up the file and then let may_open() truncate it, but
    +we can't copy it after may_open() truncates it either. The current
    +ugly hack is to pass the full nameidata to may_open() and copyup
    +inside may_open().
    +Some solutions:
    +- Create __inode_permission() and pass it a flag telling it whether or
    + not to check for a read-only fs. Create union_permission() which
    + takes a path, checks for a union mount, and sets the rofs flag.
    + Place the file copyup call after all the permission checks are
    + completed. Push down the full path into the functions that need it
    + and currently only take the dentry or inode.
    +- For each instance in which we might want to copyup, move permission
    + checks into a new function and call it from a level at which we
    + still have the full path. Pass it an "ignore read-only fs" flag if
    + the file is on a union mount. Pass around the ignore-rofs flag
    + inside the function doing permission checks. If all the permission
    + checks complete successfully, copyup the file. Would require moving
    + truncate out of may_open().
    + - On truncate, only copy up the N bytes of file data requested
    + - Make sure above handles truncate beyond EOF correctly
    + - File copyup on chown()/chmod()/chattr() etc.
    + - File copyup on open(O_APPEND)
    + - File copyup on open(O_DIRECT)
    +Impact on non-union kernels and mounts
    +Union-related data structures, extra fields, and function calls are
    +#ifdef'd out at the function/macro level with CONFIG_UNION_MOUNT in
    +nearly all cases (see include/linux/union.h). The union-specific code
    +in the cache lookup path is out of line.
    +Currently, is_unionized() is pretty heavy-weight: it walks up the
    +mount hierarchy, grabbing the vfsmount lock at each level. It may be
    +possible to simplify this greatly if a writable layer can only cover
    +exactly one mount, rather than a tree of mounts.
    + - Turn copyup in __link_path_walk() into #ifdef'd function
    + - Do performance tests
    + - Optimize is_unionized()
    + - Properly #ifdef out mount path code
    +Locking strategy
    +The current writable overlay locking strategy is based on the
    +following rules:
    +* Exactly two file systems are unioned
    +* The bottom file system is always read-only
    +* The top file system is always read-write
    + => A file system can never a top and a bottom layer at the same time
    +Additionally, the top layer (the writable overlay) may only be mounted
    +exactly once. Don't think of the writable overlay as a separate
    +independent file system; when it is mounted as a writable overlay, it
    +is only a file system in conjunction with the read-only bottom layer.
    +The read-only bottom layer is an independent file system in and of
    +itself and can be mounted elsewhere, including as the bottom layer for
    +another writable overlay.
    +Thus, we may define a stable locking order in terms of top layer and
    +bottom layer locks, since a top layer is never a bottom layer and a
    +bottom layer is never a top layer. Objects from the bottom layer are
    +never changed (so don't need write locks) and only require atomic
    +operations to manage kernel data structures (ref counts, etc.).
    +Another simplifying assumption is that all directories in a pathname
    +exist on the top layer, as they are created step-by-step during
    +lookup. This prevents us from ever having to walk backwards up the
    +path creating directory entries, which can get complicated especially
    +when you consider the need to prevent topology changes. By
    +implication, parent directories during any operation (rename(),
    +unlink(),etc.) are from the top layer. Dentries for directories from
    +the bottom layer are only ever used by lookup code.
    +The two major problems we avoid with the above rules are:
    +Lock ordering: Imagine two union stacks with the same two file
    +systems: A mounted over B, and B mounted over A. Sometimes locks on
    +objects in both A and B will have to be held simultanously. What
    +order should they be acquired in? Simply acquiring them from top to
    +bottom will create a lock-ordering problem - one thread acquires lock
    +on object from A and then tries for a lock on object from B, while
    +another thread grabs the lock on object from B and then waits for the
    +lock on object from A. Some other lock ordering must be defined.
    +Movement/change/disappearance of objects on multiple layers: A variety
    +of nasty corner cases arise when more than one layer is changing at
    +the same time. Changes in the directory topology and their effect on
    +inheritance are of special concern. Al Viro's canonical email on the
    +We don't try to solve any of these cases, just avoid them in the first
    +Todo: Prevent top layer from being mounted more than once.
    +Cross-layer interactions
    +The VFS code simultaneously holds references to and/or modifies
    +objects from both the top and bottom layers in the following cases:
    +Path lookup:
    +Holds i_mutex on top layer directory inode while doing lookups on
    +bottom layer. Grabs i_mutex on bottom layer off and on.
    + - Is i_mutex on lower directory necessary?
    +File copyup in general:
    +File copyup occurs while holding i_mutex on the parent directory of
    +the top layer. As noted before, an in-kernel file copyup is the
    +logical equivalent of a userspace rename() of an identical file on to
    +this pathname.
    +File copyup of target while holding i_mutex on parent directory on top
    +layer. Followed by a normal link() operation.
    +First, renaming of directories returns EXDEV. It's not at all
    +reasonable to recursively copy directory trees and userspace has to
    +handle this case anyway.
    +Rename involves two operations on a writable overlay: (1) creation of
    +a whiteout covering the source of the rename, (2) a copyup of the file
    +from the bottom layer. The file copyup does not need to happen
    +atomically, only the whiteout and the new link to the file.
    +I propose that we copyup the source file to the "old" name (rather
    +than directly to the "new" name), and then perform the normal file
    +system rename operation. The only addition is creation of whiteout
    +for the old name.
    +The current rename() implementation is just a hack to get things
    +working and doesn't work at all as described above.
    +Lock order: The file copyup happens before the rename() lock. When we
    +create the whiteout, we will already have the directory i_mutex.
    +Otherwise, locking as usual.
    +Directory copyup:
    +Directory entries are copied up on the first readdir(). We hold the
    +top layer directory i_mutex throughout. A fallthru is created for
    +each entry that appears only on the lower layer.
    +Current patch takes the i_mutex on the bottom layer directory, which
    +doesn't seem to be necessary.
    +VFS-fs interface
    +Read-only layer: No support necessary other than enforcement of really
    +really read-only semantics (done by VFS for local file systems).
    +Writable layer: Must implement two new inode operations:
    +int (*whiteout) (struct inode *, struct dentry *, struct dentry *);
    +int (*fallthru) (struct inode *, struct dentry *);
    +And set the MS_WHITEOUT flag.
    +Whiteouts and fallthrus are most similar to symlinks, since they
    +redirect to an object possibly located in another file system without
    +keeping a reference on it.
    +- Return correct inode number in d_ino member of struct dirent by one of:
    + - Save inode number of target in fallthru entry itself
    + - Lookup inode number during readdir()
    +- Try re-implementing ext2 as special symlinks - may be much simpler
    +- Implement ext3 (also as symlinks?)
    +- Implement btrfs
    +Supported file systems
    +Any file system can be a read-only layer. File systems must
    +explicitly support whiteouts and fallthrus in order to be a read-write
    +layer. This patch set implements whiteouts for ext2, tmpfs, and
    +jffs2. We have tested ext2, tmpfs, and iso9660 as the read-only
    + - Test corner cases of case-insensitive/oversensitive file systems
    +NFS interaction
    +NFS is currently not supported as either type of layer. NFS as
    +read-only layer requires support from the server to honor the
    +read-only guarantee needed for the bottom layer. To do this, the
    +server needs to revoke access to clients requesting read-only file
    +systems if the exported file system is remounted read-write or
    +unmounted (during which arbitrary changes can occur). Some recent
    +NFS as the read-write layer would require implementation of the
    +->whiteout() and ->fallthru() methods. DT_WHT directory entries are
    +theoretically already supported.
    +Also, technically the requirement for a readdir() cookie that is
    +stable across reboots comes only from file systems exported via NFSv2:
    +- Implement whiteout()/fallthru() for NFS
    +- Guarantee really really read-only on NFS exports
    +Userland support
    +The mount command must support the "-o union" mount option and pass
    +the corresponding MS_UNION flag to the kerel. A util-linux git
    +tree with writable overlay support is here:
    +File system utilities must support whiteouts and fallthrus. An
    +e2fsprogs git tree with writable overlay support is here:
    +Currently, whiteout directory entries are not returned to userland.
    +While the directory type for whiteouts, DT_WHT, has been defined for
    +many years, very little userland code handles them. Userland will
    +never see fallthru directory entries.
    +Known non-POSIX behaviors
    +- Any writing system call (unlink()/chmod()/etc.) can return ENOSPC or EIO
    +- Link count may be wrong for files on bottom layer with > 1 link count
    +- Link count on directories will be wrong before readdir() (fixable)
    +- File copyup is the logical equivalent of an update via copy +
    + rename(). Any existing open file descriptors will continue to refer
    + to the read-only copy on the bottom layer and will not see any
    + changes that occur after the copy-up.
    +- rename() of directory fails with EXDEV
    +The current writable overlays patch set varies between RFC/prototype
    +and pretty stable, depending on the particular patch. The current
    +patch set boots to multi-user mode with a writable overlay root file
    +system (albeit with some complaints). Some parts of the code were
    +written years ago and have been reviewed, rewritten and tested many
    +times. Other parts were written last month and need review,
    +rewriting, and testing. The commit messages note the state of each
    +The current patch set is against 2.6.31. You can find it here, in the
    +branch "overlay":
    +Features we do not currently plan to support as part of writable
    +Online upgrade: E.g., installing software on a file system NFS
    +exported to clients while the clients are still up and running.
    +Allowing the read-only bottom layer to change while the writable
    +overlay file system is mounted invalidates our locking strategy.
    +Recursive copying of directories: E.g., implementing rename() across
    +layers for directories. Doing an in-kernel copy of a single file is
    +bad enough. Recursively copying a directory is a big no-no.
    +Read-only top layer: The readdir() strategy fundamentally requires the
    +ability to create persistent directory entries on the top layer file
    +system (which may be tmpfs). Numerous alternatives (including
    +in-kernel or in-application caching) exist and are compatible with
    +writable overlays with its writing-readdir() implementation disabled.
    +Creating a readdir() cookie that is stable across multiple readdir()s
    +requires one of:
    +- Write to stable storage (e.g., fallthru dentries)
    +- Non-evictable kernel memory cache (doesn't handle NFS server reboot)
    +- Per-application caching by glibc readdir()
    +Aggregation of multiple read-only file systems: While perfectly
    +reasonable from a user perspective, we just aren't smart enough to
    +figure out the locking problems from a kernel perspective. Sorry!
    +Often these features are supported by other unioning file systems or
    +by other versions of union mounts.
    +Contributing to writable overlays
    +The writable overlays web page is here:
    +It links to:
    + - All git repositories
    + - Documentation
    + - An entire self-contained UML-based dev kit with README, etc.
    +The mailing list for discussing writable overlays is:
    +Thank you for reading!

     \ /
      Last update: 2010-04-16 01:09    [W:0.062 / U:19.116 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site