[lkml]   [2012]   [Feb]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Patch in this message
Subject[PATCH 18/73] union-mount: Union mounts documentation [ver #2]
From: Valerie Aurora <>

Document design and implementation of union mounts (a.k.a. writable overlays).

With corrections from Andreas Gruenbacher <>.

Original-author: Valerie Aurora <>
Signed-off-by: David Howells <>

Documentation/filesystems/union-mounts.txt | 712 ++++++++++++++++++++++++++++
1 files changed, 712 insertions(+), 0 deletions(-)
create mode 100644 Documentation/filesystems/union-mounts.txt

diff --git a/Documentation/filesystems/union-mounts.txt b/Documentation/filesystems/union-mounts.txt
new file mode 100644
index 0000000..596bfe6
--- /dev/null
+++ b/Documentation/filesystems/union-mounts.txt
@@ -0,0 +1,712 @@
+Union mounts (a.k.a. writable overlays)
+This document describes the architecture and current status of union mounts,
+also known as writable overlays.
+In this document:
+ - Overview of union mounts
+ - Terminology
+ - VFS implementation
+ - Locking strategy
+ - VFS/file system interface
+ - Userland interface
+ - NFS interaction
+ - Status
+ - Contributing to union mounts
+A union mount layers one read-write file system over one or more read-only file
+systems, with all writes going to the writable file system. The namespace of
+both file systems appears as a combined whole to userland, with files and
+directories on the writable file system covering up any files or directories
+with matching pathnames on the read-only file system. The read-write file
+system is the "topmost" or "upper" file system and the read-only file systems
+are the "lower" file systems. A few use cases:
+- Root file system on CD with writes saved to hard drive (LiveCD)
+- Multiple virtual machines with the same starting root file system
+- Cluster with NFS mounted root on clients
+Most if not all of these problems could be solved with a COW block device or a
+clustered file system (include NFS mounts). However, for some use cases,
+sharing is more efficient and better performing if done at the file system
+namespace level. COW block devices only increase their divergence as time goes
+on, and a fully coherent writable file system is unnecessary synchronization
+overhead if no other client needs to see the writes.
+What union mounts are not
+Union mounts are not a general-purpose unioning file system. They do not
+provide a generic "union of namespaces" operation for an arbitrary number of
+file systems. Many interesting features can be implemented with a generic
+unioning facility: dynamic insertion and removal of branches, write policies
+based on space available, online upgrade, etc. Some unioning file systems that
+do this are UnionFS and AUFS.
+The main physical metaphor for union mounts is that a writable file system is
+mounted "on top" of a read-only file system. Lookups start at the "topmost"
+read-write file system and travel "down" to the "bottom" read-only file system
+only if no blocking entry exists on the top layer.
+Topmost layer: The read-write file system. Lookups begin here.
+Bottom layer: The read-only file system. Lookups end here.
+Path: Combination of the vfsmount and dentry structure.
+Follow down: Given a path from the top layer, find the corresponding path on
+the bottom layer.
+Follow up: Given a path from the bottom layer, find the corresponding path on
+the top layer.
+Whiteout: A directory entry in the top layer that prevents lookups from
+travelling down to the bottom layer. Created on unlink()/rmdir() if a
+corresponding directory entry exists in the bottom layer.
+Opaque flag: A flag on a directory in the top layer that prevents lookups of
+entries in this directory from travelling down to the bottom layer (unless
+there is an explicit fallthru entry allowing that for a particular entry). Set
+on creation of any new directory in in the topmost layer (that is, a directory
+that does not have any matching visible directory below it).
+Fallthru: A directory entry which allows lookups to "fall through" to the
+bottom layer for that exact directory entry. This serves as a placeholder for
+directory entries from the bottom layer during readdir(). Fallthrus override
+opaque flags.
+File copyup: Create a file on the top layer that has the same metadata and
+contents as the file with the same pathname on the bottom layer.
+Directory copyup: Copy up the visible directory entries from the bottom layer
+as fallthrus in the matching top layer directory. Mark the directory opaque to
+avoid unnecessary negative lookups on the bottom layer.
+What happens when I...
+- creat() /newfile -> creates on topmost layer
+- unlink() /oldfile -> creates a whiteout on topmost layer
+- Edit /existingfile -> copies up to top layer at open(O_WR) time
+- truncate /existingfile -> copies up to topmost layer + N bytes if specified
+- touch()/chmod()/chown()/etc. -> copies up to topmost layer
+- mkdir() /newdir -> creates opaque dir on topmost layer
+- rmdir() /olddir -> creates a whiteout on topmost layer
+- mkdir() /olddir after above -> creates opaque dir on topmost layer
+- readdir() /shareddir -> copies up entries from bottom layer as
+ fallthrus, processes duplicates and whiteouts
+- link() /oldfile /newlink -> copies up /oldfile, creates /newlink on
+ topmost layer
+- symlink() /oldfile /symlink -> nothing special
+- rename() /oldfile /newfile -> copies up /oldfile to /newfile on top layer,
+ whiteouts /oldfile
+- rename() /olddir /newdir -> EXDEV
+- rename() /topmost_only_dir /topmost_only_dir2 -> success
+- stat() /oldfile - inode & dev from lower layer
+- stat() /newfile - inode & dev from topmost layer
+- readdir() /shareddir - d_ino & d_type from lower layer on fallthrus
+Getting to a root file system with union mounts:
+- Mount the base read-only file system as the root file system
+- Mount the read-only file system again on /newroot
+- Mount the read-write layer on /newroot:
+ # mount -o union /dev/sda /newroot
+- pivot_root to /newroot
+- Start init
+See scripts/ in the UML devkit linked to from:
+VFS implementation
+Union mounts are implemented as an integral part of the VFS, rather than as a
+VFS client file system (i.e., a stacked file system like unionfs or ecryptfs).
+Implementing unioning inside the VFS eliminates the need for duplicate copies
+of VFS data structures, unnecessary indirection, and code duplication, but
+requires very maintainable, low overhead code. Union mounts require no change
+to file systems serving as the read-only layer, and requires some minor support
+from file systems serving as the read-write layer. File systems that want to
+be the writable layer must implement the new ->whiteout() and ->fallthru()
+inode operations, which create special dummy directory entries.
+The union mounts code must accomplish the following major tasks:
+1) Pass lookups through to the lower level file system.
+2) Copy files and directories up to the topmost layer when written.
+3) Create whiteouts and fallthrus as necessary.
+VFS objects and union mounts
+First, some VFS basics:
+The VFS allows multiple mounts of the same file system. For example, /dev/sda
+can be mounted at /usr and also at /mnt. The same file system can be mounted
+read-only at one point and read-write at another. Each of these mounts has its
+own vfsmount data structure in the kernel. However, each underlying file
+system has exactly one in-kernel superblock structure no matter how many times
+it is mounted. All the separate vfsmounts for the same file system reference
+the same superblock data structure.
+Directory entries are cached by the VFS in dentry structures. The VFS keeps
+one dentry structure for each file or directory in a file system, no matter how
+many times it is mounted. Each dentry represents only one element of a path
+name. When the VFS looks up a pathname (e.g., "/sbin/init"), the result is a
+combination of vfsmount and dentry. This <mnt,dentry> pair is usually stored
+in a kernel structure named "path", which is simply two pointers, one to the
+vfsmount and one to the dentry. A "struct path" is this structure; a pathname
+is a string like "/etc/fstab".
+In union mounts, a file system can only be the topmost layer for one union
+mount. A file system can be part of multiple union mounts if it is a read-only
+layer. So dentries in the read-only layers can be part of multiple unions,
+while a dentry in the read-write layer can only be part of one unin.
+union_dir structure
+The first job of union mounts is to map directories from the topmost layer to
+directories with the same pathname in the lower layer. That is, given the
+<mnt,dentry> pair for a directory pathname in the topmost layer, we need to
+find all the <mnt,dentry> pairs for the directory with the same pathname in the
+lower layer. We do this with the union_dir structure, which is an array
+containing struct paths (mnt, dentry pointer pairs) for each directory unioned
+with the topmost union. The array is pointed to from the new d_union_stack
+member of struct dentry.
+ * The union_stack structure. It is an array of struct paths of
+ * directories below the topmost directory in a unioned directory, The
+ * topmost dentry has a pointer to this structure. The topmost dentry
+ * can only be part of one union, so we can reference it from the
+ * dentry, but lower dentries can be part of multiple union stacks.
+ *
+ * The number of dirs actually allocated is kept in the superblock,
+ * s_union_count.
+ */
+struct union_stack {
+ struct path u_dirs[0];
+This structure is flexible enough to support an arbitrary number of layers of
+unioned file systems. Since there can be more than two layers, this section
+will talk about mapping "upper" directories to "lower" directories, instead of
+"topmost" directories to "bottom" directories.
+Traversing the union stack
+The set of union_dir structures referring to a particular pathname are called
+collectively the union stack for that directory. To traverse the union stack,
+iterate through the number of layers in the union (stored in sb->s_union_count)
+with union_find_dir(). Example: freeing the union stack:
+void d_free_unions(struct dentry *topmost)
+ struct path *path;
+ unsigned int i, layers = topmost->d_sb->s_union_count;
+ if (!IS_DIR_UNIONED(topmost))
+ return;
+ for (i = 0; i < layers; i++) {
+ path = union_find_dir(topmost, i);
+ if (path->mnt)
+ path_put(path);
+ }
+ kfree(topmost->d_union_stack);
+ topmost->d_union_stack = NULL;
+Code paths
+Union mounts modify the following key code paths in the VFS:
+- mount()/umount()
+- Pathname lookup
+- Any path that modifies an existing file
+Union mounts are created in two steps:
+1. Mount the read-only layer file systems read-only in the usual manner, all on
+the same mountpoint. Submounts are permitted as long as they are also
+read-only and not shared (part of a mount propagation group).
+2. Mount the top layer with the "-o union" option at the same mountpoint. All
+read-only file systems mounted at this mountpoint will be included in the union
+The bottom layers must be read-only and the top layer must be read-write and
+support whiteouts and fallthrus. A file system that supports whiteouts and
+fallthrus indicates this by setting the MS_WHITEOUT and MS_FALLTHRU flags in
+the superblock. Currently, the top layer is forced to "noatime" to avoid a
+copyup on every access of a file. Supporting atime with the current
+infrastructure would require a copyup on every open(). The "relatime" option
+would be equally efficient if the atime is the same or more recent than the
+mtime/ctime for every object on the read-only file system, and if the 24-hour
+timeout on relatime was disabled. However, this is probably not worthwhile for
+the majority of union mount use cases.
+File systems can only be union mounted at their root directories, for
+simplicity and performance.
+pivot_root() to a union mounted file system is supported. The recommended way
+to get to a union mounted root file system is to boot with the read-only mount
+as the root file system, construct the union mount on an entirely new mount,
+and pivot_root() to the new union mount root. Attempting to union mount the
+root file system later in boot will result in covering other file systems,
+e.g., /proc, which isn't permitted in the current code and is a bad idea
+Hard read-only file systems
+Union mounts require the lower layer of the file system to be read-only.
+However, in Linux, any individual file system may be mounted at multiple places
+in the namespace, and a file system can be changed from read-only to read-write
+while still mounted. Thus, simply checking that the bottom layer is read-only
+at the time the writable overlay is mounted over it is pointless, since at any
+time the bottom layer may become read-write.
+We have to guarantee that a file system will be read-only for as long as it is
+the bottom layer of a union mount. To do this, we track the number of hard
+read-only users of a file system in its VFS superblock structure. When we
+union mount a writable overlay over a file system, we increment its read-only
+user count. The file system can only be mounted read-write if its read-only
+users count is zero.
+- Support hard read-only NFS mounts. See discussion here:
+Pathname lookup
+Pathname lookup in a unioned directory traverses down the union stack for the
+parent directory, looking up each pathname element in each layer of the file
+system (according to the rules of whiteouts, fallthrus, and opaque flags). At
+mount time, the union stack for the root directory of the file system is
+created, and the union stack creation for every other unioned directory in the
+file system is boot-strapped using the already-existing union stack of the
+directory's parent. In order to simplify the code greatly, every visible
+directory on the lower file system is required to have a matching directory on
+the upper file system. If this matching directory does not already exist, it
+is created during pathname lookup. Therefore, each unioned directory is the
+child of another unioned directory (or is the root directory of the file
+The actual union lookup function is called in the following code paths:
+__lookup_union() is where the rules of whiteouts, fallthrus, and opaque flags
+are actually implemented. __lookup_union() returns either the first visible
+dentry, or a negative dentry from the topmost file system if no matching dentry
+exists. If it finds a directory, it looks up any potential matching lower
+layer directories. If it finds a lower layer directory, it first creates the
+topmost dir if necessary via union_create_topmost_dir(), and then calls
+union_add_dir() to append the lower directory to the end of the union stack.
+Note that not all directories in a union mount are unioned, only those with
+matching directories on the lower layer. The macro IS_DIR_UNIONED() is a
+cheap, constant time way to check if a directory is unioned, while
+IS_MNT_UNION() checks if the entire mount is unioned (and therefore whether the
+directory in question is potentially unioned).
+Currently, lookup of a negative dentry or a directory with no matching
+directories below it requires a lookup in every directory in the union stack
+every time it is looked up. We could avoid subsequent lookups by adding the
+equivalent of a negative dcache entry.
+File copyup
+Any system call that alters the data or metadata of a file on the bottom layer,
+or creates or changes a hard link to it will trigger a copyup of the target
+file from the lower layer to the topmost layer
+ - open(O_WRITE | O_RDWR | O_APPEND)
+ - truncate()/open(O_TRUNC)
+ - link()
+ - rename()
+ - chmod()
+ - chown()/lchown()
+ - utimes()
+ - setxattr()/lsetxattr()
+Copyup of a file due to open(O_WRITE) has already occurred when:
+ - write()
+ - ftruncate()
+ - writable mmap()
+The following system calls will fail on an fd opened O_RDONLY:
+ - fchmod()
+ - fchown()
+ - fsetxattr()
+ - futimensat()
+Contrary to common sense, the above system calls are defined to succeed on
+O_RDONLY fds. The idea seems to be that the O_RDONLY/O_RDWR/O_WRITE flags only
+apply to the actual file data, not to any form of metadata (times, owner, mode,
+or even extended attributes). Applications making these system calls on
+O_RDONLY fds are correct according to the standard and work on non-union
+mounts. They will need to be rewritten (O_RDONLY -> O_RDWR) to work on union
+mounts. We suspect this usage is uncommon.
+This deviation from standard is due to technical limitations of the union mount
+implementation. Specifically, we would need to replace an open file descriptor
+from the lower layer with an open file descriptor for a file with matching
+pathname and contents on the upper layer, which is difficult to do. We avoid
+this in other system calls by doing the copyup before the file is opened.
+Unionfs doesn't encounter this problem because it creates a dummy file struct
+which redirects or fans out operations to the struct files for the underlying
+file systems.
+From an application's point of view, the result of an in-kernel file copyup is
+the logical equivalent of another application updating the file via the
+rename() pattern: creat() a new file, copy the data over, make changes the
+copy, and rename() over the old version. Any existing open file descriptors
+for that file (including those in the same application) refer to a now
+invisible object that used to have the same pathname. Only opens that occur
+after the copyup will see updates to the file.
+Permission checks
+We want to be sure we have the correct permissions to actually succeed in a
+system call before copying a file up to avoid unnecessary IO. At present, the
+permission check for a single system call may be spread out over many hundreds
+of lines of code (e.g., open()). In order to check permissions, we
+occasionally need to determine if there is a writable overlay on top of this
+inode. This requires a full path, but often we only have the inode at this
+point. In particular, inode_permission() returns EROFS if the inode is on a
+read-only file system, which is the wrong answer if there is a writable overlay
+mounted on top of it.
+The current solution is to split out the file-system-wide permission checks
+from the per-inode permission checks. inode_permission() becomes:
+inode_permission() calls sb_permission() and __inode_permission() on the same
+path. We create path_permission() which calls sb_permission() on the parent
+directory from the top layer, and __inode_permission() on the target on the
+lower layer. This gets us the correct write permissions consdering that the
+file will be copied up.
+ - Currently, we don't deal with differing directory permissions at
+ different levels of the stack. This is a bug.
+Impact on non-union kernels and mounts
+Union-related data structures, extra fields, and function calls are #ifdef'd
+out at the function/macro level with CONFIG_UNION_MOUNT in nearly all cases
+(see fs/union.h). When CONFIG_UNION_MOUNT is enabled, struct dentry has one
+more pointer, reducing the size of dentry names stored in the dentry itself by
+4 to 8 bytes.
+ - Do performance tests
+Locking strategy
+The current union mount locking strategy is based on the following
+* The lower layer file system is always read-only
+* The topmost file system is always read-write
+ => A file system can never a topmost and lower layer at the same time
+Additionally, the topmost layer may only be mounted exactly once. Don't think
+of the topmost layer as a separate independent file system; when it is part of
+a union mount, it is only a file system in conjunction with the read-only
+bottom layer. The read-only bottom layer is an independent file system in and
+of itself and can be mounted elsewhere, including as the bottom layer for
+another union mount.
+Thus, we may define a stable locking order in terms of top layer and bottom
+layer locks, since a top layer is never a bottom layer and a bottom layer is
+never a top layer. Another simplifying assumption is that all directories in a
+pathname exist on the top layer, as they are created step-by-step during
+lookup. This prevents us from ever having to walk backwards up the path
+creating directory entries, which can get complicated. By implication, parent
+directories paths during any operation (rename(), unlink(),etc.) are from the
+top layer. Dentries for directories from the bottom layer are only ever seen
+or used by the lookup code.
+The two major problems we avoid with the above rules are:
+Lock ordering: Imagine two union stacks with the same two file systems: A
+mounted over B, and B mounted over A. Sometimes locks on objects in both A and
+B will have to be held simultanously. What order should they be acquired in?
+Simply acquiring them from top to bottom will create a lock-ordering problem -
+one thread acquires lock on object from A and then tries for a lock on object
+from B, while another thread grabs the lock on object from B and then waits for
+the lock on object from A. Some other lock ordering must be defined.
+Movement/change/disappearance of objects on multiple layers: A variety of nasty
+corner cases arise when more than one layer is changing at the same time.
+Changes in the directory topology and their effect on inheritance are of
+special concern. Al Viro's canonical email on the subject:
+We don't try to solve any of these cases, just avoid them in the first place.
+Todo: Prevent top layer from being mounted more than once.
+Cross-layer interactions
+The VFS code simultaneously holds references to and/or modifies objects from
+both the top and bottom layers in the following cases:
+Path lookup:
+Grabs i_mutex on bottom layer while holding i_mutex on top layer directory
+File copyup:
+Holds i_mutex on the parent directory from the top layer while copying up file
+from lower layer.
+File copyup of target while holding i_mutex on parent directory on top layer.
+Followed by a normal link() operation.
+Holds s_vfs_rename_mutex on the top layer, i_mutex of the source's parent dir
+(top layer), and i_mutex of the target's parent dir (also top layer) while
+looking up and copying the bottom layer target and also creating the whiteout.
+Notes on rename():
+First, renaming of directories returns EXDEV. It's not at all reasonable to
+recursively copy directory trees and userspace has to handle this case anyway.
+An exception is rename() of directories that exist only on the topmost layer;
+this succeeds.
+Rename involves three steps on a union mount: (1) copyup of the file from the
+bottom layer, (2) rename of the new top-layer copy to the target in the usual
+manner, (3) creation of a whiteout covering the source of the rename.
+Directory copyup:
+Directory entries are copied up on the first readdir(). We hold the top layer
+directory i_mutex throughout and sequentially acquire and drop the i_mutex for
+each lower layer directory.
+VFS-fs interface
+Read-only layer: No support necessary other than enforcement of really really
+read-only semantics (done by VFS for local file systems).
+Writable layer: Must implement two new inode operations:
+int (*whiteout) (struct inode *, struct dentry *, struct dentry *);
+int (*fallthru) (struct inode *, struct dentry *);
+And set the MS_WHITEOUT and MS_FALLTHRU flags to indicate support of
+these operations.
+- Implement whiteouts and fallthrus in ext3
+- Implement whiteouts and fallthrus in btrfs
+Supported file systems
+Any file system can be a read-only layer. File systems must explicitly support
+whiteouts and fallthrus in order to be a read-write layer. This patch set
+implements whiteouts for ext2, tmpfs, and jffs2. We have tested ext2, tmpfs,
+and iso9660 as the read-only layer.
+ - Test corner cases of case-insensitive/oversensitive file systems
+NFS interaction
+NFS is currently not supported as either type of layer. NFS as read-only layer
+requires support from the server to honor the read-only guarantee needed for
+the bottom layer. To do this, the server needs to revoke access to clients
+requesting read-only file systems if the exported file system is remounted
+read-write or unmounted (during which arbitrary changes can occur). Some
+recent discussion:
+NFS as the read-write layer would require implementation of the ->whiteout()
+and ->fallthru() methods. DT_WHT directory entries are theoretically already
+Also, technically the requirement for a readdir() cookie that is stable across
+reboots comes only from file systems exported via NFSv2:
+- Guarantee really really read-only on NFS exports
+- Implement whiteout()/fallthru() for NFS
+Userland support
+The mount command must support the "-o union" mount option and pass the
+corresponding MS_UNION flag to the kerel. A util-linux git tree with union
+mount support is here:
+File system utilities must support whiteouts and fallthrus. An e2fsprogs git
+tree with union mount support is here:
+Currently, whiteout directory entries are not returned to userland. While the
+directory type for whiteouts, DT_WHT, has been defined for many years, very
+little userland code handles them. Userland will never see fallthru directory
+Known non-POSIX behaviors
+- Any writing system call (unlink()/chmod()/etc.) can return ENOSPC or EIO
+ Most programs are not tested and don't work well under conditions of ENOSPC.
+ The solution is to add more disk space.
+- Link count may be wrong for files on bottom layer with > 1 link count
+ A file may have more than one hard link to it. When a file with multiple
+ hard links is copied up, any other hard links pointing to the same inode will
+ remain unchanged. If the file is looked up via one of the hard links on the
+ read-only layer, it will have the original link count (which is off by one at
+ this point). An example:
+ /bin/link1 -> inode 100
+ /etc/link2 -> inode 100
+ inode 100 will have link count 2.
+ # echo "blah" > /bin/link1
+ Now /bin/link1 will be copied up to the topmost layer. But /etc/link2 will
+ still point to the original inode 100, and its link count will still be 2.
+- Link count on directories will be wrong before readdir() (fixable)
+- File copyup is the logical equivalent of an update via copy +
+ rename(). Any existing open file descriptors will continue to refer
+ to the read-only copy on the bottom layer and will not see any
+ changes that occur after the copy-up.
+- rename() of directory may fail with EXDEV
+- fchmod()/fchown()/futimensat()/fsetattr() fail on O_RDONLY fds
+The current union mounts implementation is feature-complete on local file
+systems and passes an extensive union mounts test suite, available in the union
+mounts Usermode Linux-based development kit:
+The whiteout code has had some non-trivial level of review and testing, but
+much of the code has had no external review or testing outside the authors'
+The latest version is available at:
+Check the union mounts web page for the name of the latest branch:
+- Run more tests (e.g., XFS test suite)
+- Get review from VFS maintainers
+Features we do not currently plan to support in union mounts:
+Online upgrade: E.g., installing software on a file system NFS exported to
+clients while the clients are still up and running. Allowing the read-only
+bottom layer of a union mount to change invalidates our locking strategy.
+Recursive copying of directories: E.g., implementing rename() across layers for
+directories. Doing an in-kernel copy of a single file is bad enough.
+Recursively copying a directory is a big no-no.
+Read-only top layer: The readdir() strategy fundamentally requires the ability
+to create persistent directory entries on the top layer file system (which may
+be tmpfs). However, you can union two read-only file systems by union mounting
+a third file system (such as tmpfs) over the two read-onlly file systems.
+Numerous alternatives to this readdir() strategy (including in-kernel or
+in-application caching) exist and are compatible with union mounts with its
+writing-readdir() implementation disabled. Creating a readdir() cookie that is
+stable across multiple readdir()s requires one of:
+- Write to stable storage (e.g., fallthru dentries)
+- Non-evictable kernel memory cache (doesn't handle NFS server reboot)
+- Per-application caching by glibc readdir()
+Often these features are supported by other unioning file systems or by other
+versions of union mounts.
+Contributing to union mounts
+The union mounts web page is here:
+It links to:
+ - All git repositories
+ - Documentation
+ - An entire self-contained UML-based dev kit with README, etc.
+The best mailing list for discussing union mounts is:
+Thank you for reading!

 \ /
  Last update: 2012-02-21 20:55    [W:0.260 / U:2.196 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site