[lkml]   [2007]   [Nov]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subject[ANNOUNCE] Ceph distributed file system
Hi everyone,

Ceph is a distributed network file system designed to provide excellent
performance, reliability, and scalability with POSIX semantics. I
periodically see frustration on this list with the lack of a scalable GPL
distributed file system with sufficiently robust replication and failure
recovery to run on commodity hardware, and would like to think that--with
a little love--Ceph could fill that gap. Basic features include:

* POSIX semantics.
* Seamless scaling from a few nodes to many thousands.
* Gigabytes to petabytes.
* High availability and reliability. No single points of failure.
* N-way replication of all data across multiple nodes.
* Automatic rebalancing of data on node addition/removal to efficiently
utilize device resources.
* Easy deployment; most FS components are userspace daemons.
* Fuse-based client.
- Lightweight kernel client (in progress).

- Flexible snapshots on arbitrary subdirectories (soon)
- Quotas (soon)
- Strong security (planned)

(* = current features, - = coming soon)

In contrast to cluster filesystems like GFS, OCFS2, and GPFS that rely on
symmetric access by all clients to shared block devices, Ceph separates
data and metadata management into independent server clusters, similar to
Lustre. Unlike Lustre, however, metadata and storage nodes run entirely
in userspace and require no special kernel support. Storage nodes utilize
either a raw block device or large image file to store data objects, or
can utilize an existing file system (XFS, etc.) for local object storage
(currently with weakened safety semantics). File data is striped across
storage nodes in large chunks to distribute workload and facilitate high
throughputs. When storage nodes fail, data is re-replicated in a
distributed fashion by the storage nodes themselves (with some
coordination from a cluster monitor), making the system extremely
efficient and scalable. Currently only n-way replication is supported,
although initial groundwork has been laid for parity-based data

Metadata servers effectively form a large, consistent, distributed
in-memory cache above the storage cluster that is extremely scalable,
dynamically redistributes metadata in response to workload changes, and
can tolerate arbitrary (well, non-Byzantine) node failures. The metadata
server takes a somewhat unconventional approach to metadata storage to
significantly improve performance for common workloads. In particular,
inodes with only a single link are embedded in directories, allowing
entire directories of dentries and inodes to be loaded into its cache with
a single I/O operation. The contents of extremely large directories can
be fragmented and managed by independent metadata servers, allowing
scalable concurrent access.

Most importantly (in my mind, at least), the system offers automatic data
rebalancing/migration when scaling from a small cluster of just a few
nodes to many hundreds, without requiring an administrator carve the data
set into static volumes or go through the tedious process of migrating
data between servers. When the file system approaches full, new nodes can
be easily added and things will "just work." Moreover, there is an
unfortunate scarcity of 'enterprise-grade' features among open source
Linux file systems when it comes to scalability, failure recovery, load
balancing, and snapshots. Snapshot support is a particularly sore point
for me as block device-based approaches tend to be slow, inefficient,
tedious to use, and unusable by the cluster file systems.

The following paper provides a good overview of the architecture:

Although I have been working on this project for some time (it was the
subject of my PhD thesis), I am announcing it to linux-fsdevel and lkml
now because we are just (finally) getting started with the implementation
of an in-kernel client. There is a prototype client based on FUSE, but it
is limited both in terms of correctness (due to limited control over cache
consistency) and performance. I am hoping to generate interest among file
system developers with more kernel experience than I to help guide
development as we move forward with porting the client to the kernel.

The kernel-side client code is in its infancy, but the rest of the system
is relatively mature. We have already been able to demonstrate excellent
performance and scalability (see the above referenced paper), and the
storage and metadata clusters can both tolerate and recover from
arbitrary/multiple node failures. I would describe the code base
(weighing in at around 40,000 semicolon-lines) as early alpha quality:
there is a healthy amount of debugging work to be done, but the basic
features of the system system are complete and can be tested and

If you have any questions, I would be happy to discuss the architecture
in more detail. The paper mentioned above is probably the best overview.
The source and more are available at the project web site,

Please let me know what you think.

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
More majordomo info at
Please read the FAQ at

 \ /
  Last update: 2007-11-13 03:17    [W:0.035 / U:0.120 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site