Messages in this thread | | | Subject | RE: [Linux-cluster] Re: [cgl_discussion] Re: [dcl_discussion] Clustersummit materials | From | Steven Dake <> | Date | Wed, 11 Aug 2004 11:46:03 -0700 |
| |
On Wed, 2004-08-11 at 11:19, Walker, Bruce J wrote: > > * John Cherry (cherry@osdl.org) wrote: > > At the summit I attended, we also talked about using GFS as the > initial > > "consumer" of the cluster infrastructure. The cluster infrastructure > > doesn't stand a chance of mainline acceptance without a consumer that > > both validates the interfaces and hardens the services. > > Given cman etc. was written for GFS, it doesn't prove much that it works > with GFS. Having an independent cluster effort (like OpenSSI) use the > underlying infrastructure presents a much more compelling case. The > OpenSSI project has started to look into this but help from OSDL, Intel > and/or RedHat wouldn't be discouraged. Also, having SAF layered and/or > ha-linux layered would also bolster the case as a general > infrastructure. > > Bruce walker > OpenSSI project lead > >
Bruce,
I have looked over the cman protocol and find it is suboptimal. Here is how it works: it sends a message using multicast, adds a timeout for the message, and waits for an acknowledgement from every node in the system. Once the timer expires, it resends the message. If every node responds, it deletes the timer.
This sort of protocol wont scale (imagine an ack from 32 nodes for every message sent, and you can understand why) and wont work within partitionable networks (some messages may be delivered to some in the partition and not others). It doesn't provide any sort of strong membership guarantees (the same message may be delivered under various membership views) or delivery guarantees (messages only have FIFO guarantees, when distributed systems require agreed or safe ordering).
I am not sure if you have seen the openais project at http://developer.osdl.org/dev/openais. It is a userland implementation of the SA Forum's AIS interfaces. The openais project uses a technology called virtual synchrony to provide cluster messaging and solves the above problems. The current protocol can obtain upwards of 7MB/sec multicast transfer from 1 node to 8 nodes with encryption and authentication. We use encryption and MAC to ensure security. If you want to see the group messaging api, look at exec/gmi.h from the source on the above website.
If we can't live with the cluster services in userland (although I'm still not convinced), then atleast the group messaging protocol in the kernel could be based upon 20 years of research in group messaging and work properly under _all_ fault scenarios.
I'd invite you or other interested parties, to port the openais code for virtual synchrony to the kernel. I'd do it myself, but I'm focused on implementing openais. Then, if the protocols ended up living in kernel space, atleast the protocols would be secure and be able to meet the needs of all users (including cluster userland services). The group messaging protocol code is compact (about 3500 lines). I'd expect with the address family kernel stuff, it would increase to 4500 or so.
Regards -steve
> > > > ______________________________________________________________________ > _______________________________________________ > cgl_discussion mailing list > cgl_discussion@lists.osdl.org > http://lists.osdl.org/mailman/listinfo/cgl_discussion
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |