Messages in this thread | | | Date | Sat, 13 Jun 1998 00:39:29 -0700 | From | Larry McVoy <> | Subject | cluster thoughts |
| |
mshar@vax.ipm.ac.ir said: : Unfortunately I am currently not in a position to actively work on these : problems. But I hope there are other people who have either started working, : or are considering to do so.
I don't know about the rest of the readers, but I would be much more interested in hearing from someone who
. has experience with clusters, and/or . has interest in learning from past experience of others, and/or . has experience as an architect, and/or . is willing to do some of the suggested work.
You seem to have none of the interesting qualifications, yet you find yourself qualified to tell other people how the job ought to be done. Don't you find that the slightest bit presumptuous?
Moving on to a more positive note, I personally hate it when grey haired architects tell me my ideas are stupid but don't tell me the right answer. So here's a little sketch of what I think is the right direction for clustering.
Things which are bad:
. Distributed shared memory. It has no failure model - what do you do when you page fault on page that is no longer there because that node crashed? There is no soution, you simply can't allow that to happen, which means no distributed shared memory.
. Remote procedure calls. RPCs block. It's a nice textbook way to tell people to think, but it is horribly slow in practice. Use messages, trasaction IDs and queues instead. A single process can handle 1000's or 10's of thousands of messages / second.
. Process migration. Sounds good, but is way too costly to be of any use. Good explanations of why may be found in any paper on scheduling algorthms and their effect on caches. Most of these papers end up talking about "cache affinity" which is a fancy way of saying "run once on CPU 3, run always on CPU 3". In order to see the relationship to process migration, view the scheduler as the same as the migrator, and the processor cache the same as the page cache, and everything else is the same.
. Dynamic load balancing across machines. This is only an option if you have process migration, which we've ruled out. We accomplish load balancing statically, see below.
. Remote fork(). It's a nice idea, but it doesn't work well in practice.
. Security. People try and make clusters secure by layering secure protocols on the interconnect. And throw away all the performance. See system area networks for how to do this right.
Things which are good:
. Local and global names. - processes Full PID is [node#][pid #] with node 0 meaning this node; nodes/pids are 16 bits each. With this you get: "1" is "init" on this node. 0x00040001 is "init" on node 4. - /dev vs /gdev /dev/whatever means a device on this node. /gdev/whatever may or may not be on this node. Note that /gdev has entries /only/ for real devices, unlike /dev. - /tmp vs /gtmp /tmp is for local files, it's not visible to the rest of cluster. /gtmp is a globally shared tmp space. - /proc vs /gproc /proc is just like it is now, just local processes. /gproc is a union of all of the /procs.
. Messages and message queues. Messages are nice because they don't block. A great programming model is to have a queue of outstanding messages waiting for replies; each of these messages has a unique transaction id (XID). When replies come back in, the XID is extracted from the reply, the pending queue is searched for the XID, the associated message is removed, and put on a replied queue, and [optionally] the process that wants to hear about this reply is signalled. No threads, no waiting, no overhead.
Messages also work well in a cluster because they give you positive notification that the work is done. If you try and use distributed shared memory, you end up using messages anyway. Consider a producer consumer problem in DSM. The producer puts the data in memory. How does the consumer know that the data is ready? Well, it has to get notified. What is that? That's a message. How do you do a message in DSM?
. Remote exec. You have to have some way of fanning out load across the system. You want to have a REMOTE_ON_EXEC flag which tells the OS that it is OK to consider this process for remote exec when doing a regular exec(). The flag gets cleared once the process execs; it has to be manually reset. This lets you do stuff like have login & make and other simiarl process spawners fan out the load without worrying about the subprocesses being fanned out.
. Load balancing. All the nodes need to know about each other so they can fan out load at exec() time. The node<->node load balancing is done only at exec() time. If you make a bad choice, you live with it.
. Cache coherent filesystems. NFS is a fine solution for a campus but it sucks for a cluster. The lack of coherency makes it almost useless. The reason that NFS works like it does is because it is stateless - the server does not know how many clients have the file system mounted or how many files are being cached on remote clients. Because of the stateless design, NFS works really well in the face of failure.
In a cluster, the number and names of all the possible clients of the cluster file system are well known. So it is reasonable to have the server to either (a) keep track of all clients who have data, or (b) send broadcast invalidates and count the acknowledgements. This gives you enough basis for a cache coherent file system.
. System area networks (security). Rather than layer secure protocols on top of the network, you just use what the industry calls a system area network (SAN). A SAN is a private network with no gateways. Traffic on the SAN is trusted, just like traffic on a SMP backplane is trusted.
Well, there's a start, --- Larry McVoy lm@bitmover.com http://www.bitmover.com/lm
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu
| |