lkml.org 
[lkml]   [1998]   [Jun]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
Subjectcluster thoughts
mshar@vax.ipm.ac.ir said:
: Unfortunately I am currently not in a position to actively work on these
: problems. But I hope there are other people who have either started working,
: or are considering to do so.

I don't know about the rest of the readers, but I would be much more
interested in hearing from someone who

. has experience with clusters, and/or
. has interest in learning from past experience of others, and/or
. has experience as an architect, and/or
. is willing to do some of the suggested work.

You seem to have none of the interesting qualifications, yet you find
yourself qualified to tell other people how the job ought to be done.
Don't you find that the slightest bit presumptuous?


Moving on to a more positive note, I personally hate it when grey
haired architects tell me my ideas are stupid but don't tell me the
right answer. So here's a little sketch of what I think is the right
direction for clustering.

Things which are bad:

. Distributed shared memory. It has no failure model - what do you do
when you page fault on page that is no longer there because that node
crashed? There is no soution, you simply can't allow that to happen,
which means no distributed shared memory.

. Remote procedure calls. RPCs block. It's a nice textbook way to tell
people to think, but it is horribly slow in practice. Use messages,
trasaction IDs and queues instead. A single process can handle 1000's
or 10's of thousands of messages / second.

. Process migration. Sounds good, but is way too costly to be of any
use. Good explanations of why may be found in any paper on scheduling
algorthms and their effect on caches. Most of these papers end up
talking about "cache affinity" which is a fancy way of saying "run
once on CPU 3, run always on CPU 3". In order to see the relationship
to process migration, view the scheduler as the same as the migrator,
and the processor cache the same as the page cache, and everything
else is the same.

. Dynamic load balancing across machines. This is only an option if
you have process migration, which we've ruled out. We accomplish
load balancing statically, see below.

. Remote fork(). It's a nice idea, but it doesn't work well in practice.

. Security. People try and make clusters secure by layering secure
protocols on the interconnect. And throw away all the performance.
See system area networks for how to do this right.

Things which are good:

. Local and global names.
- processes
Full PID is [node#][pid #] with node 0 meaning this node;
nodes/pids are 16 bits each. With this you get:
"1" is "init" on this node.
0x00040001 is "init" on node 4.
- /dev vs /gdev
/dev/whatever means a device on this node.
/gdev/whatever may or may not be on this node. Note that
/gdev has entries /only/ for real devices, unlike /dev.
- /tmp vs /gtmp
/tmp is for local files, it's not visible to the rest of cluster.
/gtmp is a globally shared tmp space.
- /proc vs /gproc
/proc is just like it is now, just local processes.
/gproc is a union of all of the /procs.

. Messages and message queues. Messages are nice because they don't
block. A great programming model is to have a queue of outstanding
messages waiting for replies; each of these messages has a unique
transaction id (XID). When replies come back in, the XID is
extracted from the reply, the pending queue is searched for the
XID, the associated message is removed, and put on a replied queue,
and [optionally] the process that wants to hear about this reply
is signalled. No threads, no waiting, no overhead.

Messages also work well in a cluster because they give you positive
notification that the work is done. If you try and use distributed
shared memory, you end up using messages anyway. Consider a producer
consumer problem in DSM. The producer puts the data in memory.
How does the consumer know that the data is ready? Well, it has to
get notified. What is that? That's a message. How do you do a
message in DSM?

. Remote exec. You have to have some way of fanning out load
across the system. You want to have a REMOTE_ON_EXEC flag which
tells the OS that it is OK to consider this process for remote
exec when doing a regular exec(). The flag gets cleared once the
process execs; it has to be manually reset. This lets you do stuff
like have login & make and other simiarl process spawners fan out
the load without worrying about the subprocesses being fanned out.

. Load balancing. All the nodes need to know about each other so
they can fan out load at exec() time. The node<->node load
balancing is done only at exec() time. If you make a bad choice,
you live with it.

. Cache coherent filesystems. NFS is a fine solution for a campus
but it sucks for a cluster. The lack of coherency makes it almost
useless. The reason that NFS works like it does is because it is
stateless - the server does not know how many clients have the file
system mounted or how many files are being cached on remote clients.
Because of the stateless design, NFS works really well in the face
of failure.

In a cluster, the number and names of all the possible clients of
the cluster file system are well known. So it is reasonable to have
the server to either (a) keep track of all clients who have data,
or (b) send broadcast invalidates and count the acknowledgements.
This gives you enough basis for a cache coherent file system.

. System area networks (security). Rather than layer secure
protocols on top of the network, you just use what the industry
calls a system area network (SAN). A SAN is a private network with
no gateways. Traffic on the SAN is trusted, just like traffic on
a SMP backplane is trusted.

Well, there's a start,
---
Larry McVoy lm@bitmover.com http://www.bitmover.com/lm

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu

\
 
 \ /
  Last update: 2005-03-22 13:43    [W:0.182 / U:0.180 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site