[lkml]   [2001]   [Dec]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: SMP/cc Cluster description


    I am the maintaner of SCI, the ccNUMA technology standard. I know
    alot about this stuff, and have been involved with SCI since
    1994. I work with it every day and the Dolphin guys on some huge
    supercomputer accounts, like Los Alamos and Sandia Labs in NM.
    I will tell you this from what I know.

    A shared everything approach is a programmers dream come true,
    but you can forget getting reasonable fault tolerance with it. The
    shared memory zealots want everyone to believe ccNUMA is better
    than sex, but it does not scale when compared to Shared-Nothing
    programming models. There's also a lot of tough issues for dealing
    with failed nodes, and how you recover when peoples memory is
    all over the place across a nuch of machines.

    SCI scales better in ccNUMA and all NUMA technoogies scale very
    well when they are used with "Explicit Coherence" instead of
    "Implicit Coherence" which is what you get with SMP systems.
    Years of research by Dr. Justin Rattner at Intel's High
    performance labs demonstrated that shared nothing models scaled
    into the thousands of nodes, while all these shared everything
    "Super SMP" approaches hit the wall at 64 processors generally.

    SCI is the fastest shared nothing interface out there, and it also
    can do ccNUMA. Sequent, Sun, DG and a host of other NUMA providers
    use Dolphin's SCI technology and have for years. ccNUMA is useful
    for applications that still assume a shared nothing approach but that
    use the ccNUMA and NUMA capabilities for better optimization.

    Forget trying to recreate the COMA architecture of Kendall-Square.
    The name was truly descriptive of what happened in this architecture
    when a node fails -- goes into a "COMA". This whole discussion I have
    lived through before and you will find that ccNUMA is virtually
    unimplementable on most general purpose OSs. And yes, there are
    a lot of products and software out there, but when you look under
    the cover (like ServerNet) you discover their coherence models
    for the most part relay on push/pull explicit coherence models.

    My 2 cents.


    On Thu, Dec 06, 2001 at 12:09:32AM -0800, David S. Miller wrote:
    > From: Larry McVoy <>
    > Date: Thu, 6 Dec 2001 00:02:16 -0800
    > Err, Dave, that's *exactly* the point of the ccCluster stuff. You get
    > all that seperation for every data structure for free. Think about
    > it a bit. Aren't you going to feel a little bit stupid if you do all
    > this work, one object at a time, and someone can come along and do the
    > whole OS in one swoop? Yeah, I'm spouting crap, it isn't that easy,
    > but it is much easier than the route you are taking.
    > How does ccClusters avoid the file system namespace locking issues?
    > How do all the OS nodes see a consistent FS tree?
    > All the talk is about the "magic filesystem, thread it as much as you
    > want" and I'm telling you that is the fundamental problem, the
    > filesystem name space locking.
    > -
    > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > the body of a message to
    > More majordomo info at
    > Please read the FAQ at
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to
    More majordomo info at
    Please read the FAQ at

     \ /
      Last update: 2005-03-22 13:18    [W:0.025 / U:34.280 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site