lkml.org 
[lkml]   [1998]   [Nov]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: SMP scalability: 8 -> 32 CPUs
Date
"Eloy A. Paris" <eparis@ven.ra.rockwell.com> asks:

: Guys, excuse my ignorance, but does anyone know why Linus said in his
: last Linux Journal interview that right now it is fairly unrealistic
: to expect scaling from 8 to 32 CPUs? What's stopping Linux to scale in
: terms of CPUs?

There are some slides on this topic at http://www.bitmover.com/talks/pp/
that cover the points; I'm not sure how meaningful they are without me
handwaving and filling in the gaps.

Here's what I believe to be th thinking on SMP and Linux. These thoughts
are my opinions but are based on quite a bit of experience with SMP at
Sun and SGI and are also based on several conversations with Linus on
the topic. So unless things have changed, this should still be valid.

The basic point is that if you do both SMP and UP have two choices:

- #ifdef SMP the kernel and essentially run one kernel on UP
and a different kernel on SMP

- No ifdefs, run the SMP kernel on the UP's

The first choice is the route taken by SGI, the second is the route taken by
Sun. I prefer the Sun route. The SGI route results in you having to fix
different bugs on different platforms and it becomes a support nightmare.

The Sun route has the problem that you are taking/releasing locks that
you didn't need to take/release because there wasn't another processor to
worry about (this statement ignores bottom/top driver races, you still
need some sort of locking there).

It's a basic truth that the more scaling you want, the more locks you'll
need. Very, very quickly this will become an unmanagable problem - the
kernel becomes a forest of locks - for example, there were thousands
of locks in the Solaris kernel when I left Sun almost 5 years ago.
When it gets that complex, the obvious route taken by a programmer is to
just add one more to protect their new data structure. That's a lose -
what the programme should do is to examine the call paths that can lead
to accessing his data structure and determine if there is already a lock
at a higher level which will cover this data structure. But that's too
hard and nobody does it once the number of locks gets beyond a handful.

What does this all mean? It means that if you try and scale beyond a
handful of processors, you complicate your kernel substantially and make
it almost impossible to keep it a lightweight thing on a UP (look at
Sun vs Linux lmbench numbers on the same hardware, that's all the proof
you need).

Furthermore, most people have UP or 2 way SMP machines, very very
few people have 8 way SMPs and even fewer have 32 way SMPs. And that
distribution will remain quite constant.

So ask yourself - should the Linux kernel team do an incredible amount
of work, destroying performance and reliability in the overwhelmingly
common case, for the benefit of a very, very small number of machines?

That is the question that I posed to Linus 4 years ago and he basically
agreed.

By the way - this doesn't mean that we don't have an answer for those
16-1024 way SMP boxes; we do, that's what the Linux clustering effort
is all about. As we run out of steam on the SMP front, you'll see more
and more work on the clustering front to scale up well beyond what anyone
else can do. I'm working on this with one of the government labs - they
want to scale things to 10,000 CPUs and they darn well don't expect Linux
SMP to do that :-)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:45    [W:0.076 / U:1.412 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site