Messages in this thread | | | From | (Larry McVoy) | Subject | Re: SMP scalability: 8 -> 32 CPUs | Date | Sat, 28 Nov 1998 10:33:53 -0800 |
| |
"Eloy A. Paris" <eparis@ven.ra.rockwell.com> asks:
: Guys, excuse my ignorance, but does anyone know why Linus said in his : last Linux Journal interview that right now it is fairly unrealistic : to expect scaling from 8 to 32 CPUs? What's stopping Linux to scale in : terms of CPUs?
There are some slides on this topic at http://www.bitmover.com/talks/pp/ that cover the points; I'm not sure how meaningful they are without me handwaving and filling in the gaps.
Here's what I believe to be th thinking on SMP and Linux. These thoughts are my opinions but are based on quite a bit of experience with SMP at Sun and SGI and are also based on several conversations with Linus on the topic. So unless things have changed, this should still be valid.
The basic point is that if you do both SMP and UP have two choices:
- #ifdef SMP the kernel and essentially run one kernel on UP and a different kernel on SMP
- No ifdefs, run the SMP kernel on the UP's
The first choice is the route taken by SGI, the second is the route taken by Sun. I prefer the Sun route. The SGI route results in you having to fix different bugs on different platforms and it becomes a support nightmare.
The Sun route has the problem that you are taking/releasing locks that you didn't need to take/release because there wasn't another processor to worry about (this statement ignores bottom/top driver races, you still need some sort of locking there).
It's a basic truth that the more scaling you want, the more locks you'll need. Very, very quickly this will become an unmanagable problem - the kernel becomes a forest of locks - for example, there were thousands of locks in the Solaris kernel when I left Sun almost 5 years ago. When it gets that complex, the obvious route taken by a programmer is to just add one more to protect their new data structure. That's a lose - what the programme should do is to examine the call paths that can lead to accessing his data structure and determine if there is already a lock at a higher level which will cover this data structure. But that's too hard and nobody does it once the number of locks gets beyond a handful.
What does this all mean? It means that if you try and scale beyond a handful of processors, you complicate your kernel substantially and make it almost impossible to keep it a lightweight thing on a UP (look at Sun vs Linux lmbench numbers on the same hardware, that's all the proof you need).
Furthermore, most people have UP or 2 way SMP machines, very very few people have 8 way SMPs and even fewer have 32 way SMPs. And that distribution will remain quite constant.
So ask yourself - should the Linux kernel team do an incredible amount of work, destroying performance and reliability in the overwhelmingly common case, for the benefit of a very, very small number of machines?
That is the question that I posed to Linus 4 years ago and he basically agreed.
By the way - this doesn't mean that we don't have an answer for those 16-1024 way SMP boxes; we do, that's what the Linux clustering effort is all about. As we run out of steam on the SMP front, you'll see more and more work on the clustering front to scale up well beyond what anyone else can do. I'm working on this with one of the government labs - they want to scale things to 10,000 CPUs and they darn well don't expect Linux SMP to do that :-)
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |