lkml.org 
[lkml]   [1998]   [Jun]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: Remote fork() and Parallel Programming
Date
: >My personal take on these issues is:
: >
: > BAD GOOD
: > --- ----
: > remote fork() remote exec()
: > process migration checkpoint / restart
: > dynamic load balancing static load balancing across machines
: > across machines + dynamic load balancing within
: > (SMP) machines.
:
: The items listed under "BAD" are more powerful that the "GOOD" ones:

Agreed. it does not follow from that, however, that they should be
implemented. One could implement "tar" in the kernel and that would
be a more powerful thing than open/close/read/write. It would also
be a bad idea.

: *) remote fork() handles the run-time state of a process, while remote
: exec() does not bother with that. Please keep in mind that the application
: programmer will have a hard time duplicating the run-time state of a process
: without support from the operating system.

Before you go about buggering up the kernel with a bunch of things that
sound good on marketing slides, perhaps you should ask yourself where
the applications are that need this service. Let's try that again: what
applications need rfork() and can't make do with rexec()? You might want
consider that you are taliking to the guy that started 100% of Sun's
clustering work, and spent 2 years pretty much 7 days a week working on
these problems, looking at customer applications, sorting through the
marketing hype versus real issues. I'm very pro clustering. I am not
in favor of adding complexity where there is no point.

: *) process migration is more flexible and more transparent than an explicit
: checkpoint / restart mechanism. dynamic load balancing and process migration
: should come together. The combination of these two eases the optimum use
: of the available computing resources, without bothering the programmer.

Yeah, and Java is going to save the world, threads are the greatest thing
since sliced bread, and OSI is coming back. Please step back and consider
that the things that have survived the test of time are small, simple
concepts. Just because it is possible to do something does not mean it
should be done.

: After all, the operating system is the only entity that can be aware of the
: current resource-usage situation in different computers. An application
: program(mer) can not know this entire state. More important, it can not
: in general predict the future, so in a static load balancing environment
: the reaction to changes in resource-usage will not be satisfactory.

Prove it. Show me a real parallel application that will not run
perfectly when statically load balanced over a bunch of machines (2 or
4 way SMP boxes) and dynamically load balanced within those machines.
When you start looking, you start learning about gang scheduling.
Once you know about that, this whole idea os process migration becomes
not just unnecessary, it becomes negative.

: What I want to say is that as an all-knowking authority, the operating
: system is in a good position to use mechanisms like remote fork() and dynamic
: load balancing for the good of _all_ the running application programs.

See my comment about gang scheduling. Without it, you get nowhere. With it,
process migration is a horrible idea. Checkpoint/restart is fine and one
could use it as a coarse level process migration.

: This will also need minimal recoding: The prgrammer will always call a
: fork(), and the OS will decide if the system call will be executed
: locally or remotely.

You really need to think this through. What about /tmp? Are you going to
have one global /tmp and /usr/tmp? That's a huge performance lose. If not,
when I fork() and land on another machine and expect my tmp files, where are
they?

: All the "GOOD" mechanisms,
: on the other hand, will ultimately require some work on the part of the
: application programmers.

Not true at all. login, make, xinit all need to be taught that they are
in a cluster. Other applications don't know and don't need to know.

: Yes, implementing this won't be easy, but it will be done only once, and
: then a large number of application programmers can use them.

Where exactly are those programmers and what exactly are the applications?

: The chicken-egg problem again :-) Many people will not start using clusters
: while programming them is difficult! What you suggest might ensure that this
: will remain so. I am sure you agree that we should better "change" the
: situation rather than "cope" with it.

No, I do not agree. Bad ideas should never be put in the kernel. That's
why Linux has clone() and not kernel threads.

: > [...] People who
: >haven't learned this lesson have repeatedly built overly complex and
: >expensive clusters. Let's not do that to Linux.
:
: Linux uses a monolithic kernel. I myself don't think this is a good thing,
: but we should make our efforts compatible with this design. Adding to the
: kernel is not a very bad thing here.

Disagree.

: In short: It might be better to add a few thousand lines of code to the
: kernel and be spared the troubles of adding some few hundred lined of code
: to _many_ application programs. I hope the net effect to be less lines of
: code and much less bugs.

Look, doing a clustering OS the simple way is going to be more than 100K
lines of code. Doing it the way you are discribing is an order of
magnitude worse.

: I hope Mr. Linus Torvalds will have a positive attitude to such changes.

He will, he does, his attitude, however, is much more in line with my
thinking - we've discussed this issue repeatedly over the years.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu

\
 
 \ /
  Last update: 2005-03-22 13:43    [W:0.042 / U:0.028 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site