lkml.org 
[lkml]   [1998]   [Jun]   [11]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Remote fork() and Parallel Programming
Hi,

lm@bitmover.com (Larry McVoy) wrote:

>Your technical comments aren't very technical. In my experience, it is well
>intentioned people just like you that cause creeping featurism [...]

I hope not. My work has resulted in the addition of some, IMHO, very
powerful _functionalities_ to the Linux kernel; while compared to some other
subsystems currently inside the Linux kernel, the added complexity and code
is negligible.

> [...] I'd much
>rather see you use those good intentions to keep stuff out of the kernel
>rather than shovel stuff into the kernel.

I have been trying to keep things out of the kernel during my work on
Linux, but you _have_ to provide hooks to get into this monolithic kernel.

>Getting back to your comments. The reason I think they aren't that technical
>is that you are arguing for a bunch of features which /sound/ great. They
>are not necessary or useful in practice and that is something that only
>experience will prove. As someone who has that experience (and has payed
>close attention to the experience of others working on these problems), I
>am trying to let you know that what you are asking for is not a good idea.
>It's not a marginal thing like it might be a good idea, it's an absolute
>thing: it's not a good idea.

They sound great because they are great. I wonder how experience could be of
use to you here. First one should have some mechanism before it can be used
in practice. Unless someone has polled the Linux programmers to see if they
would be willing to use such new facilities or not, it is very hard to
predict that people will not use them in the future.

>: Most of those programs that use an ordinary fork(), especially if the forked
>: processes will exist for a long time.
>
>Again, please be more specific. What you are doing is called "armchair
>architecture". You /think/ that these applications exist and would use
>this facility. I used to think just like you until I went out and waded
>through the real applications that can and do make use of clusters. They
>are nothing like what you are imagining.

As I see things, distributed programming should not be (very) different from
programming a single computer. A move in making this goal a reality is
in the right direction. _Any_application_ that forks a number of processes
can then spawn its children in other computers. This is the ultimate goal. In
Linux, we can gradually try to get as close to it as possible. I hope this
explains my insistance on there being many applications who could benefit.

BTW, Why do you consider some things that usually come in advanced text
books on operating systems as imaginary? If people before you and I were more
willing to incorporate the more modern concepts in their designs, then maybe
things would be more real.

>: Simple for who?? The application programmer or the kernel programmer? I say
>: let us make things simple for the application programmer.
>
>Wrong answer. Complexity in the base of the system is a bad idea. There are
>good reasons that Unix has a "simple" design.

It is very hard to define simplicity and complexity. For example, I have a
hard time considering a monolithic OS design as simple. Most UNIX variants
are just a jungle of code (they could have been simple once). They are
complicated by design.

But I understand if the design seems simple to the people familiar with it.
We are more willing to accept things we have known and used, and reject
things that are not very familiar. The danger is when we try to impose our
interpretations of simplicity and complexity on others.

>Furthermore, the "application programmers" that you are tlaking about are
>more sophisticated than most kernel programmers. They neither need nor
>want the features you are asking for. Go spend some time at the national
>labs (any national labs, I don't care what country) and listen and learn
>what they do. They would be completely uninterested in what you are
>proposing, they already know it doesn't work.

Again, I wonder how many of the people you refer to have actually used the
mechanisms I have in mind. It is _good_ to try to listen to people who will
be the actual users of a system, but we have to remeber that most of these
people use computers just as a tool to solve their own problems. they are not
very interested in investigating newer approches. So sometimes it is even
_better_ to provide them newer methods. They may very well like it.

>: Just consider a group of people with heavy scientific applications. They
>: can start their long-running programs anytime and on any machine. A lot of
>: these people will need hundreds or even thousands of processors to run their
>: applications in parallel. It is good to have SMP machines in a cluster, but
>: they are not the answer.
>
>Funny you should mention these applications. Have you ever talked to one
>of the people that work on systems like this? Have you ever spent time
>understanding the basic issues? How do you propose to have 2 applications
>share a cluster at the same time without thrashing? There's only one
>answer and it is gang scheduling. Think about gang scheduling for a
>while and then try and explain to me how process migration is going to
>not completely screw up the scheduling.

If needed, the (gang) scheduler can "lock" a process in a machine for any
duration.

>Maybe it's just me, but I have a real problem with you trying to tell the
>kernel list what ought to be done when it is so obvious that you are not
>speaking from experience.

I see no harm in making a number of suggestions to the Linux kernel list;
especially when I think they will benefit Linux. I was not aware of any
preconditions for posting here.

>: That's why one sees many more threading applications under Windows NT than
>: under Linux.
>
>You don't think it could have anything to do with the thousands of paid
>programmers that work on NT apps vs the unpaid programmers that work
>o Linux?

And how many Linux applications actually use threads??

>DIPC is tiny by comparision to what you are proposing. Let's just
>take process migration. Suppose you spend an hour walking through
>the kernel and come back with a list of places that you will need to
>change to make process migration work. Please consider at least the
>following: open sockets, open files, CWD, signals, process relationships,
>process namespace, shared memory. I really think it would be very
>useful for you to do this /before/ you tell the kernel community what
>they should be doing. Better yet, go implement waht you are proposing.
>Maybe I'm dead wrong and you can indeed do it in a very small, nice way.
>That would be great.

My guess is that one does not have to accomplish something before suggesting
it be done. Anyway, my experience with DIPC did not encourage me to
continue working on Linux.

And BTW, doesn't a flexible checkpoint/restart mechanism (something the
programmer can use _any_time_ in his application) require solving most of
the problems you mention?


Larry, I don't think continuing this discussion will succeed in converting
any of us, as this is looking more and more like an ideological debate to
me. Thanks for your very interesting and useful comments.


-Kamran Karimi
PS- My initial post to this thread was simply asking if some one is working
on remote fork() and dynamic load balancing. I was surprised to see some of
the replies I received. What I expected was a bit more positive attitude that
welcomed such investigations, even if some of us believe them to be useless,
meaningless, etc. I hope people who intend to work on these very interesting
problems are not discouraged in any way.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu

\
 
 \ /
  Last update: 2005-03-22 13:43    [W:0.093 / U:0.016 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site