lkml.org 
[lkml]   [2009]   [Jun]   [17]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Linux scheduler capabilities for batch jobs.
On Mon 2009-06-01 09:41:07, J Louis wrote:
> Hello All,
>
> I have been writing user land server code and increasingly I find
> myself writing resource management code which I think would be better
> handled by the scheduler. The problem can be described as "don't run
> this process if the machine is swapping." I would think that this
> would be a common enough need that it was already in the kernel. I'm
> hoping it is and I have simply overlooked it (part of the reason for
> this post,) but I've looked around a good bit and most of the
> scheduler enhancements have to do with real time and latency, not
> batch jobs.
>
> My problem is analogous to a parallel make. Say I have an 8 CPU
> machine, and I run "make -j8". If the total memory of the 8 jobs
> throws the machine into swap, it begins to thrash and runtime is
> awful. I believe this is aggravated by the scheduler trying to be

This seems to be racy by design.

make -j:
launch gcc1
gcc1 does preprocessing (10mb)
ok, still not swapping
launch gcc2
gcc2 does preprocessing (10mb)
...
gcc100 does preprocessing (10mb, 1GB total)
now gcc1..gcc100 start optimizing (100mb needed) and boom.

Pavel

--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html


\
 
 \ /
  Last update: 2009-06-17 11:47    [W:0.187 / U:0.392 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site