lkml.org 
[lkml]   [2010]   [Nov]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [Ksummit-2010-discuss] checkpoint-restart: naked patch
> 1. Resource management. Any large HPC cluster should be 100% busy and
> as such you will often fill in the gaps with low priority jobs which
> may need to be preempted. These low priority jobs need to give up their
> resources (memory, interconnect resources etc) whenever something
> important comes in.
>
> 2. Fault tolerance. Failures are a fact of life for any decent sized
> cluster. As the cluster gets larger these failures become very common.
> Speaking from an industry perspective, MTBF rates measured in the order
> of several hours for large commodity clusters are not surprising. We at
> IBM improve on that with hardware and system design, but there is only
> so much you can do. The failures also happen at the Linux kernel level
> so even if we had 100% reliable systems we would still have issues.

We have also been somewhat involved in HPC. Grant provides a nice
summary of the two usage scenarios of checkpoint-restart that we have also
observed.

Since there is continuing discussion of HPC, I was a little surprised that
there has not been more discussion of BLCR (Berkeley Lab Checkpoint/Restart).
A brief introduction to BLCR follows, in case it's of interest.

https://ftg.lbl.gov/CheckpointRestart/CheckpointRestart.shtml

In the HPC space, we have observed that many sites use BLCR for
checkpoint-restart. BLCR is based on a kernel module, and so represents a third
approach. As mentioned on the FAQ, BLCR can checkpoint/restart a
process tree/group/session but has certain limitations, such as not supporting
sockets, ptys, and restoring original pids on restart only if there is no
collision with current pids. Nevertheless, BLCR has achieved wide usage in the
HPC community. Quoting from the BLCR FAQ:

Q: Does BLCR support checkpointing parallel/distributed applications?

Not by itself. But by using checkpoint callbacks (see previous FAQ). some
MPI implementations have made themselves checkpointable by BLCR. You can
checkpoint/restart an MPI application running across an entire cluster of
machines with BLCR, without any application code modifications, if you use
one of these MPI implementations (listed alphabetically):
* LAM/MPI 7.x or later
* MPICH-V 1.0.x
* MVAPICH2 0.9.8 or later
* Open MPI 1.3 or later

Q: Is BLCR integrated with any batch systems?

We are aware of the following, but we are not always informed of new
efforts to integrate with BLCR. For the most up-to-date information you
should consult the support channels of your favorite batch system.
* TORQUE version 2.3 and later
Support for serial and parallel jobs, including periodic checkpoints and
qhold/qrls.
* SLURM version 2.0 and later
Support for automatic (periodic) and manually requested checkpoints.
* SGE (aka Sun Grid Engine)
Information on configuring SGE to use BLCR can be found here. There is
also a thread on the checkpoint@lbl.gov list about modifications to those
instructions. The thread begins with this posting.
* LSF
Information on configuring LSF to use BLCR can be found in this posting
on the checkpoint@lbl.gov list.
* Condor
Information on configuring Condor to use BLCR to checkpoint "Vanilla
Universe" jobs with the help of Parrot can be found here.

- Gene


\
 
 \ /
  Last update: 2010-11-19 07:35    [from the cache]
©2003-2011 Jasper Spaans