[lkml]   [2010]   [Nov]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [Ksummit-2010-discuss] checkpoint-restart: naked patch
    > 1. Resource management. Any large HPC cluster should be 100% busy and
    > as such you will often fill in the gaps with low priority jobs which
    > may need to be preempted. These low priority jobs need to give up their
    > resources (memory, interconnect resources etc) whenever something
    > important comes in.
    > 2. Fault tolerance. Failures are a fact of life for any decent sized
    > cluster. As the cluster gets larger these failures become very common.
    > Speaking from an industry perspective, MTBF rates measured in the order
    > of several hours for large commodity clusters are not surprising. We at
    > IBM improve on that with hardware and system design, but there is only
    > so much you can do. The failures also happen at the Linux kernel level
    > so even if we had 100% reliable systems we would still have issues.

    We have also been somewhat involved in HPC. Grant provides a nice
    summary of the two usage scenarios of checkpoint-restart that we have also

    Since there is continuing discussion of HPC, I was a little surprised that
    there has not been more discussion of BLCR (Berkeley Lab Checkpoint/Restart).
    A brief introduction to BLCR follows, in case it's of interest.

    In the HPC space, we have observed that many sites use BLCR for
    checkpoint-restart. BLCR is based on a kernel module, and so represents a third
    approach. As mentioned on the FAQ, BLCR can checkpoint/restart a
    process tree/group/session but has certain limitations, such as not supporting
    sockets, ptys, and restoring original pids on restart only if there is no
    collision with current pids. Nevertheless, BLCR has achieved wide usage in the
    HPC community. Quoting from the BLCR FAQ:

    Q: Does BLCR support checkpointing parallel/distributed applications?

    Not by itself. But by using checkpoint callbacks (see previous FAQ). some
    MPI implementations have made themselves checkpointable by BLCR. You can
    checkpoint/restart an MPI application running across an entire cluster of
    machines with BLCR, without any application code modifications, if you use
    one of these MPI implementations (listed alphabetically):
    * LAM/MPI 7.x or later
    * MPICH-V 1.0.x
    * MVAPICH2 0.9.8 or later
    * Open MPI 1.3 or later

    Q: Is BLCR integrated with any batch systems?

    We are aware of the following, but we are not always informed of new
    efforts to integrate with BLCR. For the most up-to-date information you
    should consult the support channels of your favorite batch system.
    * TORQUE version 2.3 and later
    Support for serial and parallel jobs, including periodic checkpoints and
    * SLURM version 2.0 and later
    Support for automatic (periodic) and manually requested checkpoints.
    * SGE (aka Sun Grid Engine)
    Information on configuring SGE to use BLCR can be found here. There is
    also a thread on the list about modifications to those
    instructions. The thread begins with this posting.
    * LSF
    Information on configuring LSF to use BLCR can be found in this posting
    on the list.
    * Condor
    Information on configuring Condor to use BLCR to checkpoint "Vanilla
    Universe" jobs with the help of Parrot can be found here.

    - Gene

     \ /
      Last update: 2010-11-19 07:35    [W:0.022 / U:24.200 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site