lkml.org 
[lkml]   [2010]   [Nov]   [5]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: [Ksummit-2010-discuss] checkpoint-restart: naked patch
    > I'm probably missing something but can't you stop the application
    > using PTRACE_ATTACH? You wouldn't need to hijack a signal or worry
    > about -EINTR failures (there are some exceptions but nothing really to
    > worry about). Also, unless the manager thread needs to be always
    > online, you can inject manager thread by manipulating the target
    > process states while taking a snapshot.

    In fact CryoPid uses exactly the same approach and has been around for around 5
    years. Not as much development effort has gone into CryoPid as DMTCP and so its
    application coverage is not as broad. But the larger issue for using PTRACE is
    that you can not have two superiors tracing the same inferior process. So if you
    want to checkpoint a gdb session or valgrind or tmux or strace, then you can not
    directly control and quiesce the inferior process being traced.

    Beyond that, we also have a vision (not yet implemented) of process
    virtualization by which one can change the behavior of a program. For example,
    if a distributed computation runs over infiniband, can we migrate to a TCP/IP
    cluster. For this, one needs the flexibility of wrappers around system calls.
    This vision of process virtualization also motivates why our own research
    project has steered away from in-kernel C/R.

    > > But since you ask :-), there is one thing on our wish list. We
    > > handle address space randomization, vdso, vsyscall, and so on quite
    > > well. We do not turn off address space randomization (although on
    > > restart, we map user segments back to their original addresses).
    > > Probably the randomized value of brk (end-of-data or end of heap) is
    > > the thing that gave us the most troubles and that's where the code
    > > is the most hairy.
    >
    > Can you please elaborate a bit? What do you want to see changed?

    Yes, we would love to elaborate :-). We began DMTCP with Linux kernel 2.6.3.
    When Address Space Layout Randomization was added, we were forced to add some
    hacks concerning VDSO location and end-of-data. end-of-data is the uglier part.
    On restart, we directly map each memory segment into the original address at
    checkpoint time. The issue comes in mapping heap back to its original location.
    We call sbrk() to reset the end-of-data to the end of the original heap. This
    fails if the randomized beginning-of-data/end-of-data given to us by the kernel
    for the restarted process is too far away from where we want to remap the heap.
    To get around this, we play games with legacy layout, other personality
    parameters, and RLIMIT_STACK (since the kernel uses RLIMIT_STACK in choosing the
    appropriate memory layout).

    For our wish list, we would like a way of telling the kernel, where to set
    beginning-of-data/end-of-data. Curiously enough, at the time at which Linux
    started randomizing address space, there was discussion of offering exactly this
    facility for the sake of legacy programs, but it turned out not to be needed.

    Similarly, it would be nice to tell the kernel where we want the VDSO page.
    Currently, we get around this by keeping two VDSO pages, the old one which we
    restore and the new one specified to us by the kernel when the restart process
    is created. This works well for, and so controlling the address of the VDSO page
    is less important for us.

    > I don't have much idea about the scope of the whole thing, so please
    > feel free to hammer senses into me if I go off track. From what I
    > read, it seems like once the target process is stopped, dmtcp is able
    > to get most information necessary from kernel via /proc and other
    > methods but the paper says that it needs to intercept socket related
    > calls to gather enough information to recreate them later. I'm
    > curious what's missing from the current /proc. You can map socket to
    > inode from /proc/*/fd which can be matched to an entry in
    > /proc/*/net/PROTO to find out the addresses and most socket options
    > should be readable via getsockopt. Am I missing something?

    The design of DMTCP was decided upon roughly during the period from Linux 2.6.3
    through Linux 2.6.18. At that time, /proc/*/net did not exist. You are right
    that this can provide much better design for DMTCP and eliminate some of our
    wrappers. Thanks very much for pointing this out. We are now egar to implement a
    new design based on /proc/*/net in the near future.

    Since /proc/*/net provides a simpler design for sockets, we started wondering
    what other simplifications may be possible. Here is one possibility, in the case
    of shared file descriptors, DMTCP goes through two barriers in order to decide
    which process will be responsible for checkpointing which shared-file
    descriptor. It works and the overhead is reasonable, but if you have additional
    suggestion for this case, we would be very interested.

    > I think this is why userland CR implementation makes much more sense.
    > Most of states visible to a userland process are rather rigidly
    > defined by standards and, ultimately, ABI and the kernel exports most
    > of those information to userland one way or the other. Given the
    > right set of needed features, most of which are probabaly already
    > implemented, a userland implementation should have access to most
    > information necessary to checkpoint without resorting to too messy
    > methods and then there inevitably needs to be some workarounds to make
    > CR'd processes behave properly w.r.t. other states on the system, so
    > userland workarounds are inevitable anyway unless it resorts to
    > preemtive separation using namespaces and containers, which I frankly
    > think isn't much of value already and more so going forward.

    Its a very good point and we agree completely. Here are some examples where we
    believe, a userland component is inevitable even if one begins with in-kernel
    C/R:
    1. NSCD deamon -- in calls to libc::gethostname() etc. libc arranges for
    communication by sharing a memory segment with application process. Our code
    recognized this shared memory because it starts with /var/*/nscd.
    2. syslogd -- Application using syslog have a socket open to the syslog deamon.
    DMTCP makes a system call to turnoff logging at checkpoint time.
    3. X-windows terminals -- xterm/gnome-terminal/konsole all emulate ANSI
    terminals. They support various ANSI features such as setting up scrolling
    region above status line. GNU screen uses the scrolling region feature. On
    restart, we have to convince GNU screen and similar programs to re-initialize
    their ANSI terminal. We do this successfully by sending a SIGWINCH on
    restart, since it has to re-initialize the ANSI terminal whenever the window
    size changes. In fact we send one SIGWINCH and when the application calls
    ioctl(), to get the window size, we lie and say that the window size changed,
    and we then send another SIGWINCH from within the wrapper to force the
    application to recheck the window size and discover that the window is back
    to its original size.
    4. X11 apps -- The current approach to checkpointing X-windows application is to
    checkpoint them within a VNC server. We plan to add wrappers around calls to
    libX11.so so that we can discover the state of an X11 window at checkpoint
    time and then restart just the single X11 application. This avoids the need
    to also checkpoint the X11 server which minimized the size of the the
    checkpoint image that needs to be written to the disk.
    5. GNU Screen -- DMTCP sets SCREEN_DIR to a temp directory in order to avoid the
    issue that occurs when the setsuid screen process tries to across
    /var/run/uscreen. Otherwise we would have difficulty at restart time when the
    checkpoint image has no setsuid privilege. We don't know if there are similar
    issues with an in-kernel C/R.

    We really enjoyed this discussion. If you are interested, we would be happy to
    talk further by phone in order to take advantage of the higher bandwidth.

    Best,
    -Gene and Kapil


    \
     
     \ /
      Last update: 2010-11-06 01:39    [W:0.031 / U:30.544 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site