[lkml]   [2009]   [Apr]   [14]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [PATCH 10/30] cr: core stuff

    Alexey Dobriyan wrote:
    > On Tue, Apr 14, 2009 at 01:22:03AM -0400, Oren Laadan wrote:
    >> Alexey Dobriyan wrote:
    >>> * add struct file_operations::checkpoint
    >>> The point of hook is to serialize enough information to allow restoration
    >>> of an opened file.
    >>> The idea (good one!) is that the code which supplies struct file_operations
    >>> know better what to do with file.
    >> Actually, credit is due to Dave Hansen (or Christoph Hellwig, or both?).
    >>> Hook gets C/R context (a cookie more or less) on which dump code can
    >>> cr_write() and small restrictions on what to write: globally unique object id
    >>> and correct object length to allow jumping through objects.
    >>> For usual files on on-disk filesystem add generic_file_checkpoint()
    >>> Add ext3 opened regular files and directories for start.
    >>> No ->checkpoint, checkpointing is aborted -- deny by default.
    >>> FIXME: unlinked, but opened files aren't supported yet.
    >>> * C/R image design
    >>> The thing should be flexible -- kernel internals changes every day, so we can't
    >>> really afford a format with much enforced structure.
    >>> Image consists of header, object images and terminator.
    >>> Image header consists of immutable part and mutable part (for future).
    >>> Immutable header part is magic and image version: "LinuxC/R" + __le32
    >>> Image version determines everything including image header's mutable part.
    >>> Image version is going to be bumped at earliest opportunity following changes
    >>> in kernel internals.
    >>> So far image header mutable part consists of arch of the kernel which dumped
    >>> the image (i386, x86_64, ...) and kernel version as found in utsname.
    >>> Kernel version as string is for distributions. Distro can support C/R for
    >>> their own kernels, but can't realistically be expected to bump image version --
    >>> this will conflict with mainline kernels having used same version. We also don't
    >>> want requests for private parts of image version space.
    >> So far so good, like in our patch-set.
    >> You also need to address differences in configuration (kernel could
    >> have been recompiled) and runtime environment (boot params, etc).
    >> We deferred this issue to a later time.
    >>> Distro expected to keep image version alone and on restart(2) check utsname
    >>> version and compare it against previously release kernel versions and based
    >>> on that turn on compatibility code.
    >> Are you suggesting that conversion of a checkpoint image from an older
    >> version to a newer version be done in the kernel ?
    > For mainline kernel it's completely unrealistic to support all backwards
    > compatibility code for previous versions. Some mythical userspace
    > program will convert images.
    > But it's completely realistic and much easier for distro kernel because
    > distro kernel doesn't generally include patches with significant in-kernel
    > internals changes, so they simply can support
    > '2.6.26-1-amd64' => '2.6.26-2-amd64' situation.
    > Distros can write conversion program too, but I don't expect they will.
    >> It may work for a few versions, and then you'll get a spaghetti of
    >> #ifdef's in the code, together with a plethora of legacy code.
    > Expectation is for one kernel branch like RHEL5 kernel updates during
    > RHEL5 lifecycle.
    > For RHEL5 => RHEL6, it's up to them what to do.
    > Anyway distro can add compat code _anyway_, for this we help them with
    > this image format tweak, so they won't bug mainline with "reserve bit 31
    > for Red Hat".
    > Image version is kept small (__le32) for this reason too :-)
    >> It is much better/easier to handle checkpoint image transformations
    >> in user space. The kernel will only understand its "current" version
    >> (for some definition of version).
    >>> Object image is very flexible, the only required parts are a) object type (u32)
    >>> and b) object total length (u32, [knocks wood]) which must be at the beginning
    >>> of an image. The rest is not generic C/R code problem.
    >>> Object images follow one another without holes. Holes are in theory possible but
    >>> unneeded.
    >> When would you need holes ?
    >>> Image ends with terminator object. This is mostly to be sure, that, yes, image
    >>> wasn't truncated for some reason.
    >>> * Objects subject to C/R
    >>> The idea is to not be very smart but directly dump core kernel data structures
    >>> related to processes. This includes in this patch:
    >>> struct task_struct
    >>> struct mm_struct
    >>> VMAs
    >>> dirty pages
    >>> struct file
    >>> Relations between objects (task_struct has pointer to mm_struct) are fullfilled
    >>> by dumping pointed to object first, keeping it's position in dumpfile and saving
    >>> position in a image of pointe? object:
    >> Unless you use the physical position to actually lseek to there to
    >> re-read the data, there is no reason to use the actual position. In
    >> fact it is easier to debug when the shared object identifier is a
    >> simple counter.
    >> If you do use it to lseek, then it's a poor decision -- sounds fragile:
    >> what if we change the file (legitimately) adding data in the middle -
    >> the whole concept breaks.
    > Adder of data is expected to understand image format and update all references
    > just like surgeon is expected to understand human anatomy.
    >>> struct cr_image_task_struct {
    >>> cr_pos_t cr_pos_mm;
    >>> ...
    >>> };
    >>> Code so far tries hard to dump objects in certain order so there won't be any loops.
    >>> This property of process that dumpfile can in theory be O_APPEND, will likely be
    >>> sacrifised (read: child can ptrace parent)
    >> The ability to streamline the checkpoint image IMHO is invaluable.
    >> It's the unix way (TM) of doing things; it makes the process pipe-able.
    >> You can do many nice things when the checkpoint can be streamed: you
    >> can compress, sign, encrypt etc on the fly without taking additional
    >> diskspace. You can transfer over the network (e.g. for migration),
    >> or store remotely without explicit file system support. You can easily
    >> transform the stream from one c/r version to another etc.
    >> This should be a design principle. In my experience I never hit a wall
    >> that forced me to "sacrifice" this decision.
    >>> sacrifised (read: child can ptrace parent)
    >> Hmmm... if all tasks are created in user space, then this specific
    >> becomes a no-brainer !
    > No!

    Actually yes :)

    > A ptraces B. Container is checkpointed.
    > Kernel realizes ptrace is going on. A and B in theory can have any
    > realitionship.
    > Consequently, kernel doesn't know in which order to dump A and B.
    > And there is no such order:
    > *) A can be parent of B (you dump A, B),
    > *) A can be child of B (you want to dump B, A, but this conflicts with
    > ->real_parent order)
    > *) A and B just tasks (any order).

    Current code does not support ptrace() - which has a multitude
    if tidy-bits issues to solve during restart regardless.

    However, creating tasks in userspace uses (and will uses) only
    "real" process relationships, not ptrace-relationships, when it
    comes to decide on the fork/clone order.

    Technically, that can be done in checkpoint (dumping the task tree)
    or in restart-user-space (rearranging the data before fork/clone).

    > I'm showing that whole issue can be avoided:

    If the issue can be avoided, then why would you need to sacrifice
    the stream-ability of the checkpoint image ?

    > *) all tasks are simply created regardless of who is parent of whom
    > (see kernel_thread())
    > *) Every task_struct image among other things contains references to
    > ->real_parent and ->parent.
    > *) After every task is created it's time to change references:
    > **) lookup who is ->real_parent, change ->real_parent _by hand_
    > not with some "correct clone(2)" order.
    > **) lookup who is ->parent, change ->parent.
    > You're probably escaping all of this with object numbers?

    (Will be) escaping this by arranging to fork/clone in the proper order.

    >>> * add struct vm_operations_struct::checkpoint
    >>> just like with files, code that creates special VMAs should know what to do with them
    >>> used.
    >>> just like with files, deny checkpointing by default
    >>> So far used to install vDSO to same place.
    >> VDSO can be a troublemaker; in recent kernels its location in the MM
    >> can be randomized.
    > See arch_setup_additional_pages() patch.
    >> It is not necessarily immutable - it can reflect
    >> ynamic kernel data. It may contain different code on newer versions,
    >> so must be compared or worked around during restart etc.
    > i386 if I'm not mistaken only contain syscall entry code, but, yes,
    > generally one should check if PC is inside such page.

    If you up restart on a different kernel that has a different VDSO,
    then you need to bring the old VDSO with you, and tweak it so it
    pulls the dynamic kernel data from the right place. Ugh ... :(


    >>> * add checkpoint(2)
    >>> Done by determining which tasks are subject to checkpointing, freezeing them,
    >>> collecting pointers to necessary kernel internals (task_struct, mm_struct, ...),
    >>> doing that checking supported/unsupported status and aborting if necessary,
    >>> actual dumping, unfreezeing/killing set of tasks.
    >>> Also in-checkpoint refcount is maintained to abort on possible invisible changes.
    >>> Now it works:
    >>> For every collected object (mm_struct) keep numbers of references from
    >>> other collected objects. It should match object's own refcount.
    >>> If there is a mismatch, something is likely pinning object, which means
    >>> there is "leak" to outside which means checkpoint(2) can't realistically and
    >>> without consequences proceed.
    >>> This is in some sense independent check. It's designed to protect from internals
    >>> change when C/R code was forgotten to be updated.
    >>> Userpsace supplies pid of root task and opened file descriptor of future dump file.
    >>> Kernel reports 0/-E as usual.
    >>> Runtime tracking of "checkpointable" property is explicitly not done.
    >>> This introduces overhead even if checkpoint(2) is not done as shown by proponents.
    >>> Instead any check is done at checkpoint(2) time and -E is returned if something is
    >>> suspicious or known to be unsupported.
    >>> FIXME: more checks especially in cr_check_task_struct().
    >>> * add restart(2)
    >>> Recreate tasks and evething dumped by checkpoint(2) as if nothing happened.
    >>> The focus is on correct recreating, checking every possibility that target kernel
    >>> can be on different arch (i386 => x86_64) and target kernel can be very different
    >>> from source kernel by mistake (i386 => x86_64 COMPAT=n) kernel.
    >>> restart(2) is done first by creating kernel thread and that demoting it to usual
    >>> process by adding mm_struct, VMAs, et al. This saves time against method when
    >>> userspace does fork(2)+restart(2) -- forked mm_struct will be thrown out anyway
    >>> or at least everything will be unmapped in any case.
    >> Do have figures to support your claims about "saves time" ?
    >> The *largest* component of the restart time, as you probably know,
    >> is the time it takes to restore the memory address space (pages, pages)
    >> of the tasks.
    >> If you do show that this optimization is worth our attention, then it
    >> takes < 10 lines to change current mktree.c to use CLONE_VM ... voila.
    >> I'm interested in hearing more convincing arguments in favor of kernel
    >> creations of restarting tasks (see my other post about it).
    > OK, in another post.
    >>> Restoration is done in current context except CPU registers at last stage.
    >>> This is because "creation is done by current" is in many, many places,
    >>> e.g. mmap(2) code.
    >>> It's expected that filesystem state will be the same. Kernel can't do anything
    >>> about it expect probably virtual filesystems. If a file is not there anymore,
    >>> it's not kernel fault, -E will be returned, restart aborted.
    >>> FIXME: errors aren't propagated correctly out of kernel thread context
    >> Heh .. I guess they always propagate correctly out of regular task
    >> context ;)
    > :-)

     \ /
      Last update: 2009-04-14 20:25    [W:0.050 / U:53.252 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site