lkml.org 
[lkml]   [2005]   [Jul]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: Fault tolerance. . .
From
Date
On Sun, 24 Jul 2005 21:59:59 EDT, John Richard Moser said:

> I'm thinking of application level fault tolerance using roll-back states
> or something weird, to restore the system as affected by that
> application to a point before the error. The obvious visual effect
> would be that if an application were to crash, it and potentially
> interrelated applications would suddenly reset to a state a few seconds
> to a few minutes earlier.

Google for "checkpoint-restart" - it's a big field in scientific
computing, where you don't want to lose the results of a 3 week run on a
supercomputer just because the system crashes 5 minutes before it's done.

(Just think - if they'd had a proper checkpointing scheme, most of the
Hitchhiker's trilogy wouldn't have happened... :)

> Maintaining the state is also easy:
>
> - When a file is changed, track the changes and attach them to the last
> state save
> - When memory pages are written to, cache the old copies first
> (unfortunately each page has to be made CoW after every state save)

This is actually a lot harder than it looks - most of the real-life applications
of checkpoint-restart have been to programs that were designed to play nice
with checkpointing. It's *really* hard to do it with a program that wasn't
designed to to be checkpointed, as you noticed yourself:

> This of course raises many questions and concerns that make this
> rediculous and probably not entirely possible:
>
> - What about huge modifications to files in a short time? Make a new
> file, then write 10,000,000,000 bytes past the end and watch it crash.
> - What about lost work in interrelated applications?
> - Will the system state remain consistent?
> - Will it crash over and over and over?
> - Connecting to named pipes? (easily handled, not discussed here)
> - Crashes are usually trappable, and then programs exit cleanly. They
> won't care about this
> - How does a process know to change course if it gets restored?

Exactly the sort of things that make it hard...
[unhandled content-type:application/pgp-signature]
\
 
 \ /
  Last update: 2005-07-25 17:00    [W:0.040 / U:0.052 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site