[lkml]   [2009]   [Aug]   [26]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [patch] document flash/RAID dangers
    On 08/26/2009 08:40 AM, Theodore Tso wrote:
    > On Wed, Aug 26, 2009 at 07:58:40AM -0400, Ric Wheeler wrote:
    >>> Drive in raid 5 failed; hot spare was available (no idea about
    >>> UPS). System apparently locked up trying to talk to the failed drive,
    >>> or maybe admin just was not patient enough, so he just powercycled the
    >>> array. He lost the array.
    >>> So while most people will not agressively powercycle the RAID array,
    >>> drive failure still provokes little tested error paths, and getting
    >>> unclean shutdown is quite easy in such case.
    >> Then what we need to document is do not power cycle an array during a
    >> rebuild, right?
    > Well, the softwar raid layer could be improved so that it implements
    > scrubbing by default (i.e., have the md package install a cron job to
    > implement a periodict scrub pass automatically). The MD code could
    > also regularly check to make sure the hot spare is OK; the other
    > possibility is that hot spare, which hadn't been used in a long time,
    > had silently failed.

    Actually, MD does this scan already (not automatically, but you can set up a
    simple cron job to kick off a periodic "check"). It is a delicate balance to get
    the frequency of the scrubbing correct.

    On one hand, you want to make sure that you detect errors in a timely fashion,
    certainly detection of single sector errors before you might develop a second
    sector level error on another drive.

    On the other hand, running scans/scrubs continually impacts the performance of
    your real workload and can potentially impact your components' life span by
    subjecting them to a heavy workload.

    Rule of thumb seems from my experience is that most people settle in with a scan
    once a week or two (done at a throttled rate).

    >> In the end, there are cascading failures that will defeat any data
    >> protection scheme, but that does not mean that the value of that scheme
    >> is zero. We need to be get more people to use RAID (including MD5) and
    >> try to enhance it as we go. Just using a single disk is not a good
    >> thing...
    > Yep; the solution is to improve the storage devices. It is *not* to
    > encourage people to think RAID is not worth it, or that somehow ext2
    > is better than ext3 because it runs fsck's all the time at boot up.
    > That's just crazy talk.
    > - Ted



     \ /
      Last update: 2009-08-26 15:13    [W:0.021 / U:21.508 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site