lkml.org 
[lkml]   [2009]   [Sep]   [17]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: MD/RAID time out writing superblock
    Tejun Heo wrote:
    > Hello,
    >
    > Chris Webb wrote:
    >> Hi Tejun. Thanks for following up to this. We've done some more
    >> experimentation over the last couple of days based on your
    >> suggestions and thoughts.
    >>
    >> Tejun Heo <tj@kernel.org> writes:
    >>> Seriously, it's most likely a hardware malfunction although I can't tell
    >>> where the problem is with the given data. Get the hardware fixed.
    >> We know this isn't caused by a single faulty piece of hardware,
    >> because we have a cluster of identical machines and all have shown
    >> this behaviour. This doesn't mean that there isn't a hardware
    >> problem, but if there is one, it's a design problem or firmware bug
    >> affecting all of our hosts.
    >
    > If it's multiple machines, it's much less likely to be faulty drives,
    > but if the machines are configured mostly identically, hardware
    > problems can't be ruled out either.
    >
    >> There have also been a few reports of problems which look very
    >> similar in this thread from people with somewhat different hardware
    >> and drives to ours.
    >
    > I wouldn't connect the reported cases too eagerly at this point. Too
    > many different causes end up showing similar symptoms especially with
    > timeouts.
    >
    >>> The aboves are IDENTIFY. Who's issuing IDENTIFY regularly? It isn't
    >>> from the regular IO paths or md. It's probably being issued via SG_IO
    >>> from userland. These failures don't affect normal operation.
    >> [...]
    >>> Oooh, another possibility is the above continuous IDENTIFY tries.
    >>> Doing things like that generally isn't a good idea because vendors
    >>> don't expect IDENTIFY to be mixed regularly with normal IOs and
    >>> firmwares aren't tested against that. Even smart commands sometimes
    >>> cause problems. So, finding out the thing which is obsessed with the
    >>> identity of the drive and stopping it might help.
    >> We tracked this down to some (excessively frequent!) monitoring we
    >> were doing using smartctl. Things were improved considerably by
    >> stopping smartd and disabling all callers of smartctl, although it
    >> doesn't appear to have been a cure. The frequency of these timeouts
    >> during resync seems to have gone from about once every two hours to
    >> about once a day, which means we've been able to complete some
    >> resyncs whereas we were unable to before.
    >
    > That's interesting. One important side effect of issuing IDENTIFY is
    > that they will serialize command streams as they are not NCQ commands
    > and thus could change command patterns significantly.
    ..

    SMART is the opcode that is most frequently implicated here, not IDENTIFY.
    Note that even a barrier FLUSH CACHE is non NCQ and will serialize the stream.

    Cheers



    \
     
     \ /
      Last update: 2009-09-17 15:37    [W:4.648 / U:0.552 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site