lkml.org 
[lkml]   [2006]   [Jan]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: [PATCH 000 of 5] md: Introduction
On Saturday January 21, akropel1@rochester.rr.com wrote:
> NeilBrown <neilb@suse.de> wrote:
> > In line with the principle of "release early", following are 5 patches
> > against md in 2.6.latest which implement reshaping of a raid5 array.
> > By this I mean adding 1 or more drives to the array and then re-laying
> > out all of the data.
>
> I've been looking forward to a feature like this, so I took the
> opportunity to set up a vmware session and give the patches a try. I
> encountered both success and failure, and here are the details of both.
>
> On the first try I neglected to read the directions and increased the
> number of devices first (which worked) and then attempted to add the
> physical device (which didn't work; at least not the way I intended).
> The result was an array of size 4, operating in degraded mode, with
> three active drives and one spare. I was unable to find a way to coax
> mdadm into adding the 4th drive as an active device instead of a
> spare. I'm not an mdadm guru, so there may be a method I overlooked.
> Here's what I did, interspersed with trimmed /proc/mdstat output:

Thanks, this is exactly the sort of feedback I was hoping for - people
testing thing that I didn't think to...

>
> mdadm --create -l5 -n3 /dev/md0 /dev/sda /dev/sdb /dev/sdc
>
> md0 : active raid5 sda[0] sdc[2] sdb[1]
> 2097024 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>
> mdadm --grow -n4 /dev/md0
>
> md0 : active raid5 sda[0] sdc[2] sdb[1]
> 3145536 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

I assume that no "resync" started at this point? It should have done.

>
> mdadm --manage --add /dev/md0 /dev/sdd
>
> md0 : active raid5 sdd[3](S) sda[0] sdc[2] sdb[1]
> 3145536 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
>
> mdadm --misc --stop /dev/md0
> mdadm --assemble /dev/md0 /dev/sda /dev/sdb /dev/sdc /dev/sdd
>
> md0 : active raid5 sdd[3](S) sda[0] sdc[2] sdb[1]
> 3145536 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

This really should have started a recovery.... I'll look into that
too.

>
> For my second try I actually read the directions and things went much
> better, aside from a possible /proc/mdstat glitch shown below.
>
> mdadm --create -l5 -n3 /dev/md0 /dev/sda /dev/sdb /dev/sdc
>
> md0 : active raid5 sda[0] sdc[2] sdb[1]
> 2097024 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>
> mdadm --manage --add /dev/md0 /dev/sdd
>
> md0 : active raid5 sdd[3](S) sdc[2] sdb[1] sda[0]
> 2097024 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>
> mdadm --grow -n4 /dev/md0
>
> md0 : active raid5 sdd[3] sdc[2] sdb[1] sda[0]
> 2097024 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
> ...should this be... --> [4/3] [UUU_] perhaps?

Well, part of the array is "4/4 UUUU" and part is "3/3 UUU". How do
you represent that? I think "4/4 UUUU" is best.

> [>....................] recovery = 0.4% (5636/1048512) finish=9.1min speed=1878K/sec
>
> [...time passes...]
>
> md0 : active raid5 sdd[3] sdc[2] sdb[1] sda[0]
> 3145536 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>
> My final test was a repeat of #2, but with data actively being written
> to the array during the reshape (the previous tests were on an idle,
> unmounted array). This one failed pretty hard, with several processes
> ending up in the D state. I repeated it twice and sysrq-t dumps can be
> found at <http://www.kroptech.com/~adk0212/md-raid5-reshape-wedge.txt>.
> The writeout load was a kernel tree untar started shortly before the
> 'mdadm --grow' command was given. mdadm hung, as did tar. Any process
> which subsequently attmpted to access the array hung as well. A second
> attempt at the same thing hung similarly, although only pdflush shows up
> hung in that trace. mdadm and tar are missing for some reason.

Hmmm... I tried similar things but didn't get this deadlock. Somehow
the fact that mdadm is holding the reconfig_sem semaphore means that
some IO cannot proceed and so mdadm cannot grab and resize all the
stripe heads... I'll have to look more deeply into this.

>
> I'm happy to do more tests. It's easy to conjur up virtual disks and
> load them with irrelevant data (like kernel trees ;)

Great. I'll probably be putting out a new patch set late this week
or early next. Hopefully it will fix the issues you can found and you
can try it again..


Thanks again,
NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2006-01-22 23:55    [from the cache]
©2003-2011 Jasper Spaans