Messages in this thread | | | Date | Tue, 2 Aug 2011 10:27:08 -0400 | From | Ted Ts'o <> | Subject | Re: corrupted ext4 1000gb filesystem (2.6.32, debian stable) |
| |
On Tue, Aug 02, 2011 at 02:14:19PM +0100, Luke Kenneth Casson Leighton wrote: > On Tue, Aug 2, 2011 at 1:15 AM, Luke Kenneth Casson Leighton > <luke.leighton@gmail.com> wrote: > > > ok, um... i have a bit more information about this situation, to > > report. two consecutive runs of fsck.ext4, and the filesystem still > > reports errors after the first run "corrected" all errors. i'd say > > that was a bit serious. > > rright - apologies but i've located the likely source of the problem > - e2fsck. the issue is that the bitmaps for the 3-way RAID1 mirror > were corrupted. thus, the filesystem would be fixed by e2fsck, only > to be completely buggered up by picking wildly inappropriate sections > of the drive... that presumably by either bad luck or by a powercut > and writes occurring at the time happened to be on inode blocks.
E2fsck doesn't depend on the bitmaps; those are regenerated based on the information from the inode tables.
Assuming that the disks are stable --- that is, a read from a block returns the same contents all the time, and writes are not lost (i.e., after a write, reads to that block return the written data consistently), then there should not be any corruptions found after the first run of e2fsck fixes all errors.
That being said, there have been cases where that's not true, and I consider that a bug in e2fsck.
*However*, if you have a RAID1 setup where the data on the disks are consistent, this can be the cause of much mischief. Depending on which disk you read from the mirror, you might get different results. Once that's the case, all bets with e2fsck are off.
I suggest you make sure that your RAID1 mirror is stable first of all; in general, you *have* to fix problems with the storage stack from the lowest level on up. First make sure the hard drives are all sane; then make sure the partition table and/or LVM setups are sane; then make sure any RAID setups are sane; and only *then* run a filesystem-level checker. This is true regardless of what file system you use.
Finally, I strongly recommend that when you are doing this kind of repair work, that you save a copy of everything you do useing a program like "script". A transcript of the e2fsck output can be critally useful. Reviewing the transcript can also be useful in identifying mistakes that you might have made during the recovery process.
Regards,
- Ted
P.S. Note that if you are running e2fsck, and you haven't mounted the disk yet, if you are seeing failures after a second run of e2fsck, then it obviously can be a failing in the ext4 kernel code. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |