lkml.org 
[lkml]   [1999]   [Jun]   [24]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: File Corruption Bug.. continued
Date

> I've now been through most of the 2.2.7->2.2.9 diff set discarding stuff
> that seems not to be a viable candiate.
>
> The remaining suspects are:
> o Quota - which has big 2.2.7->2.2.9 changes.
> o The small scsi changes (dubious)
> o A small mm change
>
> There are other candiates - notably
> TCP changes
> interrupt changes
> IRDA
> NFS
>
> The tcp changes ought to have shown up more than this does. The interrupt
> changes dont really seem to explain cross platform stuff. And I doubt most
> people running these tests were running irda. NFS is a possibility.
>
> So - are most people seeing the problems running quotas ? And would
someone
> with their brain firmly wrapped around the page cache/vfs verify this
change
> that was made is absolutely safe
>

I pasted Chris Adams' reply as well as it is related to my message.

Chris Adams <cadams@ro.com> wrote in message news:7kr74r$h48$1@sh1.ro.com...
> Once upon a time, Alan Cox <alan@lxorguk.ukuu.org.uk> said:
> >I've now been through most of the 2.2.7->2.2.9 diff set discarding stuff
> >that seems not to be a viable candiate.
> >
> >The remaining suspects are:
> >that was made is absolutely safe
>
> Well, I don't run quotas on the system I had the FS corruption with (it
> is my desktop box at work). I do have SCSI (aic7xxx, which had a driver
> update in 2.2.10 that now enables tagged queueing by default IIRC), but
> I don't have IRDA or NFS (client or server). I did have NFS compiled in
> 2.2.9 when I had trouble, but not in 2.2.10 when I had trouble again.
>
> With 2.2.5 on it, I ran a script (taken from the pages about the AMD K6
> with Linux) that basically does "make -j4" of the kernel and then
> compares the results from run to run. I use this as a good burnin test
> of systems these days. I ran this burnin on 2.2.5 for about 18 hours
> and it didn't have any trouble.
>
> What I will try to do (probably will be a day or two) is setup a scratch
> partition on my system and boot 2.2.10. I'll mount all the other
> partitions read-only, and then run the burnin script with the kernel
> source on the scratch partition. If it fails (and I think it will after
> a while), then at least I'll have a way to reproduce the problem and
> test changes.
>
> The first time I had corruption (under 2.2.9) I was doing something
> like:
>
> find /usr/local/src/linux -type f -name '*.[ch]' -print | xargs fgrep foo
>
> That would run okay, but then I'd try a different search and I'd start
> getting the "attempt to read past end of device" and readdir errors.
> Then, when I stopped the find and did "ls /usr/local/src/linux", I got
> an empty directory.
>
> IIRC I was doing something similar when the corruption happened again.
>
> I thought it was a hardware problem at first, because I am using an
> older drive. I've never seen corruption like this in all the time I
> have been using Linux (back to 0.99 days), although I do avoid the known
> bad development kernels always. But then I realized that I didn't get
> ANY SCSI errors and that when I run 2.2.5 I can't make the same thing
> happen no matter how hard I try.
> --
> Chris Adams <cadams@ro.com> - System Administrator

Hi,

I have a strong suspicion that the above may be an important clue. I
hadn't had any corruption with post 2.2.7 kernels (note: I never installed
the ill-fated 2.2.8) until one day on a machine which at the time was
running 2.2.9. One this particular machine, on that particular day, I
decided to free up some space on the drive. It happens that the drive
contained a few complete copies of the entire file system from another
system's previous installs (as I had upgraded the other system I had backed
up the old system to the ill fated system). When I was looking around for
stuff to remove these system backups proved very attractive. Before
removing one copy I decided to run a du on it. The du ran fine reporting
back about a gig of usage by the about to be removed files. I then "rm -r"
the directory containing those files and directories. When the rm operation
was done I then started looking at the other system backups. I started
another du operation. For whatever reason I decided to cancel it (I think
it was because I mistyped and entered "du --total /" though I do not really
remember now). However I soon realized that ctrl+c wasn't canceling du's
operation. I then switched virtual terminals and tried to kill the du
operation with the kill command. du still would not die. I then decided to
kill the shell under which du was running. the shell died but mot du. I
probably shouldn't have done that because that left me unable to try the
sysrq special key to kill everything running under the current virtual
terminal. By the way another sysrq option that would allow the killing of
any process would be nice. In any case this left du running, still
unkillable, in the background. Normally at this state I would just reboot
the machine, but since at the same time I was doing an extremely slow
windows 98 install over the network in which the CD was in the afflicted
machine's CDROM being accessed remotely by the soon to be windows 98
machine through samba. As I thought du was a harmless command I decided to
leave the machine alone while the install finished. After several more
minutes during which I was fiddling with the machine (doing nothing but
waiting really) I noticed errors start to appear. All of the sudden command
wouldn't run, files started to disappear from their directories; at least
their entries did, further logins became impossible, even ls disappeared.
Kernel error messages also started to appear on the screen when I tried to
run some programs. I'm virtually certain that those messages said "attempt
to read past end of device" or something very much like that. In
desperation, I read the documentation on the special sysrq keys from another
machine and used the key combination to remount all file systems read-only.
As soon as that happened the error messages stopped. Samba itself continued
running and the installed, fortunately, was able to finish. This is
probably because samba and its related libraries were already in memory and
the CDROM was a separate filesystem. After that I brought the system down
using the specially sysrq key combination and restarted the system. The
filesystem had been so corrupted that the system couldn't even do the
initial read-only mount of the root filesystem. Fortunately my ready to run
Slackware on a bootable CD was around, so I was able to boot with that to
attempt to repair the filesystem. I ran e2fsck and e2fsck found various
problems many of which stated something to the effect that an invalid inode
outside of the device's range was being referenced. All these problems were
dealt with and fortunately the system was bootable after that. Despite my
ability to get the system back up file corruption appears to remain as KDE
no longer starts up correctly because of missing and/or invalid files. At
the time I was unsure whether this problem had been related to the
corruption I kept hearing about, beside which I had been under the
impression that the corruption had been a 2.2.8 issue. As the other reply
mentions a kernel error message that I am virtually certain was the same one
I encountered and he was doing an operation that probably mimicked some of
the behavior of the operation(s) which I ran I suspect that a commonality
may be present which can make the corruption reproducible.

Chris Adams' operation

> find /usr/local/src/linux -type f -name '*.[ch]' -print | xargs fgrep foo
>
> That would run okay, but then I'd try a different search and I'd start

My operation

du --total something
rm -r something
du --total / (? or something-else)

Both operations do some similar stuff:

1. Access many file and directory entries as well as directories quickly
and in a short period of time.
2. Access many files quickly and in a short period of time.
3. Access file and directories in a specific manner. Start at a with "a"
directory as the argument.
a. Opens the directory
b. Reads the next available entry, if none goes to e
c. If the entry is a directory it then loops back to a using the
new directory as its argument
d. loops back to b
e. closes the directory
f. Then start accessing files whose entries had just been read,
which probably means those entries are located in the dcache and now are
being accessed from there.
g. Then attempt to start at "a" with a different directory as the
argument.

This pattern appears very, very similar to me which could be an
indication that this pattern can be used to make the filesystem corruption
bug reproducible and any bug that can be found to be reproducible can be
isolated and debugged. If this pattern is the irritant that causes the bug,
then this could explain why many do not seam to be affected by the bug. It
may be that the bug only occurs when the above pattern is followed and done
on a large number of files in a short period of time.

- A. Barbachan



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:52    [W:0.513 / U:0.168 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site