Messages in this thread | | | From | "Anthony Barbachan" <> | Subject | Re: File Corruption Bug.. continued | Date | Thu, 24 Jun 1999 02:49:42 -0400 |
| |
> I've now been through most of the 2.2.7->2.2.9 diff set discarding stuff > that seems not to be a viable candiate. > > The remaining suspects are: > o Quota - which has big 2.2.7->2.2.9 changes. > o The small scsi changes (dubious) > o A small mm change > > There are other candiates - notably > TCP changes > interrupt changes > IRDA > NFS > > The tcp changes ought to have shown up more than this does. The interrupt > changes dont really seem to explain cross platform stuff. And I doubt most > people running these tests were running irda. NFS is a possibility. > > So - are most people seeing the problems running quotas ? And would someone > with their brain firmly wrapped around the page cache/vfs verify this change > that was made is absolutely safe >
I pasted Chris Adams' reply as well as it is related to my message.
Chris Adams <cadams@ro.com> wrote in message news:7kr74r$h48$1@sh1.ro.com... > Once upon a time, Alan Cox <alan@lxorguk.ukuu.org.uk> said: > >I've now been through most of the 2.2.7->2.2.9 diff set discarding stuff > >that seems not to be a viable candiate. > > > >The remaining suspects are: > >that was made is absolutely safe > > Well, I don't run quotas on the system I had the FS corruption with (it > is my desktop box at work). I do have SCSI (aic7xxx, which had a driver > update in 2.2.10 that now enables tagged queueing by default IIRC), but > I don't have IRDA or NFS (client or server). I did have NFS compiled in > 2.2.9 when I had trouble, but not in 2.2.10 when I had trouble again. > > With 2.2.5 on it, I ran a script (taken from the pages about the AMD K6 > with Linux) that basically does "make -j4" of the kernel and then > compares the results from run to run. I use this as a good burnin test > of systems these days. I ran this burnin on 2.2.5 for about 18 hours > and it didn't have any trouble. > > What I will try to do (probably will be a day or two) is setup a scratch > partition on my system and boot 2.2.10. I'll mount all the other > partitions read-only, and then run the burnin script with the kernel > source on the scratch partition. If it fails (and I think it will after > a while), then at least I'll have a way to reproduce the problem and > test changes. > > The first time I had corruption (under 2.2.9) I was doing something > like: > > find /usr/local/src/linux -type f -name '*.[ch]' -print | xargs fgrep foo > > That would run okay, but then I'd try a different search and I'd start > getting the "attempt to read past end of device" and readdir errors. > Then, when I stopped the find and did "ls /usr/local/src/linux", I got > an empty directory. > > IIRC I was doing something similar when the corruption happened again. > > I thought it was a hardware problem at first, because I am using an > older drive. I've never seen corruption like this in all the time I > have been using Linux (back to 0.99 days), although I do avoid the known > bad development kernels always. But then I realized that I didn't get > ANY SCSI errors and that when I run 2.2.5 I can't make the same thing > happen no matter how hard I try. > -- > Chris Adams <cadams@ro.com> - System Administrator
Hi,
I have a strong suspicion that the above may be an important clue. I hadn't had any corruption with post 2.2.7 kernels (note: I never installed the ill-fated 2.2.8) until one day on a machine which at the time was running 2.2.9. One this particular machine, on that particular day, I decided to free up some space on the drive. It happens that the drive contained a few complete copies of the entire file system from another system's previous installs (as I had upgraded the other system I had backed up the old system to the ill fated system). When I was looking around for stuff to remove these system backups proved very attractive. Before removing one copy I decided to run a du on it. The du ran fine reporting back about a gig of usage by the about to be removed files. I then "rm -r" the directory containing those files and directories. When the rm operation was done I then started looking at the other system backups. I started another du operation. For whatever reason I decided to cancel it (I think it was because I mistyped and entered "du --total /" though I do not really remember now). However I soon realized that ctrl+c wasn't canceling du's operation. I then switched virtual terminals and tried to kill the du operation with the kill command. du still would not die. I then decided to kill the shell under which du was running. the shell died but mot du. I probably shouldn't have done that because that left me unable to try the sysrq special key to kill everything running under the current virtual terminal. By the way another sysrq option that would allow the killing of any process would be nice. In any case this left du running, still unkillable, in the background. Normally at this state I would just reboot the machine, but since at the same time I was doing an extremely slow windows 98 install over the network in which the CD was in the afflicted machine's CDROM being accessed remotely by the soon to be windows 98 machine through samba. As I thought du was a harmless command I decided to leave the machine alone while the install finished. After several more minutes during which I was fiddling with the machine (doing nothing but waiting really) I noticed errors start to appear. All of the sudden command wouldn't run, files started to disappear from their directories; at least their entries did, further logins became impossible, even ls disappeared. Kernel error messages also started to appear on the screen when I tried to run some programs. I'm virtually certain that those messages said "attempt to read past end of device" or something very much like that. In desperation, I read the documentation on the special sysrq keys from another machine and used the key combination to remount all file systems read-only. As soon as that happened the error messages stopped. Samba itself continued running and the installed, fortunately, was able to finish. This is probably because samba and its related libraries were already in memory and the CDROM was a separate filesystem. After that I brought the system down using the specially sysrq key combination and restarted the system. The filesystem had been so corrupted that the system couldn't even do the initial read-only mount of the root filesystem. Fortunately my ready to run Slackware on a bootable CD was around, so I was able to boot with that to attempt to repair the filesystem. I ran e2fsck and e2fsck found various problems many of which stated something to the effect that an invalid inode outside of the device's range was being referenced. All these problems were dealt with and fortunately the system was bootable after that. Despite my ability to get the system back up file corruption appears to remain as KDE no longer starts up correctly because of missing and/or invalid files. At the time I was unsure whether this problem had been related to the corruption I kept hearing about, beside which I had been under the impression that the corruption had been a 2.2.8 issue. As the other reply mentions a kernel error message that I am virtually certain was the same one I encountered and he was doing an operation that probably mimicked some of the behavior of the operation(s) which I ran I suspect that a commonality may be present which can make the corruption reproducible.
Chris Adams' operation
> find /usr/local/src/linux -type f -name '*.[ch]' -print | xargs fgrep foo > > That would run okay, but then I'd try a different search and I'd start
My operation
du --total something rm -r something du --total / (? or something-else)
Both operations do some similar stuff:
1. Access many file and directory entries as well as directories quickly and in a short period of time. 2. Access many files quickly and in a short period of time. 3. Access file and directories in a specific manner. Start at a with "a" directory as the argument. a. Opens the directory b. Reads the next available entry, if none goes to e c. If the entry is a directory it then loops back to a using the new directory as its argument d. loops back to b e. closes the directory f. Then start accessing files whose entries had just been read, which probably means those entries are located in the dcache and now are being accessed from there. g. Then attempt to start at "a" with a different directory as the argument.
This pattern appears very, very similar to me which could be an indication that this pattern can be used to make the filesystem corruption bug reproducible and any bug that can be found to be reproducible can be isolated and debugged. If this pattern is the irritant that causes the bug, then this could explain why many do not seam to be affected by the bug. It may be that the bug only occurs when the above pattern is followed and done on a large number of files in a short period of time.
- A. Barbachan
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |