Messages in this thread | | | From | "Michael H. Warfield" <> | Subject | Re: Something wrong in 2.2.8????? | Date | Thu, 13 May 1999 11:14:58 -0400 (EDT) |
| |
Pete Clements enscribed thusly: > Similiar results/symptoms observed last evening. A UP running NFR software. > In blocked state for 12 hours until condition discovered...power switch > reboot required.
I've now gone back and done some more investigation based on the bdflush thread and some private E-Mails....
I'll say this up front, this is a NASTY one to try and work on. This has resulted in file system corruption, the loss of half of my mailbox, hours of frustrating work, and required me to manually fsck file systems 5 times. I've had all kinds of unexpect inode inconsistancy and several inodes that were nothing but full of illegal blocks. The reboot time to boot up and fsck 8 Gig of file systems is painful enough...
Now... With that noise out of my system...
Having lost half of my mailbox, it became more difficult to reproduce. I came up with a torture scheme that seems to do the job pretty well, though. Taking one other persons suggestion, this was all done in run level 3 with no X Windows running, in case it was covering up an Opps (it wasn't - no opps encountered at any time).
On one console I started the following loop:
while : ; do dd if=/dev/zero of=dd.1 bs=1048576 count=50 sync rm dd.1 sync done
In another console I had my mailbox open in elm.
In a third console I was repeatedly removing and unpacking a kernel source tree.
I would only have to resync my mailbox once or twice to crash it (having carefully backed up my mailbox and syncing before starting this).
One wildcard factor in all of this is that sendmail would still be delivering mail to my mailbox and I saw several sendmail processes up at times during this operation.
The hung processes appeared to be hung in state "D" wait channel "wakeup_bdflush". "D" is uninterruptable sleeping. Yeah, I can sure confirm that state!
After verifying that I could still blow it up on command, I commented out "update" out of my inittab and rebooted. This was based on remarks in the bdflush rewrite thread.
I was now unable to reproduce the problem after more that an hour of trying. It also felt like the system was MUCH more responsive when I was running the torture test.
I then reenabled the update line in inittab and rebooted.
I was able to reproduce the hang in less than five minutes.
The last time the hang occured, I looked real carefully at several of the processes (sorry, I forgot to do an strace). The "tar" and "dd" processes were both hung in state "D" in "wakeup_bdflush". The "elm" process was also in state "D" but in "wait_on_super" instead. When I attempted to do a man on ps to lookup something, man hung and was also left in state "D" in "wait_on_super". I hung when I tried to su or to login on another virtual console. I don't have the states of those processes since I was then out of active logins. Hence, I was also left with no way to do an strace... Sigh...
While running the torture test this last time, I was also able to verify my feeling that the system was much more "jerky" and "slow" with update enabled. That leaves me with the feeling that I really can't say if update was "causing" this problem or if it was merely changing the operating conditions sufficiently to open up the opportunity for the problem to arise elsewhere.
IAC... This does seem to be entangled with the bdflush changes and the update program. The file system corruption that can result is really very serious. If update is "causing" this, there needs to be some way to prevent this. Anyone updating to 2.2.8 should get a "danger Will Robinson, danger - disable update or your file systems are in danger of being corrupted and your system may hang". I'll have to move "update" into a script that decides if 2.2.8 or above is running and not to run it, or run it with anything less. This is a pain.
If there is something else in there that is merely being agravated by update, we really need to shoot it before anyone loose anything more serious.
I'm going to continue to run 2.2.8 now with update disabled. We shall see... :-(
> > All...
> > Just had some disturbing problems after switching to 2.2.8. > > > > I was only switched over for a short period of time and was working > > on some kernel drivers (Computone serial card drivers - not in the kernel > > I was running at the time of the problems). In one window I had my mail > > up in elm. In the other window, I went to unpack the kernel sources to > > test out a patch. I started a resync on my mailbox about the same time > > I started to untar the 2.2.8 for a fresh set of sources. Both got about > > half way through and died. At that point, I could not kill any of my > > xterm windows although I could run commands like top. Top showed system > > load over 16 and climbing! Could not shut down. Sync would hang. Could > > switch to a virtual console but Cntrl-Alt-Delete failed to do anything. > > Cntr-Alt-Backspace out of X Windows just left a grey screen and then couldn't > > switch to another virtual console. Had to reboot from the hardware switch... > > > > Ok... Rebooted system and came back up on 2.2.8. Fsck reported > > all kinds of problems in the kernel source directory I was untaring. That > > figured and wasn't real surprising. > > > > Once the system back was up, I logged back into X Windows and brought > > my usual array of xterms back up. I then performed the same actions that > > I did before. I deleted the old kernel source tree, remade the directory, > > restarted the untar, and then deliberately resynced my mailbox. Bang... > > Dead again! Sync still hangs, but I was able to get to another virtual > > console. I executed an init 6 and this time it shut down until it was > > trying to unmount the local file systems. It died then and there. Could > > not even get the "Num Lock" light on the keyboard to toggle. Grrr... > > > > Hit the reset switch and brought it back up. This time back on > > 2.2.7 which had been running perfect for me since it first came out (I > > patched up from the E-Mail announcement). Fsck found some serious bad > > juju on one of the SCSI drives (I use a mix of IDE and SCSI). Had to > > fsck the drive manually. One inode was chock full of illegal blocks. > > > > Once again got the system back up and got back into X Windows. > > Repeated same fatal steps, this time on 2.2.7. No problem. Did it twice. > > No problem. I'm about to go back to 2.2.8 and reconfirm, but something > > does not seem right here. I know there's been a lot of work down in the > > scheduler, but this seems to be a file system handler problem. At least > > it seems to be hanging on intense file system activity. > > > > Lot's of user level programs, such as ps, df, and top can still > > run so I can get status on some things. It doesn't die until it seems > > to need to write to the disk. (That's just my HIGHLY SUBJECTIVE opinion > > and may be way off base.) > > > > Notes: > > > > System: 350MHz K6 Asus P5A-B Mother Board > > Kernel: 2.2.8 > > Dstrib: RedHet 5.2 > > Memory: 128Meg > > Swap: 128Meg > > Drives: Mix of 2 Gig IDE and SCSI drives. > > SCSI: aic7xxx > > > > Note: /var file system (and subsequently the /var/spool/mail spool > > files) is on an IDE drive. My source directories are on one of the SCSI > > drives. Both would have been busier than one-armed paper hangers at the > > time of the crash. > > > > Ok... Once is chance, twice is co-incidence, three times is enemy > > action. I'm about to switch back over to 2.2.8 and go for three. Any > > suggestions from anyone as to what else I might look out for or what I > > might do to determine what blew its brains out???? > > > > If it still dies, I'll follow up but then drop back to 2.2.7. I've > > got work to get done... Sigh... > > > > Mike > > -- > > Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com > > (The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/ > > NIC whois: MHW9 | An optimist believes we live in the best of all > > PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!
> -- > Pete Clements > clem@clem.digital.net
Mike -- Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com (The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |