Messages in this thread | | | Date | Thu, 13 May 1999 21:37:54 -0400 | From | Edward Muller <> | Subject | Re: Something wrong in 2.2.8????? |
| |
I've read the man/info pages for update....Last updated aparently in 1994....After doing this..and reading the following emails....the only thing I have to ask is:
Is this technique still needed?
"Michael H. Warfield" wrote:
> Pete Clements enscribed thusly: > > Similiar results/symptoms observed last evening. A UP running NFR software. > > In blocked state for 12 hours until condition discovered...power switch > > reboot required. > > I've now gone back and done some more investigation based on the > bdflush thread and some private E-Mails.... > > I'll say this up front, this is a NASTY one to try and work on. > This has resulted in file system corruption, the loss of half of my mailbox, > hours of frustrating work, and required me to manually fsck file systems 5 > times. I've had all kinds of unexpect inode inconsistancy and several > inodes that were nothing but full of illegal blocks. The reboot time to > boot up and fsck 8 Gig of file systems is painful enough... > > Now... With that noise out of my system... > > Having lost half of my mailbox, it became more difficult to reproduce. > I came up with a torture scheme that seems to do the job pretty well, though. > Taking one other persons suggestion, this was all done in run level 3 with > no X Windows running, in case it was covering up an Opps (it wasn't - no > opps encountered at any time). > > On one console I started the following loop: > > while : ; do > dd if=/dev/zero of=dd.1 bs=1048576 count=50 > sync > rm dd.1 > sync > done > > In another console I had my mailbox open in elm. > > In a third console I was repeatedly removing and unpacking a kernel > source tree. > > I would only have to resync my mailbox once or twice to crash it > (having carefully backed up my mailbox and syncing before starting this). > > One wildcard factor in all of this is that sendmail would still > be delivering mail to my mailbox and I saw several sendmail processes > up at times during this operation. > > The hung processes appeared to be hung in state "D" wait channel > "wakeup_bdflush". "D" is uninterruptable sleeping. Yeah, I can sure > confirm that state! > > After verifying that I could still blow it up on command, I commented > out "update" out of my inittab and rebooted. This was based on remarks in > the bdflush rewrite thread. > > I was now unable to reproduce the problem after more that an hour of > trying. It also felt like the system was MUCH more responsive when I was > running the torture test. > > I then reenabled the update line in inittab and rebooted. > > I was able to reproduce the hang in less than five minutes. > > The last time the hang occured, I looked real carefully at several > of the processes (sorry, I forgot to do an strace). The "tar" and "dd" > processes were both hung in state "D" in "wakeup_bdflush". The "elm" > process was also in state "D" but in "wait_on_super" instead. When I > attempted to do a man on ps to lookup something, man hung and was also > left in state "D" in "wait_on_super". I hung when I tried to su or to > login on another virtual console. I don't have the states of those processes > since I was then out of active logins. Hence, I was also left with no way > to do an strace... Sigh... > > While running the torture test this last time, I was also able to > verify my feeling that the system was much more "jerky" and "slow" with > update enabled. That leaves me with the feeling that I really can't say > if update was "causing" this problem or if it was merely changing the > operating conditions sufficiently to open up the opportunity for the > problem to arise elsewhere. > > IAC... This does seem to be entangled with the bdflush changes and > the update program. The file system corruption that can result is really > very serious. If update is "causing" this, there needs to be some way > to prevent this. Anyone updating to 2.2.8 should get a "danger Will > Robinson, danger - disable update or your file systems are in danger > of being corrupted and your system may hang". I'll have to move "update" > into a script that decides if 2.2.8 or above is running and not to run it, > or run it with anything less. This is a pain. > > If there is something else in there that is merely being agravated > by update, we really need to shoot it before anyone loose anything more > serious. > > I'm going to continue to run 2.2.8 now with update disabled. We > shall see... :-( > > > > All... > > > > Just had some disturbing problems after switching to 2.2.8. > > > > > > I was only switched over for a short period of time and was working > > > on some kernel drivers (Computone serial card drivers - not in the kernel > > > I was running at the time of the problems). In one window I had my mail > > > up in elm. In the other window, I went to unpack the kernel sources to > > > test out a patch. I started a resync on my mailbox about the same time > > > I started to untar the 2.2.8 for a fresh set of sources. Both got about > > > half way through and died. At that point, I could not kill any of my > > > xterm windows although I could run commands like top. Top showed system > > > load over 16 and climbing! Could not shut down. Sync would hang. Could > > > switch to a virtual console but Cntrl-Alt-Delete failed to do anything. > > > Cntr-Alt-Backspace out of X Windows just left a grey screen and then couldn't > > > switch to another virtual console. Had to reboot from the hardware switch... > > > > > > Ok... Rebooted system and came back up on 2.2.8. Fsck reported > > > all kinds of problems in the kernel source directory I was untaring. That > > > figured and wasn't real surprising. > > > > > > Once the system back was up, I logged back into X Windows and brought > > > my usual array of xterms back up. I then performed the same actions that > > > I did before. I deleted the old kernel source tree, remade the directory, > > > restarted the untar, and then deliberately resynced my mailbox. Bang... > > > Dead again! Sync still hangs, but I was able to get to another virtual > > > console. I executed an init 6 and this time it shut down until it was > > > trying to unmount the local file systems. It died then and there. Could > > > not even get the "Num Lock" light on the keyboard to toggle. Grrr... > > > > > > Hit the reset switch and brought it back up. This time back on > > > 2.2.7 which had been running perfect for me since it first came out (I > > > patched up from the E-Mail announcement). Fsck found some serious bad > > > juju on one of the SCSI drives (I use a mix of IDE and SCSI). Had to > > > fsck the drive manually. One inode was chock full of illegal blocks. > > > > > > Once again got the system back up and got back into X Windows. > > > Repeated same fatal steps, this time on 2.2.7. No problem. Did it twice. > > > No problem. I'm about to go back to 2.2.8 and reconfirm, but something > > > does not seem right here. I know there's been a lot of work down in the > > > scheduler, but this seems to be a file system handler problem. At least > > > it seems to be hanging on intense file system activity. > > > > > > Lot's of user level programs, such as ps, df, and top can still > > > run so I can get status on some things. It doesn't die until it seems > > > to need to write to the disk. (That's just my HIGHLY SUBJECTIVE opinion > > > and may be way off base.) > > > > > > Notes: > > > > > > System: 350MHz K6 Asus P5A-B Mother Board > > > Kernel: 2.2.8 > > > Dstrib: RedHet 5.2 > > > Memory: 128Meg > > > Swap: 128Meg > > > Drives: Mix of 2 Gig IDE and SCSI drives. > > > SCSI: aic7xxx > > > > > > Note: /var file system (and subsequently the /var/spool/mail spool > > > files) is on an IDE drive. My source directories are on one of the SCSI > > > drives. Both would have been busier than one-armed paper hangers at the > > > time of the crash. > > > > > > Ok... Once is chance, twice is co-incidence, three times is enemy > > > action. I'm about to switch back over to 2.2.8 and go for three. Any > > > suggestions from anyone as to what else I might look out for or what I > > > might do to determine what blew its brains out???? > > > > > > If it still dies, I'll follow up but then drop back to 2.2.7. I've > > > got work to get done... Sigh... > > > > > > Mike > > > -- > > > Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com > > > (The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/ > > > NIC whois: MHW9 | An optimist believes we live in the best of all > > > PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it! > > > -- > > Pete Clements > > clem@clem.digital.net > > Mike > -- > Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com > (The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/ > NIC whois: MHW9 | An optimist believes we live in the best of all > PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it! > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.rutgers.edu > Please read the FAQ at http://www.tux.org/lkml/
-- Edward Muller Waste Not Computers & Supplies 94 Washington Ave. Dumont, NJ 07628 (201) 384-4444 x204 (201) 384-4024 (fax) (201) 906-4207 (cel) edwardm@wastenotcomputers.com edwardam@home.com
begin:vcard n:Muller;Edward tel;cell:(201) 906-4207 tel;fax:(201) 384-4024 tel;home:(973) XXX-XXXX tel;work:(201) 384-4444 x204 x-mozilla-html:TRUE url:http://www.wastenotcomputers.com/users/edwardm org:Waste Not Computers & Supplies;PC/Networking adr:;;94 Washington Ave..;Dumont;NJ;07628;USA version:2.1 email;internet:edwardm@wastenotcomputers.com title:Senior Consultant note;quoted-printable:Alt:=0D=0Aedwardam@home.com x-mozilla-cpt:;-8864 fn:Edward Muller end:vcard
| |