lkml.org 
[lkml]   [1999]   [May]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: Something wrong in 2.2.8?????
I've read the man/info pages for update....Last updated aparently in 1994....After
doing this..and reading the following emails....the only thing I have to ask is:

Is this technique still needed?

"Michael H. Warfield" wrote:

> Pete Clements enscribed thusly:
> > Similiar results/symptoms observed last evening. A UP running NFR software.
> > In blocked state for 12 hours until condition discovered...power switch
> > reboot required.
>
> I've now gone back and done some more investigation based on the
> bdflush thread and some private E-Mails....
>
> I'll say this up front, this is a NASTY one to try and work on.
> This has resulted in file system corruption, the loss of half of my mailbox,
> hours of frustrating work, and required me to manually fsck file systems 5
> times. I've had all kinds of unexpect inode inconsistancy and several
> inodes that were nothing but full of illegal blocks. The reboot time to
> boot up and fsck 8 Gig of file systems is painful enough...
>
> Now... With that noise out of my system...
>
> Having lost half of my mailbox, it became more difficult to reproduce.
> I came up with a torture scheme that seems to do the job pretty well, though.
> Taking one other persons suggestion, this was all done in run level 3 with
> no X Windows running, in case it was covering up an Opps (it wasn't - no
> opps encountered at any time).
>
> On one console I started the following loop:
>
> while : ; do
> dd if=/dev/zero of=dd.1 bs=1048576 count=50
> sync
> rm dd.1
> sync
> done
>
> In another console I had my mailbox open in elm.
>
> In a third console I was repeatedly removing and unpacking a kernel
> source tree.
>
> I would only have to resync my mailbox once or twice to crash it
> (having carefully backed up my mailbox and syncing before starting this).
>
> One wildcard factor in all of this is that sendmail would still
> be delivering mail to my mailbox and I saw several sendmail processes
> up at times during this operation.
>
> The hung processes appeared to be hung in state "D" wait channel
> "wakeup_bdflush". "D" is uninterruptable sleeping. Yeah, I can sure
> confirm that state!
>
> After verifying that I could still blow it up on command, I commented
> out "update" out of my inittab and rebooted. This was based on remarks in
> the bdflush rewrite thread.
>
> I was now unable to reproduce the problem after more that an hour of
> trying. It also felt like the system was MUCH more responsive when I was
> running the torture test.
>
> I then reenabled the update line in inittab and rebooted.
>
> I was able to reproduce the hang in less than five minutes.
>
> The last time the hang occured, I looked real carefully at several
> of the processes (sorry, I forgot to do an strace). The "tar" and "dd"
> processes were both hung in state "D" in "wakeup_bdflush". The "elm"
> process was also in state "D" but in "wait_on_super" instead. When I
> attempted to do a man on ps to lookup something, man hung and was also
> left in state "D" in "wait_on_super". I hung when I tried to su or to
> login on another virtual console. I don't have the states of those processes
> since I was then out of active logins. Hence, I was also left with no way
> to do an strace... Sigh...
>
> While running the torture test this last time, I was also able to
> verify my feeling that the system was much more "jerky" and "slow" with
> update enabled. That leaves me with the feeling that I really can't say
> if update was "causing" this problem or if it was merely changing the
> operating conditions sufficiently to open up the opportunity for the
> problem to arise elsewhere.
>
> IAC... This does seem to be entangled with the bdflush changes and
> the update program. The file system corruption that can result is really
> very serious. If update is "causing" this, there needs to be some way
> to prevent this. Anyone updating to 2.2.8 should get a "danger Will
> Robinson, danger - disable update or your file systems are in danger
> of being corrupted and your system may hang". I'll have to move "update"
> into a script that decides if 2.2.8 or above is running and not to run it,
> or run it with anything less. This is a pain.
>
> If there is something else in there that is merely being agravated
> by update, we really need to shoot it before anyone loose anything more
> serious.
>
> I'm going to continue to run 2.2.8 now with update disabled. We
> shall see... :-(
>
> > > All...
>
> > > Just had some disturbing problems after switching to 2.2.8.
> > >
> > > I was only switched over for a short period of time and was working
> > > on some kernel drivers (Computone serial card drivers - not in the kernel
> > > I was running at the time of the problems). In one window I had my mail
> > > up in elm. In the other window, I went to unpack the kernel sources to
> > > test out a patch. I started a resync on my mailbox about the same time
> > > I started to untar the 2.2.8 for a fresh set of sources. Both got about
> > > half way through and died. At that point, I could not kill any of my
> > > xterm windows although I could run commands like top. Top showed system
> > > load over 16 and climbing! Could not shut down. Sync would hang. Could
> > > switch to a virtual console but Cntrl-Alt-Delete failed to do anything.
> > > Cntr-Alt-Backspace out of X Windows just left a grey screen and then couldn't
> > > switch to another virtual console. Had to reboot from the hardware switch...
> > >
> > > Ok... Rebooted system and came back up on 2.2.8. Fsck reported
> > > all kinds of problems in the kernel source directory I was untaring. That
> > > figured and wasn't real surprising.
> > >
> > > Once the system back was up, I logged back into X Windows and brought
> > > my usual array of xterms back up. I then performed the same actions that
> > > I did before. I deleted the old kernel source tree, remade the directory,
> > > restarted the untar, and then deliberately resynced my mailbox. Bang...
> > > Dead again! Sync still hangs, but I was able to get to another virtual
> > > console. I executed an init 6 and this time it shut down until it was
> > > trying to unmount the local file systems. It died then and there. Could
> > > not even get the "Num Lock" light on the keyboard to toggle. Grrr...
> > >
> > > Hit the reset switch and brought it back up. This time back on
> > > 2.2.7 which had been running perfect for me since it first came out (I
> > > patched up from the E-Mail announcement). Fsck found some serious bad
> > > juju on one of the SCSI drives (I use a mix of IDE and SCSI). Had to
> > > fsck the drive manually. One inode was chock full of illegal blocks.
> > >
> > > Once again got the system back up and got back into X Windows.
> > > Repeated same fatal steps, this time on 2.2.7. No problem. Did it twice.
> > > No problem. I'm about to go back to 2.2.8 and reconfirm, but something
> > > does not seem right here. I know there's been a lot of work down in the
> > > scheduler, but this seems to be a file system handler problem. At least
> > > it seems to be hanging on intense file system activity.
> > >
> > > Lot's of user level programs, such as ps, df, and top can still
> > > run so I can get status on some things. It doesn't die until it seems
> > > to need to write to the disk. (That's just my HIGHLY SUBJECTIVE opinion
> > > and may be way off base.)
> > >
> > > Notes:
> > >
> > > System: 350MHz K6 Asus P5A-B Mother Board
> > > Kernel: 2.2.8
> > > Dstrib: RedHet 5.2
> > > Memory: 128Meg
> > > Swap: 128Meg
> > > Drives: Mix of 2 Gig IDE and SCSI drives.
> > > SCSI: aic7xxx
> > >
> > > Note: /var file system (and subsequently the /var/spool/mail spool
> > > files) is on an IDE drive. My source directories are on one of the SCSI
> > > drives. Both would have been busier than one-armed paper hangers at the
> > > time of the crash.
> > >
> > > Ok... Once is chance, twice is co-incidence, three times is enemy
> > > action. I'm about to switch back over to 2.2.8 and go for three. Any
> > > suggestions from anyone as to what else I might look out for or what I
> > > might do to determine what blew its brains out????
> > >
> > > If it still dies, I'll follow up but then drop back to 2.2.7. I've
> > > got work to get done... Sigh...
> > >
> > > Mike
> > > --
> > > Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com
> > > (The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/
> > > NIC whois: MHW9 | An optimist believes we live in the best of all
> > > PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!
>
> > --
> > Pete Clements
> > clem@clem.digital.net
>
> Mike
> --
> Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com
> (The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/
> NIC whois: MHW9 | An optimist believes we live in the best of all
> PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.rutgers.edu
> Please read the FAQ at http://www.tux.org/lkml/

--
Edward Muller
Waste Not Computers & Supplies
94 Washington Ave.
Dumont, NJ 07628
(201) 384-4444 x204
(201) 384-4024 (fax)
(201) 906-4207 (cel)
edwardm@wastenotcomputers.com
edwardam@home.com


begin:vcard
n:Muller;Edward
tel;cell:(201) 906-4207
tel;fax:(201) 384-4024
tel;home:(973) XXX-XXXX
tel;work:(201) 384-4444 x204
x-mozilla-html:TRUE
url:http://www.wastenotcomputers.com/users/edwardm
org:Waste Not Computers & Supplies;PC/Networking
adr:;;94 Washington Ave..;Dumont;NJ;07628;USA
version:2.1
email;internet:edwardm@wastenotcomputers.com
title:Senior Consultant
note;quoted-printable:Alt:=0D=0Aedwardam@home.com
x-mozilla-cpt:;-8864
fn:Edward Muller
end:vcard
\
 
 \ /
  Last update: 2005-03-22 13:51    [W:0.044 / U:0.160 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site