lkml.org 
[lkml]   [1999]   [May]   [14]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectRe: Something wrong in 2.2.8?????
Date
Pete Clements enscribed thusly:
> Similiar results/symptoms observed last evening. A UP running NFR software.
> In blocked state for 12 hours until condition discovered...power switch
> reboot required.

I've now gone back and done some more investigation based on the
bdflush thread and some private E-Mails....
I'll say this up front, this is a NASTY one to try and work on.
This has resulted in file system corruption, the loss of half of my mailbox,
hours of frustrating work, and required me to manually fsck file systems 5
times. I've had all kinds of unexpect inode inconsistancy and several
inodes that were nothing but full of illegal blocks. The reboot time to
boot up and fsck 8 Gig of file systems is painful enough...

Now... With that noise out of my system...

Having lost half of my mailbox, it became more difficult to reproduce.
I came up with a torture scheme that seems to do the job pretty well, though.
Taking one other persons suggestion, this was all done in run level 3 with
no X Windows running, in case it was covering up an Opps (it wasn't - no
opps encountered at any time).

On one console I started the following loop:

while : ; do
dd if=/dev/zero of=dd.1 bs=1048576 count=50
sync
rm dd.1
sync
done

In another console I had my mailbox open in elm.

In a third console I was repeatedly removing and unpacking a kernel
source tree.

I would only have to resync my mailbox once or twice to crash it
(having carefully backed up my mailbox and syncing before starting this).

One wildcard factor in all of this is that sendmail would still
be delivering mail to my mailbox and I saw several sendmail processes
up at times during this operation.

The hung processes appeared to be hung in state "D" wait channel
"wakeup_bdflush". "D" is uninterruptable sleeping. Yeah, I can sure
confirm that state!

After verifying that I could still blow it up on command, I commented
out "update" out of my inittab and rebooted. This was based on remarks in
the bdflush rewrite thread.

I was now unable to reproduce the problem after more that an hour of
trying. It also felt like the system was MUCH more responsive when I was
running the torture test.

I then reenabled the update line in inittab and rebooted.

I was able to reproduce the hang in less than five minutes.

The last time the hang occured, I looked real carefully at several
of the processes (sorry, I forgot to do an strace). The "tar" and "dd"
processes were both hung in state "D" in "wakeup_bdflush". The "elm"
process was also in state "D" but in "wait_on_super" instead. When I
attempted to do a man on ps to lookup something, man hung and was also
left in state "D" in "wait_on_super". I hung when I tried to su or to
login on another virtual console. I don't have the states of those processes
since I was then out of active logins. Hence, I was also left with no way
to do an strace... Sigh...

While running the torture test this last time, I was also able to
verify my feeling that the system was much more "jerky" and "slow" with
update enabled. That leaves me with the feeling that I really can't say
if update was "causing" this problem or if it was merely changing the
operating conditions sufficiently to open up the opportunity for the
problem to arise elsewhere.

IAC... This does seem to be entangled with the bdflush changes and
the update program. The file system corruption that can result is really
very serious. If update is "causing" this, there needs to be some way
to prevent this. Anyone updating to 2.2.8 should get a "danger Will
Robinson, danger - disable update or your file systems are in danger
of being corrupted and your system may hang". I'll have to move "update"
into a script that decides if 2.2.8 or above is running and not to run it,
or run it with anything less. This is a pain.

If there is something else in there that is merely being agravated
by update, we really need to shoot it before anyone loose anything more
serious.

I'm going to continue to run 2.2.8 now with update disabled. We
shall see... :-(

> > All...

> > Just had some disturbing problems after switching to 2.2.8.
> >
> > I was only switched over for a short period of time and was working
> > on some kernel drivers (Computone serial card drivers - not in the kernel
> > I was running at the time of the problems). In one window I had my mail
> > up in elm. In the other window, I went to unpack the kernel sources to
> > test out a patch. I started a resync on my mailbox about the same time
> > I started to untar the 2.2.8 for a fresh set of sources. Both got about
> > half way through and died. At that point, I could not kill any of my
> > xterm windows although I could run commands like top. Top showed system
> > load over 16 and climbing! Could not shut down. Sync would hang. Could
> > switch to a virtual console but Cntrl-Alt-Delete failed to do anything.
> > Cntr-Alt-Backspace out of X Windows just left a grey screen and then couldn't
> > switch to another virtual console. Had to reboot from the hardware switch...
> >
> > Ok... Rebooted system and came back up on 2.2.8. Fsck reported
> > all kinds of problems in the kernel source directory I was untaring. That
> > figured and wasn't real surprising.
> >
> > Once the system back was up, I logged back into X Windows and brought
> > my usual array of xterms back up. I then performed the same actions that
> > I did before. I deleted the old kernel source tree, remade the directory,
> > restarted the untar, and then deliberately resynced my mailbox. Bang...
> > Dead again! Sync still hangs, but I was able to get to another virtual
> > console. I executed an init 6 and this time it shut down until it was
> > trying to unmount the local file systems. It died then and there. Could
> > not even get the "Num Lock" light on the keyboard to toggle. Grrr...
> >
> > Hit the reset switch and brought it back up. This time back on
> > 2.2.7 which had been running perfect for me since it first came out (I
> > patched up from the E-Mail announcement). Fsck found some serious bad
> > juju on one of the SCSI drives (I use a mix of IDE and SCSI). Had to
> > fsck the drive manually. One inode was chock full of illegal blocks.
> >
> > Once again got the system back up and got back into X Windows.
> > Repeated same fatal steps, this time on 2.2.7. No problem. Did it twice.
> > No problem. I'm about to go back to 2.2.8 and reconfirm, but something
> > does not seem right here. I know there's been a lot of work down in the
> > scheduler, but this seems to be a file system handler problem. At least
> > it seems to be hanging on intense file system activity.
> >
> > Lot's of user level programs, such as ps, df, and top can still
> > run so I can get status on some things. It doesn't die until it seems
> > to need to write to the disk. (That's just my HIGHLY SUBJECTIVE opinion
> > and may be way off base.)
> >
> > Notes:
> >
> > System: 350MHz K6 Asus P5A-B Mother Board
> > Kernel: 2.2.8
> > Dstrib: RedHet 5.2
> > Memory: 128Meg
> > Swap: 128Meg
> > Drives: Mix of 2 Gig IDE and SCSI drives.
> > SCSI: aic7xxx
> >
> > Note: /var file system (and subsequently the /var/spool/mail spool
> > files) is on an IDE drive. My source directories are on one of the SCSI
> > drives. Both would have been busier than one-armed paper hangers at the
> > time of the crash.
> >
> > Ok... Once is chance, twice is co-incidence, three times is enemy
> > action. I'm about to switch back over to 2.2.8 and go for three. Any
> > suggestions from anyone as to what else I might look out for or what I
> > might do to determine what blew its brains out????
> >
> > If it still dies, I'll follow up but then drop back to 2.2.7. I've
> > got work to get done... Sigh...
> >
> > Mike
> > --
> > Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com
> > (The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/
> > NIC whois: MHW9 | An optimist believes we live in the best of all
> > PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!

> --
> Pete Clements
> clem@clem.digital.net

Mike
--
Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com
(The Mad Wizard) | (770) 925-8248 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:51    [from the cache]
©2003-2014 Jasper Spaans. Advertise on this site