lkml.org 
[lkml]   [2000]   [Mar]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectSMP kernel lockup, 2.2.14 and 2.2.15pre15
Date
We are able to consistently wedge 2.2.14 and 2.2.15pre15 on SMP
systems using only user-mode processes with modest resource
requirements.

Unfortunately, this lockup involves a commercial product (Perforce),
so we may need to debug it ourselves. This message is a request for
help.

Here is the deal:

We are converting an internal source code repository from CVS to
Perforce. We have a conversion script which takes about 9 hours to
run. All processes involved (script, client, and server) are running
as a normal user, not as root.

About 5-8 hours into the conversion, we usually (about 80% of the
time) see the machine hang. It does not respond to pings nor to any
activity on the keyboard (e.g., ctrl-alt-del). The magic sysrq key
works, however; see below.

We have now reproduced this on three different SMP systems, with
different hardware configurations, each while running 2.2.14 compiled
by egcs-1.1.2 and while running 2.2.15pre15 compiled by gcc-2.95.2.
(I did not realize until recently that someone installed 2.95.2 as our
default compiler; sorry for the inconsistency.) So this is pretty
clearly not a hardware problem.

We have magic sysrq support compiled into 2.2.15pre15. While the
machine was locked, we pressed "alt-sysrq-p" repeatedly and wrote down
the various EIP values. Mapping those by hand to kernel addresses
(using "nm vmlinux"), we determined that we are spending some portion
of our time in tcp_delack_timer, tcp_send_delayed_ack, add_timer, and
timer_bh. (This appears to be a loop waiting for a socket to become
unlocked.) Some of the EIP values had no obvious entry in the symbol
table; I am guessing they are in dynamically loaded modules.

Alt-sysrq-s and Alt-sysrq-u each print the message saying what they
are trying to do, but no further output. We see no messages about
specific disks being synced; no "OK" for the unmount; nothing. So the
kernel does appear to be locked pretty solid.

Finally, we are running the same experiment on a uniprocessor box and
on 2.2.13 SMP, neither of which has deadlocked yet. We have only done
one or two such trials so far, however. They are ongoing.

Here are some of my questions:

1) Does the magic sysrq always go to the same CPU? I would like to
know what both processors are doing when the system is locked.
Ideally I would like to see a backtrace (similar to an OOPS). I
do not see any way to do that with the existing magic sysrq
implementation; perhaps somebody has patches or can give me
suggestions for rolling my own?

2) Should I compile 2.2.15pre15 with egcs 1.1.2 instead? The same
deadlock happens on 2.2.14 compiled with egcs, so I do not think
this will make any difference.

3) Should I recompile without modules so I can map more EIP values
to kernel symbols? Or am I totally barking up the wrong tree
here?

4) Is the loop through tcp_send_delayed_ack etc. likely to be
relevant? One thing this import process does which is somewhat
unusual is to make many client-server connections *to the local
machine* and push lots of data through them. Maybe something in
loopback networking is broken?

In general, what should I try next?

Thanks!

- Pat

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:57    [W:0.026 / U:0.324 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site