Messages in this thread | | | From | (Patrick J. LoPresti) | Subject | SMP kernel lockup, 2.2.14 and 2.2.15pre15 | Date | 23 Mar 2000 11:16:53 -0500 |
| |
We are able to consistently wedge 2.2.14 and 2.2.15pre15 on SMP systems using only user-mode processes with modest resource requirements.
Unfortunately, this lockup involves a commercial product (Perforce), so we may need to debug it ourselves. This message is a request for help.
Here is the deal:
We are converting an internal source code repository from CVS to Perforce. We have a conversion script which takes about 9 hours to run. All processes involved (script, client, and server) are running as a normal user, not as root.
About 5-8 hours into the conversion, we usually (about 80% of the time) see the machine hang. It does not respond to pings nor to any activity on the keyboard (e.g., ctrl-alt-del). The magic sysrq key works, however; see below.
We have now reproduced this on three different SMP systems, with different hardware configurations, each while running 2.2.14 compiled by egcs-1.1.2 and while running 2.2.15pre15 compiled by gcc-2.95.2. (I did not realize until recently that someone installed 2.95.2 as our default compiler; sorry for the inconsistency.) So this is pretty clearly not a hardware problem.
We have magic sysrq support compiled into 2.2.15pre15. While the machine was locked, we pressed "alt-sysrq-p" repeatedly and wrote down the various EIP values. Mapping those by hand to kernel addresses (using "nm vmlinux"), we determined that we are spending some portion of our time in tcp_delack_timer, tcp_send_delayed_ack, add_timer, and timer_bh. (This appears to be a loop waiting for a socket to become unlocked.) Some of the EIP values had no obvious entry in the symbol table; I am guessing they are in dynamically loaded modules.
Alt-sysrq-s and Alt-sysrq-u each print the message saying what they are trying to do, but no further output. We see no messages about specific disks being synced; no "OK" for the unmount; nothing. So the kernel does appear to be locked pretty solid.
Finally, we are running the same experiment on a uniprocessor box and on 2.2.13 SMP, neither of which has deadlocked yet. We have only done one or two such trials so far, however. They are ongoing.
Here are some of my questions:
1) Does the magic sysrq always go to the same CPU? I would like to know what both processors are doing when the system is locked. Ideally I would like to see a backtrace (similar to an OOPS). I do not see any way to do that with the existing magic sysrq implementation; perhaps somebody has patches or can give me suggestions for rolling my own?
2) Should I compile 2.2.15pre15 with egcs 1.1.2 instead? The same deadlock happens on 2.2.14 compiled with egcs, so I do not think this will make any difference.
3) Should I recompile without modules so I can map more EIP values to kernel symbols? Or am I totally barking up the wrong tree here?
4) Is the loop through tcp_send_delayed_ack etc. likely to be relevant? One thing this import process does which is somewhat unusual is to make many client-server connections *to the local machine* and push lots of data through them. Maybe something in loopback networking is broken?
In general, what should I try next?
Thanks!
- Pat
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |