Messages in this thread | | | From | root <> | Subject | Oops in kernel 2.2.xx (tcp_recvmsg) | Date | Fri, 9 Jul 1999 17:10:48 +0100 (BST) |
| |
1. PROBLEM: Kernel 2.2.xx - Oops in tcp_recvmsg()
=========================================================================== 2. Full description
Normally my system hangs with no Oops message and no information at all on screen. I have once recieved an "oops" message, the detail of which is included below. The sequence of events, according to the syslog file, is always the same - A person logs in from a Windows NT box via ISDN and picks up mail using MS Outlook (yechh!) (using ipop3d service). Perhaps by coincidence, although a number of people login remotely to the system, the crashes have only occurred when one particular user logs in. The final message, regarding "kfree_skb" does not always make it into the syslog file although the preceding sequence of events is always the same.
When the "Oops" occurred the active process was "ipop3d".
The "Oops" was received using kernel 2.2.7 #SMP. However, the problem has occurred with 2.2.5 and 2.2.10.
I have tried to include as much information as I think is relevant. Thanks for your assistance in this matter. Please e-mail (or call) if further information is required.
============================== Peter Colman email: pete@mssint.demon.co.uk Tel: +44 1932 342 797
=========================================================================== 3. Keywords kernel networking ISDN pop3 mail ipppd
=========================================================================== 4. Version Information 4.1 ipop3d v7.59 4.2 isdn drivers (drivers/isdn): isdn-2.2-1999-06-21 4.3 isdn utils: isdn4k-utils-3.0beta2 4.4 kernel (/proc/version) Linux version 2.2.7 (root@amber.mss) (gcc version egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)) #9 SMP Thu Jul 8 09:42:50 BST 1999
NOTE 1: Kernel's 2.2.5 and 2.2.10 have the same problem NOTE 2: Older version of ipop3d, isdn utils and isdn drivers also.
The base system is Redhat V6 =========================================================================== 5. Oops message (run through ksymoops)
Options used: -v /usr/src/linux/vmlinux (specified) -o /usr/src/linux/modules (specified) -k /proc/ksyms (specified) -l /proc/modules (default) -m /boot/System.map (specified) -c 1 (default)
DEBUG: level 2 DEBUG: env default KSYMOOPS_NM=/usr/bin/nm DEBUG: env default KSYMOOPS_FIND=/usr/bin/find DEBUG: env default KSYMOOPS_OBJDUMP=/usr/bin/objdump DEBUG: re_compile '^([0-9a-fA-F]{4,}) +([^ ]) +([^ ]+)$' 3 sub expression(s) DEBUG: re_compile '^[ ]*\[*<([0-9a-fA-F]{4,})>\]*[ ]*' 1 sub expression(s) DEBUG: re_compile '^[ *]*([0-9a-fA-F]{4,})[ ]*' 1 sub expression(s) DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/vmlinux' DEBUG: re_compile '^(.*)_R[0-9a-fA-F]{8,}$' 1 sub expression(s) DEBUG: read_nm_symbols vmlinux used 7155 out of 8023 entries DEBUG: sorting symbols for vmlinux (na) DEBUG: ss_compress on table vmlinux, before 7155 after 7155 DEBUG: ss_compress on table vmlinux, before 7155 after 7128 DEBUG: add_Version vmlinux 131591 2.2.7 DEBUG: read_ksyms /proc/ksyms DEBUG: re_compile '^([0-9a-fA-F]{4,}) +([^ ]+)([ ]+\[([^ ]+)\])?$' 4 sub expression(s) DEBUG: sorting symbols for hisax (na) DEBUG: ss_compress on table hisax, before 1 after 1 DEBUG: ss_compress on table hisax, before 1 after 1 DEBUG: sorting symbols for isdn (na) DEBUG: ss_compress on table isdn, before 5 after 5 DEBUG: ss_compress on table isdn, before 5 after 5 DEBUG: sorting symbols for slhc (na) DEBUG: ss_compress on table slhc, before 6 after 6 DEBUG: ss_compress on table slhc, before 6 after 6 DEBUG: sorting symbols for tulip (na) DEBUG: ss_compress on table tulip, before 2 after 2 DEBUG: ss_compress on table tulip, before 2 after 2 DEBUG: sorting symbols for aic7xxx (na) DEBUG: ss_compress on table aic7xxx, before 14 after 14 DEBUG: ss_compress on table aic7xxx, before 14 after 14 DEBUG: sorting symbols for ksyms_base (na) DEBUG: ss_compress on table ksyms_base, before 760 after 760 DEBUG: ss_compress on table ksyms_base, before 760 after 760 DEBUG: read_ksyms hisax used 1 out of 10 entries DEBUG: read_ksyms isdn used 5 out of 10 entries DEBUG: read_ksyms slhc used 6 out of 10 entries DEBUG: read_ksyms tulip used 2 out of 10 entries DEBUG: read_ksyms aic7xxx used 14 out of 20 entries DEBUG: read_ksyms ksyms_base used 760 out of 901 entries DEBUG: expand_objects checking '/usr/src/linux/modules' - directory, expanding DEBUG: find_objects command '/usr/bin/find /usr/src/linux/modules -follow -name '*.o' -print' DEBUG: find_objects - /usr/src/linux/modules/loop.o DEBUG: find_objects - /usr/src/linux/modules/rd.o DEBUG: find_objects - /usr/src/linux/modules/ide-tape.o DEBUG: find_objects - /usr/src/linux/modules/ide-floppy.o DEBUG: find_objects - /usr/src/linux/modules/nbd.o DEBUG: find_objects - /usr/src/linux/modules/ftape.o DEBUG: find_objects - /usr/src/linux/modules/zftape.o DEBUG: find_objects - /usr/src/linux/modules/zft-compressor.o DEBUG: find_objects - /usr/src/linux/modules/lp.o DEBUG: find_objects - /usr/src/linux/modules/qpmouse.o DEBUG: find_objects - /usr/src/linux/modules/ppp.o DEBUG: find_objects - /usr/src/linux/modules/slhc.o DEBUG: find_objects - /usr/src/linux/modules/ppp_deflate.o DEBUG: find_objects - /usr/src/linux/modules/bsd_comp.o DEBUG: find_objects - /usr/src/linux/modules/dummy.o DEBUG: find_objects - /usr/src/linux/modules/tulip.o DEBUG: find_objects - /usr/src/linux/modules/parport.o DEBUG: find_objects - /usr/src/linux/modules/parport_pc.o DEBUG: find_objects - /usr/src/linux/modules/st.o DEBUG: find_objects - /usr/src/linux/modules/sr_mod.o DEBUG: find_objects - /usr/src/linux/modules/aha1542.o DEBUG: find_objects - /usr/src/linux/modules/aic7xxx.o DEBUG: find_objects - /usr/src/linux/modules/ide-scsi.o DEBUG: find_objects - /usr/src/linux/modules/parport_probe.o DEBUG: find_objects - /usr/src/linux/modules/hisax.o DEBUG: find_objects - /usr/src/linux/modules/isdn.o DEBUG: find_objects - /usr/src/linux/modules/isdn_bsdcomp.o DEBUG: find_objects - /usr/src/linux/modules/coda.o DEBUG: find_objects - /usr/src/linux/modules/fat.o DEBUG: find_objects - /usr/src/linux/modules/msdos.o DEBUG: find_objects - /usr/src/linux/modules/vfat.o DEBUG: find_objects - /usr/src/linux/modules/isofs.o DEBUG: find_objects - /usr/src/linux/modules/umsdos.o DEBUG: find_objects - /usr/src/linux/modules/smbfs.o DEBUG: find_objects - /usr/src/linux/modules/ntfs.o DEBUG: find_objects - /usr/src/linux/modules/nls_cp437.o DEBUG: find_objects - /usr/src/linux/modules/nls_cp850.o DEBUG: find_objects - /usr/src/linux/modules/nls_iso8859-1.o DEBUG: find_objects - /usr/src/linux/modules/ip_masq_user.o DEBUG: find_objects - /usr/src/linux/modules/ip_masq_ftp.o DEBUG: find_objects - /usr/src/linux/modules/ip_masq_irc.o DEBUG: find_objects - /usr/src/linux/modules/ip_masq_raudio.o DEBUG: find_objects - /usr/src/linux/modules/ip_masq_quake.o DEBUG: find_objects - /usr/src/linux/modules/ip_masq_vdolive.o DEBUG: find_objects - /usr/src/linux/modules/ip_masq_cuseeme.o DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/loop.o' DEBUG: read_nm_symbols /usr/src/linux/modules/loop.o used 34 out of 40 entries DEBUG: sorting symbols for /usr/src/linux/modules/loop.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/loop.o, before 34 after 34 DEBUG: ss_compress on table /usr/src/linux/modules/loop.o, before 34 after 34 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/rd.o' DEBUG: read_nm_symbols /usr/src/linux/modules/rd.o used 15 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/rd.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/rd.o, before 15 after 15 DEBUG: ss_compress on table /usr/src/linux/modules/rd.o, before 15 after 15 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ide-tape.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ide-tape.o used 88 out of 103 entries DEBUG: sorting symbols for /usr/src/linux/modules/ide-tape.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ide-tape.o, before 88 after 88 DEBUG: ss_compress on table /usr/src/linux/modules/ide-tape.o, before 88 after 88 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ide-floppy.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ide-floppy.o used 46 out of 50 entries DEBUG: sorting symbols for /usr/src/linux/modules/ide-floppy.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ide-floppy.o, before 46 after 46 DEBUG: ss_compress on table /usr/src/linux/modules/ide-floppy.o, before 46 after 46 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/nbd.o' DEBUG: read_nm_symbols /usr/src/linux/modules/nbd.o used 22 out of 30 entries DEBUG: sorting symbols for /usr/src/linux/modules/nbd.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/nbd.o, before 22 after 22 DEBUG: ss_compress on table /usr/src/linux/modules/nbd.o, before 22 after 22 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ftape.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ftape.o used 384 out of 435 entries DEBUG: sorting symbols for /usr/src/linux/modules/ftape.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ftape.o, before 384 after 384 DEBUG: ss_compress on table /usr/src/linux/modules/ftape.o, before 384 after 383 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/zftape.o' DEBUG: read_nm_symbols /usr/src/linux/modules/zftape.o used 198 out of 211 entries DEBUG: sorting symbols for /usr/src/linux/modules/zftape.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/zftape.o, before 198 after 198 DEBUG: ss_compress on table /usr/src/linux/modules/zftape.o, before 198 after 197 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/zft-compressor.o' DEBUG: read_nm_symbols /usr/src/linux/modules/zft-compressor.o used 46 out of 50 entries DEBUG: sorting symbols for /usr/src/linux/modules/zft-compressor.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/zft-compressor.o, before 46 after 46 DEBUG: ss_compress on table /usr/src/linux/modules/zft-compressor.o, before 46 after 46 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/lp.o' DEBUG: read_nm_symbols /usr/src/linux/modules/lp.o used 28 out of 30 entries DEBUG: sorting symbols for /usr/src/linux/modules/lp.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/lp.o, before 28 after 28 DEBUG: ss_compress on table /usr/src/linux/modules/lp.o, before 28 after 28 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/qpmouse.o' DEBUG: read_nm_symbols /usr/src/linux/modules/qpmouse.o used 22 out of 30 entries DEBUG: sorting symbols for /usr/src/linux/modules/qpmouse.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/qpmouse.o, before 22 after 22 DEBUG: ss_compress on table /usr/src/linux/modules/qpmouse.o, before 22 after 22 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ppp.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ppp.o used 73 out of 86 entries DEBUG: sorting symbols for /usr/src/linux/modules/ppp.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ppp.o, before 73 after 73 DEBUG: ss_compress on table /usr/src/linux/modules/ppp.o, before 73 after 73 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/slhc.o' DEBUG: read_nm_symbols /usr/src/linux/modules/slhc.o used 28 out of 30 entries DEBUG: sorting symbols for /usr/src/linux/modules/slhc.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/slhc.o, before 28 after 28 DEBUG: ss_compress on table /usr/src/linux/modules/slhc.o, before 28 after 28 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ppp_deflate.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ppp_deflate.o used 120 out of 123 entries DEBUG: sorting symbols for /usr/src/linux/modules/ppp_deflate.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ppp_deflate.o, before 120 after 120 DEBUG: ss_compress on table /usr/src/linux/modules/ppp_deflate.o, before 120 after 120 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/bsd_comp.o' DEBUG: read_nm_symbols /usr/src/linux/modules/bsd_comp.o used 19 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/bsd_comp.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/bsd_comp.o, before 19 after 19 DEBUG: ss_compress on table /usr/src/linux/modules/bsd_comp.o, before 19 after 19 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/dummy.o' DEBUG: read_nm_symbols /usr/src/linux/modules/dummy.o used 13 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/dummy.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/dummy.o, before 13 after 13 DEBUG: ss_compress on table /usr/src/linux/modules/dummy.o, before 13 after 13 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/tulip.o' DEBUG: read_nm_symbols /usr/src/linux/modules/tulip.o used 64 out of 72 entries DEBUG: sorting symbols for /usr/src/linux/modules/tulip.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/tulip.o, before 64 after 64 DEBUG: ss_compress on table /usr/src/linux/modules/tulip.o, before 64 after 64 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/parport.o' DEBUG: read_nm_symbols /usr/src/linux/modules/parport.o used 68 out of 72 entries DEBUG: sorting symbols for /usr/src/linux/modules/parport.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/parport.o, before 68 after 68 DEBUG: ss_compress on table /usr/src/linux/modules/parport.o, before 68 after 68 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/parport_pc.o' DEBUG: read_nm_symbols /usr/src/linux/modules/parport_pc.o used 62 out of 72 entries DEBUG: sorting symbols for /usr/src/linux/modules/parport_pc.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/parport_pc.o, before 62 after 62 DEBUG: ss_compress on table /usr/src/linux/modules/parport_pc.o, before 62 after 62 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/st.o' DEBUG: read_nm_symbols /usr/src/linux/modules/st.o used 60 out of 60 entries DEBUG: sorting symbols for /usr/src/linux/modules/st.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/st.o, before 60 after 60 DEBUG: ss_compress on table /usr/src/linux/modules/st.o, before 60 after 60 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/sr_mod.o' DEBUG: read_nm_symbols /usr/src/linux/modules/sr_mod.o used 50 out of 50 entries DEBUG: sorting symbols for /usr/src/linux/modules/sr_mod.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/sr_mod.o, before 50 after 50 DEBUG: ss_compress on table /usr/src/linux/modules/sr_mod.o, before 50 after 50 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/aha1542.o' DEBUG: read_nm_symbols /usr/src/linux/modules/aha1542.o used 41 out of 50 entries DEBUG: sorting symbols for /usr/src/linux/modules/aha1542.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/aha1542.o, before 41 after 41 DEBUG: ss_compress on table /usr/src/linux/modules/aha1542.o, before 41 after 41 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/aic7xxx.o' DEBUG: read_nm_symbols /usr/src/linux/modules/aic7xxx.o used 127 out of 147 entries DEBUG: sorting symbols for /usr/src/linux/modules/aic7xxx.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/aic7xxx.o, before 127 after 127 DEBUG: ss_compress on table /usr/src/linux/modules/aic7xxx.o, before 127 after 127 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ide-scsi.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ide-scsi.o used 34 out of 40 entries DEBUG: sorting symbols for /usr/src/linux/modules/ide-scsi.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ide-scsi.o, before 34 after 34 DEBUG: ss_compress on table /usr/src/linux/modules/ide-scsi.o, before 34 after 34 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/parport_probe.o' DEBUG: read_nm_symbols /usr/src/linux/modules/parport_probe.o used 12 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/parport_probe.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/parport_probe.o, before 12 after 12 DEBUG: ss_compress on table /usr/src/linux/modules/parport_probe.o, before 12 after 12 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/hisax.o' DEBUG: read_nm_symbols /usr/src/linux/modules/hisax.o used 427 out of 435 entries DEBUG: sorting symbols for /usr/src/linux/modules/hisax.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/hisax.o, before 427 after 427 DEBUG: ss_compress on table /usr/src/linux/modules/hisax.o, before 427 after 427 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/isdn.o' DEBUG: read_nm_symbols /usr/src/linux/modules/isdn.o used 264 out of 303 entries DEBUG: sorting symbols for /usr/src/linux/modules/isdn.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/isdn.o, before 264 after 264 DEBUG: ss_compress on table /usr/src/linux/modules/isdn.o, before 264 after 264 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/isdn_bsdcomp.o' DEBUG: read_nm_symbols /usr/src/linux/modules/isdn_bsdcomp.o used 18 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/isdn_bsdcomp.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/isdn_bsdcomp.o, before 18 after 18 DEBUG: ss_compress on table /usr/src/linux/modules/isdn_bsdcomp.o, before 18 after 18 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/coda.o' DEBUG: read_nm_symbols /usr/src/linux/modules/coda.o used 170 out of 176 entries DEBUG: sorting symbols for /usr/src/linux/modules/coda.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/coda.o, before 170 after 170 DEBUG: ss_compress on table /usr/src/linux/modules/coda.o, before 170 after 170 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/fat.o' DEBUG: read_nm_symbols /usr/src/linux/modules/fat.o used 174 out of 176 entries DEBUG: sorting symbols for /usr/src/linux/modules/fat.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/fat.o, before 174 after 174 DEBUG: ss_compress on table /usr/src/linux/modules/fat.o, before 174 after 173 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/msdos.o' DEBUG: read_nm_symbols /usr/src/linux/modules/msdos.o used 51 out of 60 entries DEBUG: sorting symbols for /usr/src/linux/modules/msdos.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/msdos.o, before 51 after 51 DEBUG: ss_compress on table /usr/src/linux/modules/msdos.o, before 51 after 51 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/vfat.o' DEBUG: read_nm_symbols /usr/src/linux/modules/vfat.o used 72 out of 72 entries DEBUG: sorting symbols for /usr/src/linux/modules/vfat.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/vfat.o, before 72 after 72 DEBUG: ss_compress on table /usr/src/linux/modules/vfat.o, before 72 after 72 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/isofs.o' DEBUG: read_nm_symbols /usr/src/linux/modules/isofs.o used 67 out of 72 entries DEBUG: sorting symbols for /usr/src/linux/modules/isofs.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/isofs.o, before 67 after 67 DEBUG: ss_compress on table /usr/src/linux/modules/isofs.o, before 67 after 66 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/umsdos.o' DEBUG: read_nm_symbols /usr/src/linux/modules/umsdos.o used 123 out of 123 entries DEBUG: sorting symbols for /usr/src/linux/modules/umsdos.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/umsdos.o, before 123 after 123 DEBUG: ss_compress on table /usr/src/linux/modules/umsdos.o, before 123 after 123 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/smbfs.o' DEBUG: read_nm_symbols /usr/src/linux/modules/smbfs.o used 149 out of 176 entries DEBUG: sorting symbols for /usr/src/linux/modules/smbfs.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/smbfs.o, before 149 after 149 DEBUG: ss_compress on table /usr/src/linux/modules/smbfs.o, before 149 after 149 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ntfs.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ntfs.o used 156 out of 176 entries DEBUG: sorting symbols for /usr/src/linux/modules/ntfs.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ntfs.o, before 156 after 156 DEBUG: ss_compress on table /usr/src/linux/modules/ntfs.o, before 156 after 155 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/nls_cp437.o' DEBUG: read_nm_symbols /usr/src/linux/modules/nls_cp437.o used 17 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/nls_cp437.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/nls_cp437.o, before 17 after 17 DEBUG: ss_compress on table /usr/src/linux/modules/nls_cp437.o, before 17 after 17 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/nls_cp850.o' DEBUG: read_nm_symbols /usr/src/linux/modules/nls_cp850.o used 14 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/nls_cp850.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/nls_cp850.o, before 14 after 14 DEBUG: ss_compress on table /usr/src/linux/modules/nls_cp850.o, before 14 after 14 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/nls_iso8859-1.o' DEBUG: read_nm_symbols /usr/src/linux/modules/nls_iso8859-1.o used 11 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/nls_iso8859-1.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/nls_iso8859-1.o, before 11 after 11 DEBUG: ss_compress on table /usr/src/linux/modules/nls_iso8859-1.o, before 11 after 11 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ip_masq_user.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ip_masq_user.o used 21 out of 30 entries DEBUG: sorting symbols for /usr/src/linux/modules/ip_masq_user.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_user.o, before 21 after 21 DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_user.o, before 21 after 21 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ip_masq_ftp.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ip_masq_ftp.o used 17 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/ip_masq_ftp.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_ftp.o, before 17 after 17 DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_ftp.o, before 17 after 17 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ip_masq_irc.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ip_masq_irc.o used 16 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/ip_masq_irc.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_irc.o, before 16 after 16 DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_irc.o, before 16 after 16 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ip_masq_raudio.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ip_masq_raudio.o used 16 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/ip_masq_raudio.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_raudio.o, before 16 after 16 DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_raudio.o, before 16 after 16 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ip_masq_quake.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ip_masq_quake.o used 14 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/ip_masq_quake.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_quake.o, before 14 after 14 DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_quake.o, before 14 after 14 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ip_masq_vdolive.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ip_masq_vdolive.o used 15 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/ip_masq_vdolive.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_vdolive.o, before 15 after 15 DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_vdolive.o, before 15 after 15 DEBUG: read_nm_symbols command '/usr/bin/nm /usr/src/linux/modules/ip_masq_cuseeme.o' DEBUG: read_nm_symbols /usr/src/linux/modules/ip_masq_cuseeme.o used 16 out of 20 entries DEBUG: sorting symbols for /usr/src/linux/modules/ip_masq_cuseeme.o (na) DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_cuseeme.o, before 16 after 16 DEBUG: ss_compress on table /usr/src/linux/modules/ip_masq_cuseeme.o, before 16 after 16 DEBUG: read_lsmod /proc/modules DEBUG: re_compile '^[ ]*([^ ]+)[ ]*([^ ]+)[ ]*([^ ]+)[ ]*(.*)$' 4 sub expression(s) DEBUG: sorting symbols for lsmod (na) DEBUG: ss_compress on table lsmod, before 11 after 11 DEBUG: ss_compress on table lsmod, before 11 after 11 DEBUG: read_lsmod lsmod used 11 out of 20 entries DEBUG: read_system_map /boot/System.map DEBUG: sorting symbols for System.map (na) DEBUG: ss_compress on table System.map, before 6853 after 6853 DEBUG: ss_compress on table System.map, before 6853 after 6853 DEBUG: add_Version System.map 131591 2.2.7 DEBUG: read_system_map System.map used 6853 out of 8023 entries DEBUG: merge_maps DEBUG: sorting symbols for Version_ (na) DEBUG: ss_compress on table Version_, before 2 after 2 DEBUG: ss_compress on table Version_, before 2 after 2 DEBUG: Version 2.2.7 Warning in compare_ksyms_lsmod, module ip_masq_cuseeme is in lsmod but not in ksyms, probably no symbols exported Warning in compare_ksyms_lsmod, module ip_masq_ftp is in lsmod but not in ksyms, probably no symbols exported Warning in compare_ksyms_lsmod, module ip_masq_irc is in lsmod but not in ksyms, probably no symbols exported Warning in compare_ksyms_lsmod, module ip_masq_quake is in lsmod but not in ksyms, probably no symbols exported Warning in compare_ksyms_lsmod, module ip_masq_raudio is in lsmod but not in ksyms, probably no symbols exported Warning in compare_ksyms_lsmod, module ip_masq_vdolive is in lsmod but not in ksyms, probably no symbols exported DEBUG: compare_maps ksyms_base vs vmlinux, vmlinux takes precedence DEBUG: compare_maps System.map vs vmlinux, vmlinux takes precedence DEBUG: compare_maps vmlinux vs System.map, vmlinux takes precedence DEBUG: compare_maps ksyms_base vs System.map, System.map takes precedence DEBUG: ksyms hisax matches to /usr/src/linux/modules/hisax.o based on unique symbol HiSax_closecard DEBUG: adjust_object_offsets hisax DEBUG: compare_maps hisax vs /usr/src/linux/modules/hisax.o, hisax takes precedence DEBUG: ksyms isdn matches to /usr/src/linux/modules/isdn.o based on unique symbol isdn_ppp_register_compressor DEBUG: adjust_object_offsets isdn DEBUG: compare_maps isdn vs /usr/src/linux/modules/isdn.o, isdn takes precedence DEBUG: ksyms slhc matches to /usr/src/linux/modules/slhc.o based on unique symbol slhc_compress DEBUG: adjust_object_offsets slhc DEBUG: compare_maps slhc vs /usr/src/linux/modules/slhc.o, slhc takes precedence DEBUG: ksyms tulip matches to /usr/src/linux/modules/tulip.o based on unique symbol tulip_debug DEBUG: adjust_object_offsets tulip DEBUG: compare_maps tulip vs /usr/src/linux/modules/tulip.o, tulip takes precedence DEBUG: ksyms aic7xxx matches to /usr/src/linux/modules/aic7xxx.o based on unique symbol aic7xxx_abort DEBUG: adjust_object_offsets aic7xxx DEBUG: compare_maps aic7xxx vs /usr/src/linux/modules/aic7xxx.o, aic7xxx takes precedence DEBUG: append_map vmlinux to merged DEBUG: append_map ksyms_base to merged DEBUG: append_map System.map to merged DEBUG: append_map /usr/src/linux/modules/hisax.o to merged DEBUG: append_map /usr/src/linux/modules/isdn.o to merged DEBUG: append_map /usr/src/linux/modules/slhc.o to merged DEBUG: append_map /usr/src/linux/modules/tulip.o to merged DEBUG: append_map /usr/src/linux/modules/aic7xxx.o to merged DEBUG: sorting symbols for merged (atn) DEBUG: ss_compress on table merged, before 15511 after 15511 DEBUG: sorting symbols for merged (atn) DEBUG: ss_compress on table merged, before 15511 after 7600 DEBUG: re_compile '^(([^ ]{3} [ 0-9][0-9] [0-9]{2}:[0-9]{2}:[0-9]{2} [^ ]+ kernel: +)|(<[0-9]+>)|([ ]+))+' 4 sub expression(s) DEBUG: re_compile '^((Stack: )|(Stack from )|([0-9a-fA-F]{4,})|(Call Trace: )|(\[*<([0-9a-fA-F]{4,})>\]*[ ]*)|(Version_[0-9]+)|(Trace: )|(Call backtrace:)|(Process .*stackpage=)|(Call Trace:[ ])|(Code *:[ ])|(Kernel panic)|(In swapper task)|(Corrupted stack page)|(invalid operand: )|(Oops: )|(Cpu: +[0-9])|(current->tss)|(\*pde +=)|(EIP: )|(EFLAGS: )|(eax: )|(esi: )|(ds: )|(pc[:=])|(68060 access)|(Exception at )|(d[04]: )|(Frame format=)|(wb [0-9] stat)|(push data: )|(baddr=)|(Bad unaligned kernel)|(Forwarding unaligned exception)|(: unhandled unaligned exception)|(<sc)|(pc *=)|(r[0-9]+ *=)|(gp *=)|(tsk->)|(PSR: )|([goli][04]: )|(Instruction DUMP: )|(Caller\[)|(TSTATE: )|(MSR: )|(TASK = )|(last math )|(GPR[0-9]+: )|(\$[0-9 ]+:)|(epc )|(Status:)|(Cause :)|(Backtrace:)|(Function entered at)|(\*pgd =)|(Internal error)|(pc :)|(sp :)|(r[0-9][0-9 ]:)|(Flags:)|(Control:))' 64 sub expression(s) DEBUG: re_compile '(Unable to handle kernel)|(Aiee)|(die_if_kernel)|(\([0-9]\): Oops )|(: memory violation)|(: Exception at)|(: Arithmetic fault)|(: Instruction fault)|(: arithmetic trap)|(: unaligned trap)|(\([0-9]+\): ((Whee)|(Oops)|(Kernel)|(Penguin)|(Too many Penguin)|(BOGUS)))|(kernel pc )|(trap at PC: )|(bad area pc )|(NIP: )|( ra *=))' 23 sub expression(s) DEBUG: re_compile '^(i[04]: |Instruction DUMP: |Caller\[)' 1 sub expression(s) Jul 1 18:33:10 amber kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000004 DEBUG: re_compile '^PSR: [0-9a-fA-F]+ PC: ([0-9a-fA-F]{4,})[ ]*' 1 sub expression(s) DEBUG: re_compile '^TSTATE: [0-9a-fA-F]{16} TPC: ([0-9a-fA-F]{4,})[ ]*' 1 sub expression(s) DEBUG: re_compile '((kernel pc )|(trap at PC: )|(bad area pc )|(NIP: ))([0-9a-fA-F]{4,})[ ]*' 6 sub expression(s) DEBUG: re_compile '^(epc[ ]*:+[ ]*)([0-9a-fA-F]{4,})[ ]*' 2 sub expression(s) DEBUG: re_compile '^((EIP:[ ]+.*)|(PC[ ]*=[ ]*)|(pc *: *))\[*<([0-9a-fA-F]{4,})>\]*[ ]*' 5 sub expression(s) DEBUG: re_compile '(ra[ ]*=+[ ]*)([0-9a-fA-F]{4,})[ ]*' 2 sub expression(s) DEBUG: re_compile '^[io][04]: [0-9a-fA-F iosp:]+ ([io]7|ret_pc): ([0-9a-fA-F]{4,})[ ]*' 2 sub expression(s) DEBUG: re_compile '^((Call Trace: )|(Trace: )|(\[*<([0-9a-fA-F]{4,})>\]*[ ]*)|(Call backtrace:)|(([0-9a-fA-F]{4,})[ ]*)|(Function entered at (\[*<([0-9a-fA-F]{4,})>\]*[ ]*))|(Caller\[([0-9a-fA-F]{4,})[ ]*\]))' 13 sub expression(s) DEBUG: re_compile '^((Instruction DUMP)|(Code *)):[ ]+((general protection.*)|(<[0-9]+>)|(([<(]?[0-9a-fA-F]+[>)]?[ ]+)+[<(]?[0-9a-fA-F]+[>)]?))(.*)$' 9 sub expression(s) Jul 1 18:33:10 amber kernel: current->tss.cr3 = 020c7000, %cr3 = 020c7000 Jul 1 18:33:10 amber kernel: *pde = 00000000 Jul 1 18:33:10 amber kernel: Oops: 0002 Jul 1 18:33:10 amber kernel: CPU: 1 Jul 1 18:33:10 amber kernel: EIP: 0010:[cleanup_rbuf+68/184] Jul 1 18:33:10 amber kernel: EFLAGS: 00010246 Jul 1 18:33:10 amber kernel: eax: 00000000 ebx: c17f7644 ecx: c9fddb40 edx: 00000000 Jul 1 18:33:10 amber kernel: esi: c17f7600 edi: c17f7600 ebp: c17f776c esp: c228de9c Jul 1 18:33:10 amber kernel: ds: 0018 es: 0018 ss: 0018 Jul 1 18:33:10 amber kernel: Process ipop3d (pid: 3915, process nr: 68, stackpage=c228d000) Jul 1 18:33:10 amber kernel: Stack: c016a49e c17f7600 00000000 c17f7600 00000000 c228df74 00000400 0ea1cc64 Jul 1 18:33:10 amber kernel: 00000001 00000000 00000001 00000000 c17f76b0 c228c000 c1a7fd64 00015478 Jul 1 18:33:10 amber kernel: c017875a c17f7600 c228df74 00000400 00000000 00000000 c228df08 c1a7fd4c Jul 1 18:33:10 amber kernel: Call Trace: [tcp_recvmsg+758/1412] [inet_recvmsg+142/168] [sock_recvmsg+58/172] [sock_read+150/160] [sys_read+210/260] [system_call+52/56] Jul 1 18:33:10 amber kernel: Code: 89 42 04 89 10 8d 41 70 f0 ff 49 70 0f 94 c0 84 c0 74 b5 51 Warning: trailing garbage ignored on Code: line Text: 'Code: 89 42 04 89 10 8d 41 70 f0 ff 49 70 0f 94 c0 84 c0 74 b5 51 ' Garbage: ' ' DEBUG: Oops_decode DEBUG: re_compile '^([<(]?)([0-9a-fA-F]+)[>)]?[ ]*' 2 sub expression(s) DEBUG: find_fullpath ./ksymoops DEBUG: Oops_objdump command '/usr/bin/objdump -dhf /tmp/fileRBb19a' DEBUG: Oops_decode - DEBUG: re_compile '^[ ]*([0-9a-fA-F]+)( <_EIP[^>]*>)?:(.* +(0?x?[0-9a-fA-F]+ +)<_EIP\+0?x?([0-9a-fA-F]+)>[ ]*$)?.*' 5 sub expression(s) DEBUG: Oops_decode - /tmp/fileRBb19a: file format elf32-i386 DEBUG: Oops_decode - architecture: i386, flags 0x00000112: DEBUG: Oops_decode - EXEC_P, HAS_SYMS, D_PAGED DEBUG: Oops_decode - start address 0x00000000 DEBUG: Oops_decode - DEBUG: Oops_decode - Sections: DEBUG: Oops_decode - Idx Name Size VMA LMA File off Algn DEBUG: Oops_decode - 0 .text 00000040 00000000 00000000 00001000 2**4 DEBUG: Oops_decode - CONTENTS, ALLOC, LOAD, READONLY, CODE DEBUG: Oops_decode - Disassembly of section .text: DEBUG: Oops_decode - DEBUG: Oops_decode - 00000000 <_EIP>: DEBUG: Oops_decode - 0: 89 42 04 movl %eax,0x4(%edx) DEBUG: Oops_decode - 3: 89 10 movl %edx,(%eax) DEBUG: Oops_decode - 5: 8d 41 70 leal 0x70(%ecx),%eax DEBUG: Oops_decode - 8: f0 ff 49 70 lock decl 0x70(%ecx) DEBUG: Oops_decode - c: 0f 94 c0 sete %al DEBUG: Oops_decode - f: 84 c0 testb %al,%al DEBUG: Oops_decode - 11: 74 b5 je ffffffc8 <_EIP+0xffffffc8> DEBUG: Oops_decode - 13: 51 pushl %ecx DEBUG: Oops_decode - ... DEBUG: Oops_format
Code: 00000000 Before first symbol 00000000 <_IP>: <=== Code: 00000000 Before first symbol 0: 89 42 04 movl %eax,0x4(%edx) <=== Code: 00000003 Before first symbol 3: 89 10 movl %edx,(%eax) Code: 00000005 Before first symbol 5: 8d 41 70 leal 0x70(%ecx),%eax Code: 00000008 Before first symbol 8: f0 ff 49 70 lock decl 0x70(%ecx) Code: 0000000c Before first symbol c: 0f 94 c0 sete %al Code: 0000000f Before first symbol f: 84 c0 testb %al,%al Code: 00000011 Before first symbol 11: 74 b5 je ffffffc8 <END_OF_CODE+2f78b470/????> Code: 00000013 Before first symbol 13: 51 pushl %ecx
7 warnings issued. Results may not be reliable.
=========================================================================== 6. Cannot force the problem to happen. The person who logs in is successful most of the time. She logs in maybe 20 times a day. The problem happens once or twice a week. Here are the last few lines out of syslog before the crash. The lines are the same regardless of whether the Oops followed or not.
********************SYSLOG OUTPUT********************* Jul 8 08:45:03 amber ipppd[708]: Check_passwd called with user=tina Jul 8 08:45:03 amber ipppd[708]: user tina logged in Jul 8 08:45:03 amber ipppd[708]: MPPP negotiation, He: No We: No Jul 8 08:45:03 amber ipppd[708]: CCP enabled! Trying CCP. Jul 8 08:45:03 amber ipppd[708]: CCP: got ccp-unit 0 for link 0 (protocol: 0x80fd) Jul 8 08:45:03 amber ipppd[708]: ccp_resetci! Jul 8 08:45:03 amber ipppd[708]: Kernel check for LZS failed Jul 8 08:45:03 amber ipppd[708]: local IP address 192.168.1.103 Jul 8 08:45:03 amber ipppd[708]: remote IP address 192.168.22.103 Jul 8 08:45:09 amber ipppd[708]: ccp_resetci! Jul 8 08:45:09 amber ipppd[708]: Kernel check for LZS failed Jul 8 08:45:10 amber kernel: ippp: no decompressor defined! Jul 8 08:45:13 amber last message repeated 2 times Jul 8 08:45:13 amber kernel: Warning: kfree_skb passed an skb still on a list (from c016d797). ********************END OF SYSLOG OUTPUT************************
=========================================================================== 7. Environment
7.1 ver_linux -- Versions installed: (if some fields are empty or looks -- unusual then possibly you have very old versions) Linux amber.mss 2.2.7 #9 SMP Thu Jul 8 09:42:50 BST 1999 i686 unknown Kernel modules 2.1.121 Gnu C egcs-2.91.66 Binutils 2.9.1.0.23 Linux C Library 2.1.1 Dynamic linker ldd (GNU libc) 2.1.1 Procps 2.0.2 Mount 2.9o Net-tools 1.51 Kbd [option...] Sh-utils 1.16 Modules Loaded ip_masq_vdolive ip_masq_raudio ip_masq_quake ip_masq_irc ip_masq_ftp ip_masq_cuseeme hisax isdn slhc tulip aic7xxx
[7.2] /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 5 model name : Pentium II (Deschutes) stepping : 2 cpu MHz : 451.032446 cache size : 512 KB fdiv_bug : no hlt_bug : no sep_bug : no f00f_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx osfxsr bogomips : 448.92
processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 5 model name : Pentium II (Deschutes) stepping : 2 cpu MHz : 451.032446 cache size : 512 KB fdiv_bug : no hlt_bug : no sep_bug : no f00f_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx osfxsr bogomips : 450.56
[7.3] /proc/modules ip_masq_vdolive 1000 0 (unused) ip_masq_raudio 2624 0 (unused) ip_masq_quake 1008 0 (unused) ip_masq_irc 1324 0 (unused) ip_masq_ftp 2096 0 ip_masq_cuseeme 776 0 (unused) hisax 112832 5 isdn 90056 18 [hisax] slhc 4132 7 [isdn] tulip 23568 1 (autoclean) aic7xxx 86088 5
[7.4] /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: SEAGATE Model: ST39140W Rev: 1444 Type: Direct-Access ANSI SCSI revision: 02 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: IBM Model: DCAS-34330W Rev: S61A Type: Direct-Access ANSI SCSI revision: 02 Host: scsi0 Channel: 00 Id: 04 Lun: 00 Vendor: IBM Model: DCAS-34330W Rev: S65A Type: Direct-Access ANSI SCSI revision: 02 Host: scsi0 Channel: 00 Id: 06 Lun: 00 Vendor: IBM Model: DCAS-34330W Rev: S65A Type: Direct-Access ANSI SCSI revision: 02
[7.5] ISDN subsystem Command to load hisax driver... /sbin/modprobe -v hisax id=HiSax1%HiSax2 type=27,27 protocol=2,2
[7.6] /proc/pci Isdn cards (from /proc/pci) Bus 0, device 17, function 0: Network controller: AVM A1 (Fritz) (rev 2). Medium devsel. Fast back-to-back capable. IRQ 18. Non-prefetchable 32 bit memory at 0xe9002000 [0xe9002000]. I/O at 0xdc00 [0xdc01]. Bus 0, device 18, function 0: Network controller: AVM A1 (Fritz) (rev 2). Medium devsel. Fast back-to-back capable. IRQ 19. Non-prefetchable 32 bit memory at 0xe9003000 [0xe9003000]. I/O at 0xe000 [0xe001].
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |