lkml.org 
[lkml]   [2004]   [Nov]   [14]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
Subject2.6.9: unkillable processes during heavy IO
We have two database servers which freeze up during heavy IO load.  The
machines themselves are responsible, but the mysqld processes are forever
locked, unkillable with even kill -9. I can't restart with MySQL without
rebooting the machines.

I can reasonable rule out hardware, since this is happening in the
same way on two identical machines.

I'd like to know how I can debug this, to file a proper bug report.

The hardware/software stack is:

- Dual Opteron 246, SMP kernel, w/ NUMA
- 9 GB of memory (4GB in one zone, 5GB in the other)
- MySQL, running mostly InnoDB, but some MyISAM
- MegaRAID raid-10
- device mapper
- XFS (used as both O_DIRECT from InnoDB and regularly from MyISAM)

At this point I'm going to try changing different variables on
different machines in order to try and isolate it, but it's a slow
process.

- on raw partions, instead of device mapper
- ext3 instead of XFS
- not using O_DIRECT

"Screenshot":

roast:~# killall -9 mysqld
roast:~# killall -9 mysqld
roast:~# ps afx | grep mysqld
31495 ? D 0:08 /usr/local/mysql/bin/mysqld --defaults-extra-file=/usr/local/mysql/data/my.cnf --basedir=/usr/local/mysql/ --datadir=/data/mydb --user=root
--pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
32391 ? D 0:01 /usr/local/mysql/bin/mysqld --defaults-extra-file=/usr/local/mysql/data/my.cnf --basedir=/usr/local/mysql/ --datadir=/data/mydb --user=root
--pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
515 ? D 0:00 /usr/local/mysql/bin/mysqld --defaults-extra-file=/usr/local/mysql/data/my.cnf --basedir=/usr/local/mysql/ --datadir=/data/mydb --user=root
--pid-file=/var/run/mysqld/mysqld.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
517 ? Z 0:00 \_ [mysqld] <defunct>

Next time it hangs like this, how can I get a kernel backtrace or other useful information
for a certain process?

Thanks!

- Brad

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 14:08    [W:0.070 / U:1.240 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site