lkml.org 
[lkml]   [2004]   [Feb]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Subjectdual Opteron very slow I/O (2.4.22)
From
Date
Dear kernel developers!

I'm really out of ideas on this one and also didn't find anything in the
archives. It all began when we installed an AFS fileserver on our
brandnew dual Opteron 240 machine, which has a 3ware 7506-12 RAID5
(2TB), two bcm5702 onboard GigE ports (tg3) and 2GB RAM. I have attached
an archive with more detailed information (lspci, cpuinfo, bootlog,
.config, mtab, ksymoopsed traces, vmstat).

The problem is that the I/O to the 3ware RAID is awfully slow (vmstat
shows 11MB/s) and AFS file operations are even slower: copying files
from the network (scp or plain simple netcat) to a volume stored on the
local fileserver does not get faster than 5MB/s with long stalls in
between, even dd'ing from /dev/zero tops out at 7MB/s. While such a
stall was happening I have triggered a sysrq-t to see what's up and this
showed:

There are four "netcat" type processes (readdata[1-4]), two scp's
(scp[12]), the AFS fileserver.lwp process, three afs_background kernel
tasks, syslogd and klogd in state D, most of them waiting in
__lock_page.

The six processes writing to AFS (readdata* and scp*) called sys_write,
which goes through all of AFS to get back to generic_file_read. Three of
them are in __lock_page, two of them in submit_bh (from ext2_get_branch
as the AFS cache is on ext2) and one is in ext2_get_block.

The fileserver is in sys_readv->generic_file_read->__lock_page->
__run_task_queue.

Klogd is in do_syslog->schedule->do_page_fault-->__lock_page.

The afs_background tasks are waiting inside the AFS module
(afs_osi_Sleep).

If anyone of you has any suggestions what I can try to sort this out, I
would be most grateful. I already have tried the multithreaded
fileserver, but then I have more than 250.000 context switches per
second, which renders the machine unusable. I also have tried a memory
cache for AFS, but that was actually worse, now I have 4GB of disk cache
on a separate IDE disk (hdparm and hdparm -I attached).

Thanks for your help,
Roland

--
+---------------------------+-------------------------+
| TU Muenchen | |
| Physik-Department E18 | Raum 3558 |
| James-Franck-Str. | Telefon 089/289-12592 |
| 85747 Garching | Telefax 089/289-12570 |
+---------------------------+-------------------------+
[unhandled content-type:application/x-compressed-tar]
\
 
 \ /
  Last update: 2005-03-22 14:00    [W:0.031 / U:0.544 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site