Messages in this thread | | | Subject | dual Opteron very slow I/O (2.4.22) | From | Roland Kuhn <> | Date | Tue, 03 Feb 2004 17:23:52 +0100 |
| |
Dear kernel developers!
I'm really out of ideas on this one and also didn't find anything in the archives. It all began when we installed an AFS fileserver on our brandnew dual Opteron 240 machine, which has a 3ware 7506-12 RAID5 (2TB), two bcm5702 onboard GigE ports (tg3) and 2GB RAM. I have attached an archive with more detailed information (lspci, cpuinfo, bootlog, .config, mtab, ksymoopsed traces, vmstat).
The problem is that the I/O to the 3ware RAID is awfully slow (vmstat shows 11MB/s) and AFS file operations are even slower: copying files from the network (scp or plain simple netcat) to a volume stored on the local fileserver does not get faster than 5MB/s with long stalls in between, even dd'ing from /dev/zero tops out at 7MB/s. While such a stall was happening I have triggered a sysrq-t to see what's up and this showed:
There are four "netcat" type processes (readdata[1-4]), two scp's (scp[12]), the AFS fileserver.lwp process, three afs_background kernel tasks, syslogd and klogd in state D, most of them waiting in __lock_page.
The six processes writing to AFS (readdata* and scp*) called sys_write, which goes through all of AFS to get back to generic_file_read. Three of them are in __lock_page, two of them in submit_bh (from ext2_get_branch as the AFS cache is on ext2) and one is in ext2_get_block.
The fileserver is in sys_readv->generic_file_read->__lock_page-> __run_task_queue.
Klogd is in do_syslog->schedule->do_page_fault-->__lock_page.
The afs_background tasks are waiting inside the AFS module (afs_osi_Sleep).
If anyone of you has any suggestions what I can try to sort this out, I would be most grateful. I already have tried the multithreaded fileserver, but then I have more than 250.000 context switches per second, which renders the machine unusable. I also have tried a memory cache for AFS, but that was actually worse, now I have 4GB of disk cache on a separate IDE disk (hdparm and hdparm -I attached).
Thanks for your help, Roland
-- +---------------------------+-------------------------+ | TU Muenchen | | | Physik-Department E18 | Raum 3558 | | James-Franck-Str. | Telefon 089/289-12592 | | 85747 Garching | Telefax 089/289-12570 | +---------------------------+-------------------------+ [unhandled content-type:application/x-compressed-tar] | |