Messages in this thread | | | Date | Fri, 21 Jul 2000 00:57:34 +1000 | From | Robert Cohen <> | Subject | Performance problems with Linux file serving. in recent 2.4 kernels |
| |
I would like to report on my findings about Linux file serving performance. I will start with results for 2.2 kernels and finish with 2.4 results.
I have been doing some evaluation of Linux as a Appleshare file server using Netatalk. The benchmark I am using is mac program called lantest, but the part of it I am using just writes a 30 Meg file and then reads it back and repeats this continuously. And I vary the number of client machines I run it on simultaneously. All tests are done over 100Mbit ethernet.
I started with Kernel version 2.2.17(preX). With 1 client, performance is fine 5-6 Megabytes/sec. But on a 128 Meg server with 2 clients, I hit a performance wall and the cumulative throughput for the write test drops to less than a meg per second. It appears to be seek bound as I can hear the disk seeking madly. There appears to be a effect from the memory size since a 256 Meg server can handle 2 clients, but goes seek bound with 4 clients. However, 4 clients are only writing files with a total size of 120 Meg so it should be able to cache it all anyway.
I decided to check it was disk seeks that were the performance limitation. So I ran the test with 2 clients writing to separate disks. The performance is OK even with as little as 32 Megs of memory. So there are no bottlenecks in the network code or the buffer cache etc.
The mildly odd thing is that performance does seem to be effected by the fact that its a networked test. The file serving gets handled by a separate process for each client. I can strace these processes and see that they are just doing an 8k read from the network followed by an 8k write to disk. If I use the IOzone throughput test with multiple processes doing 8k reads and writes of 30 Meg files, I don't see the same sort of performance problems.
My current theory is that waiting for the network read is causing a context switch to the next process hence interleaving the writes from all the processes. Under IOzone, each process is probably doing multiple 8k writes in its timeslice.
Since the benchmark is going seek bound, it sounds like the elevator algorithm isn't doing its job. The 2.2 kernel is supposed to have an elevator, it just isn't guaranteed to be fair. So we should still see high total throughput, just not evenly spread. But even total throughput is very low; rarely more than 150 blocks in 5 seconds according to vmstat. Less than 1 Meg per second aggregate according to the clients. So this is a problem for the 2.2 kernels. I tested a 2.0 kernel as well and saw the same problem there, so this isn't new for 2.2.
Anyway, the 2.4 kernels are supposed to have a funky new elevator. So I decided to try some 2.4 kernels. I have tried 2.4.0-test4 and 2.4.0-test5pre2, 2.4.0-test1-ac22-class and 2.4.0-test1-ac22-riel.
Kernels test1-ac22-class and test1-acc22-riel perform quite well even with 5 clients on a 128 Meg server. The new elevator really helps with write performance. The blocks written per 5 secs result from output is over 1000. Aggregate client output is over 10 Megs per second. However there is still some tendency towards client starvation. Often one client will drop to less than 300 K per second for around a minute.
The more recent test4 and test5pre2 don't fair quite so well. They handle 2 clients on a 128 Meg server fine, so they're doing better than 2.2 but they choke and go seek bound with 4 clients. So something has definitely taken a turn for the worse since test1-ac22.
I was hoping to do some experimentation with elvtune and see how much effect various elevator settings have on the performance. However, I have been unable to get elvtune to run with any of the 2.4 kernels. The ioctl which it uses gives an IO error. If anyone has suggestions as to why elvtune isnt working, please let me know.
-- Robert Cohen Unix Support TLTSU - Australian National University
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |