Messages in this thread | | | Date | Tue, 06 Jul 1999 14:11:54 -0700 | From | Fred Polizo <> | Subject | 2.2.5 IP forwarding: 2 cpus worse than one? |
| |
I've been doing some basic network load testing with 2.2.5 using a dual PII/400 box. We found a case where a SMP kernel performs worse than a uniprocessor (UP) kernel on the same hardware. I checked the DejaNews archives and couldn't find any mention of this particular problem. So,
I thought I'd see if anyone else has seen such a case or could offer a remedy. ;)
I saw that there are fixes in the 2.3.4+ code branch for a network bottom half scalability issue. But, in that case, I couldn't find any mention of performance being worse with MP than with UP. So, this may or may not be the same problem. Informed comments would be greatly appreciated.
TEST SUMMARY: The dual cpu machine was configured with two NICs, each on a different isolated network. IP forwarding was enabled. Using a standalone packet-generator/analyzer box, we sent a continuous series of identical UDP packets on one network with a destination address on the other network. We used UDP port 9 (discard) so that all packets were discarded by the destination host. So, all traffic was in one direction. The analyzer box has two NICs. One is used to send packets. The other listens (in promiscuous mode) for the forwarded packets on the destination network. Packet loss statistics are displayed on the analyzer.
RESULTS: Booting a UP kernel, we were able to reach a steady state where 45000 (67 byte) packets/sec could be forwarded without packet loss. However,
when we switched to an SMP kernel, we could only reach ~22500 (67 byte) packets/sec. Any more than that and we started losing packets on the destination net.
CPU usage from "top" and "vmstat" indicated that at 45000 pps, the UP kernel was using 94% system time and 6% remained idle (see note 1 below). At 22500 pps, the SMP kernel was using ~59% (aggregated) system time with 41% idle.
I was able to reach 26000 pps with the SMP kernel after binding both network IRQs to the same cpu (see note 2 below). That reduced cpu usage to ~50% system with ~50% idle. Could this this be the same or different
lock granularity problem fixed in 2.3.4+?
We were hoping to stick with a stable kernel as that is what we'll need to use for deployment. But, if someone is pretty sure this is fixed in a "stable enough" 2.3.x kernel, then we might be able to try that.
NOTES: 1) Unless there is a runnable user process, interrupt handling is not accounted for as system time. The system can be 100%busy handling interrupts, yet appear idle when looking at "top", "vmstat", etc. So, I
wrote a cpu consumer program "while (1) {;}" and ran it "nice -20 ./loop". That way, interrupt handling activity appears as system time and "free" cpu appears as "nice" time in the cpu usage statistics. Crude, but it seems to work.
2) I modified io_apic.c to bind the two network interrupts to the same cpu and then to different cpus. Got some "interesting" results in each case, but it didn't fix the poor MP performance problem.
3) Here are the configuration details of the System Under Test: Motherboard: Intel N440BX dual-Slot 1 Chipset: Intel 440BX Integrated video: Cirrus Logic GD5480 CPU: 2 x Intel Pentium II/400 NIC1: Integrated Intel EtherExpress Pro 10/100 NIC2: Intel EtherExpress Pro 10/100 (PCI card) Linux Distribution: RedHat 6.0 Kernels: generic kernels off CD (2.2.5-15 and 2.2.5-15smp). We also built various stripped down kernels, but they didn't fix the poor MP
routing performance problem.
TIA, ---Fred P.
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |