Messages in this thread | | | Date | Fri, 3 Jul 2009 13:45:57 -0600 | From | Matthew Wilcox <> | Subject | Re: >10% performance degradation since 2.6.18 |
| |
On Fri, Jul 03, 2009 at 09:22:35PM +0200, Jens Axboe wrote: > On Fri, Jul 03 2009, Matthew Wilcox wrote: > > Yes, but the irqs/sec increase doesn't appear to be due to MPT interrupts. > > In the /proc/interrupt summaries, RH5 did 388666895 IOC interrupts and > > 2.6.30 did 378419042. As a percentage of interrupts, the IOC interrupts > > were 59.4% with RH and 51.8% with 2.6.30. > > OK. So where are the extra irqs from?
Let's see:
Source 2.6.18 2.6.30 Delta qla 0.8% 0.8% 0 eth 20% 27.6% +7.6% ioc 59.4% 51.8% -7.6% NMI 7.6% 7.9% +0.3% LOC 12.2% 10% -2.2% RES - 1.8% +1.8%
I wouldn't be surprised to find out that 2.6.18 accounted rescheduling interrupts as 'LOC'. So the difference in interrupts is all about the ethernet card. I believe these systems have an igb card.
The big difference between 2.6.18 and 2.6.30 is that the cards now have eight interrupts in use each, instead of one each (four for rx queues and four for tx queues). Distressingly, these interrupts are all affine to the same CPUs (eth1's eight interrupts are all on CPU 9 and eth0's interrupts are all on CPU 1). That would seem to be a fruitful avenue of investigation -- whether limiting the cards to a single RX/TX interrupt would be advantageous, or whether spreading the eight interrupts out over the CPUs would be advantageous.
-- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
| |