Messages in this thread | | | Date | Sun, 18 Jul 2010 11:52:48 -0700 | Subject | Re: Interrupt Affinity in SMP | From | Bryan Hundven <> |
| |
On Sun, Jul 18, 2010 at 11:38 AM, Ciju Rajan K <ciju@linux.vnet.ibm.com> wrote: > Bryan Hundven wrote: >> >> On Sat, Jul 10, 2010 at 6:20 PM, Robert Hancock <hancockrwd@gmail.com> >> wrote: >> >>> >>> On Sat, Jul 10, 2010 at 1:46 PM, Bryan Hundven <bryanhundven@gmail.com> >>> wrote: >>> >>>> >>>> I was able to set eth0 and it's TxRx queues to cpu1, but it is my >>>> understanding that 0xFFFFFFFF should distribute the interrupts across >>>> all >>>> cpus, much like LOC in my output of /proc/interrupts. >>>> >>>> I don't have access to the computer this weekend, but I will provide >>>> more >>>> info on Monday. >>>> >>> >>> That may be chipset dependent, I don't think all chipsets have the >>> ability to distribute the interrupts like that. Round-robin interrupt >>> distribution for a given handler isn't optimal for performance anyway >>> since it causes the relevant cache lines for the interrupt handler to >>> be ping-ponged between the different CPUs. >>> >>> >>>> >>>> -bryan >>>> >>>> On Jul 9, 2010 5:48 PM, "Robert Hancock" <hancockrwd@gmail.com> wrote: >>>> >>>> On 07/09/2010 04:59 PM, Bryan Hundven wrote: >>>> >>>>> >>>>> Mauro, list, >>>>> >>>>> (please CC me in replies, I am not... >>>>> >>>> >>>> Tried changing these files to exclude CPU0? >>>> >>>> Have you tried running the irqbalance daemon? That's what you likely >>>> want to >>>> be doing anyway.. >>>> >>>> >>>>> >>>>> =====8<=====8<=====8<=====8<=====8<=====8<=====8<=====8<=====8<===== >>>>> >>>>> =====8<=====8<=====8<==... >>>>> >> >> Please see the two attached examples. >> >> Notice on the 5410 example how we start with the affinity set to 0xff, >> and change it to 0xf0. >> This should spread the interrupts over the last 4 cores of this quad >> core - dual processor system. >> >> Also notice on the 5645 example, with the same commands we start with >> 0xffffff and change to 0xfff000 to spread the interrupts over the last >> 12 cores, but only the first of the last twelve cores receive >> interrupts. >> >> This is the inconsistency I was trying to explain before. >> > > What was the status of irqbalance daemon? Was it turned on? If it is > running, there is a chance that the interrupt count is within the threshold > limit and interrupts are not being routed to the other core.
irqbalance daemon was not running on either setup.
> Could you also try with increasing the interrupt load and see if the > distribution is happening among the cores?
We use spirent testcenter l2/l3 test equipment and pushed 100% throughput with the same distribution. Nothing changed.
This isn't affecting just ethernet drivers. I have also seen the same issues with hardware encryption devices and other hardware that gets a software interrupt.
--Bryan
> > -Ciju >> >> --Bryan >> > >
-- Bryan Hundven bryanhundven@gmail.com
| |