lkml.org 
[lkml]   [2008]   [Nov]   [20]   [last100]   RSS Feed
Views: [more markup]  [less markup]  [headers]  [forward] 
 
Messages in this thread
/
DateThu, 20 Nov 2008 20:48:42 +0100 (CET)
FromJesper Dangaard Brouer <>
SubjectRegression: Bisected, IRQ and MSI allocations screwed without sparse irq
Hi Thomas Gleixner,

I have bisected a regression to your commit
3235e936c0cc3589309280b6f59e5096779adae3,
"x86: remove sparse irq from Kconfig".

Its actually not necessary your fault, as your commit simply removes
the config option HAVE_SPARSE_IRQ.  This revels the bug / regression
I'm exposted to.

Guess I should bisect again to find the exact faulty commit, but I'm
rather sick of bisecting at the moment, and though you might have a
better idea whats going wrong.  I would rather spend my time
performance tuning the multiqueue routing code...

[The regression]:

During my testing of the Sun Neptune based NICs.  On kernel 2.6.27 I
get really good performance (900-1200kpps) compared to 2.6.28 (davem
git net-2.6).

The cause of this problem (tracked down together with Robert Olsson)
is that on 2.6.28 I have a lot less IRQs available.  It seems max 34
IRQs.  Due the reduced number of IRQs the NIU driver cannot get
enough IRQs to the interfaces, and starts to use "IO-APIC" based
IRQs.

On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge".  BUT
my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with
the usb driver.  That my performance problem on 2.6.28.

[Other related bugs]:
  Is that unloading the "niu" driver will give a kernel BUG during
  deallocation og MSI interrupts. (See dmesg output below if interested)

(I have attached full bisect history)

Cheers,
   Jesper Brouer

--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------

On Wed, 19 Nov 2008, David Miller wrote:
> From: Jesper Dangaard Brouer <hawk@diku.dk>
> Date: Wed, 19 Nov 2008 23:58:12 +0100 (CET)
>
>> Well that was not the real cause of the performance loss.  Because
>> on kernel 2.6.27 I get really good performance (900-1200kpps)
>> compared to 2.6.28 (git net-2.6).
>>
>> The cause of this problem (tracked down together with Robert Olsson)
>> is that on 2.6.28 I have a lot less IRQs available.  It seems max 34
>> IRQs.
>>
>> Due the reduced number of IRQs the NIU driver cannot get enough IRQs
>> to the interfaces, and starts to use "IO-APIC" based IRQs.
>
> This is almost certainly related to the driver unload bug.
>
> I know you ran into unbuildable/unbootable kernels during a bisect,
> but you really need to track down this regression.


------------[ cut here ]------------
kernel BUG at drivers/pci/msi.c:632!
invalid opcode: 0000 [#1] PREEMPT SMP
Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo 
niu(-)
Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5
EIP: 0060:[<c021ecac>] EFLAGS: 00010286 CPU: 2
EIP is at msi_free_irqs+0xdc/0xe0
EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500
ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8
  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000)
Stack:
  f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000
  f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858
  f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4
Call Trace:
  [<c021ecb8>] ? msix_free_all_irqs+0x8/0x10
  [<c021ef41>] ? pci_disable_msix+0x31/0x40
  [<f809d3f8>] ? niu_pci_remove_one+0x88/0x8a [niu]
  [<c0219c29>] ? pci_device_remove+0x19/0x40
  [<c026b0d4>] ? __device_release_driver+0x54/0x80
  [<c026b197>] ? driver_detach+0x97/0xa0
  [<c026a475>] ? bus_remove_driver+0x75/0xa0
  [<c026b609>] ? driver_unregister+0x39/0x40
  [<c0219e51>] ? pci_unregister_driver+0x21/0x80
  [<f809a0ad>] ? niu_exit+0xd/0x10 [niu]
  [<c0145d74>] ? sys_delete_module+0x114/0x1d0
  [<c016810a>] ? remove_vma+0x3a/0x50
  [<c0168c29>] ? do_munmap+0x189/0x1e0
  [<c0103229>] ? sysenter_do_call+0x12/0x21
  [<c0330000>] ? quirk_disable_msi+0x30/0x50
Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b 
14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe 
55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55
EIP: [<c021ecac>] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8
---[ end trace f72de2e283920207 ]---~~ -*-text-*-


       -------------------------------------------------------
			 Bisecting IRQ change:
		      What change reduced the IRQs
       -------------------------------------------------------
		 Jesper Dangaard Brouer (jdb@comx.dk)
       -------------------------------------------------------
	$LastChangedRevision: 786 $
	$Date: 2008-11-20 20:44:51 +0100 (Thu, 20 Nov 2008) $
       -------------------------------------------------------

git clone

~~~~~~~~~

+---------
 cd /var/kernels/git/davem
 git clone git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git net-2.6-bisect-irqs
+---------

Description / Reason to find

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 During my testing of the Sun Neptune based NICs.


 On kernel 2.6.27 I get really good performance (900-1200kpps)
 compared to 2.6.28 (git net-2.6).


 The cause of this problem (tracked down together with Robert Olsson)
 is that on 2.6.28 I have a lot less IRQs available.  It seems max 34
 IRQs.


 Due the reduced number of IRQs the NIU driver cannot get enough IRQs
 to the interfaces, and starts to use "IO-APIC" based IRQs.

 On kernel 2.6.28:


  My eth2 is using 10 IRQs all "PCI-MSI-edge".

  BUT my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared
  with the usb driver...


 Think thats must be my performance problem on 2.6.28.

Known: Good and bad

~~~~~~~~~~~~~~~~~~~

 GOOD:

  git bisect good v2.6.27


 BAD:

  git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43

 [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] #Merge branch
 'tracing-v28-for-linus' of

 git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip


HiSTORY:

~~~~~~~~

+--------
 cd /var/kernels/git/davem/net-2.6-bisect-irqs/
 git bisect start
 git bisect good v2.6.27
+--------

+--------------
git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
Bisecting: 3220 revisions left to test after this
[af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2
+--------------

 CONFIG_LOCALVERSION="-bisect"

+-------------
cp ../net-2.6-bisect/.config .
script make_oldconfig_01
make oldconfig
exit
#Script done, file is make_oldconfig_01
+-------------

+----------------
time make -j6 bzImage modules
#
#real    9m22.739s
#user    16m56.776s
#sys     1m4.672s
+----------------

 Booted kernel: GOOD: irqs and (niu rmmod good)


+----------------
git bisect good
Bisecting: 1614 revisions left to test after this
[36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
+----------------

 Compiling:


+----------------
time make -j6 bzImage modules
+----------------

 Booted kernel: GOOD: irqs and (niu rmmod good)


+-----------
git bisect good
Bisecting: 807 revisions left to test after this
[1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen()
+-----------

 Compiling...


+----------------
time make -j6 bzImage modules
#real    10m1.561s
#user    17m23.293s
#sys     1m5.744s
+----------------

 Installing...


 Booted kernel:


+----
dcu-router-ng:~# uname -a
Linux dcu-router-ng 2.6.27-bisect #3 SMP PREEMPT Thu Nov 20 12:33:02 CET 2008 i686 GNU/Linux
+----

 Results: GOOD: irqs and (niu rmmod good)


+------
git bisect good
Bisecting: 403 revisions left to test after this
[1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
+------

 Compiling...


+----------------
time make -j6 bzImage modules
#real    10m9.371s
#user    17m21.781s
#sys     1m6.052s
+----------------

 Installing...


 Booting ...


+-------
dcu-router-ng:~# uname -a
Linux dcu-router-ng 2.6.27-bisect #4 SMP PREEMPT Thu Nov 20 12:50:39 CET 2008 i686 GNU/Linux
+-------

 Results: GOOD: irqs and (niu rmmod good)



+-------
 git-bisect good
Bisecting: 223 revisions left to test after this
[dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files
+-------

+----------------
time make -j6 bzImage modules
+----------------

 Booting ...


+--------
Linux dcu-router-ng 2.6.27-bisect #5 SMP PREEMPT Thu Nov 20 13:58:34 CET 2008 i686 GNU/Linux
+--------

 Results: BAD: irqs and (niu rmmod also BAD)


+-------
cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       
  0:        125          0          0          0   IO-APIC-edge      timer
  1:          0          0          1          1   IO-APIC-edge      i8042
  3:          2          1          2          2   IO-APIC-edge      serial
  8:          0          2          0          0   IO-APIC-edge      rtc
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          1          2          1          0   IO-APIC-edge      i8042
 16:        103        108        108        112   IO-APIC-fasteoi   uhci_hcd:usb1, ehci_hcd:usb6, eth0
 17:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
 18:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 19:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4, eth3
 20:          0          0          0          0   PCI-MSI-edge      eth2
 21:          0          0          0          0   PCI-MSI-edge      eth2
 22:         24         23         23         23   IO-APIC-fasteoi   uhci_hcd:usb5, eth2
 23:          0          0          0          0   PCI-MSI-edge      eth2
 24:          0          0          0          0   PCI-MSI-edge      eth2
 25:          0          0          0          0   PCI-MSI-edge      eth2
 26:          0          0          0          0   PCI-MSI-edge      eth2
 27:          0          0          0          0   PCI-MSI-edge      eth2
 28:          0          0          0          0   PCI-MSI-edge      eth2
 29:          0          0          0          0   PCI-MSI-edge      eth2
 30:          0          0          0          0   PCI-MSI-edge      eth2
 31:          0          0          0          0   PCI-MSI-edge      eth2
 32:          0          0          0          0   PCI-MSI-edge      eth2
 34:        271        268        268        264   PCI-MSI-edge      cciss0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:       3301       2970       2594       2389   Local timer interrupts
RES:         28        560          6         13   Rescheduling interrupts
CAL:         50        104         99         62   Function call interrupts
TLB:        241        224        287        279   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0
+-------

 OUTPUT "rmmod niu" (gives segfault) and "dmesg"

+-------
------------[ cut here ]------------
kernel BUG at drivers/pci/msi.c:632!
invalid opcode: 0000 [#1] PREEMPT SMP 
Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo niu(-)

Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5
EIP: 0060:[<c021ecac>] EFLAGS: 00010286 CPU: 2
EIP is at msi_free_irqs+0xdc/0xe0

EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500
ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000)
Stack:
 f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000
 f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858
 f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4
Call Trace:
 [<c021ecb8>] ? msix_free_all_irqs+0x8/0x10
 [<c021ef41>] ? pci_disable_msix+0x31/0x40
 [<f809d3f8>] ? niu_pci_remove_one+0x88/0x8a [niu]
 [<c0219c29>] ? pci_device_remove+0x19/0x40
 [<c026b0d4>] ? __device_release_driver+0x54/0x80
 [<c026b197>] ? driver_detach+0x97/0xa0
 [<c026a475>] ? bus_remove_driver+0x75/0xa0
 [<c026b609>] ? driver_unregister+0x39/0x40
 [<c0219e51>] ? pci_unregister_driver+0x21/0x80
 [<f809a0ad>] ? niu_exit+0xd/0x10 [niu]
 [<c0145d74>] ? sys_delete_module+0x114/0x1d0
 [<c016810a>] ? remove_vma+0x3a/0x50
 [<c0168c29>] ? do_munmap+0x189/0x1e0
 [<c0103229>] ? sysenter_do_call+0x12/0x21
 [<c0330000>] ? quirk_disable_msi+0x30/0x50
Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b 14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe 55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55 
EIP: [<c021ecac>] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8
---[ end trace f72de2e283920207 ]---
+-------

+------
git-bisect bad
Bisecting: 89 revisions left to test after this
[db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function
+------

 Related config change? (make oldconfig)


+------
 script make_oldconfig_02
 make oldconfig
 Script done, file is make_oldconfig_02
+------

+------
Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) ? ?Y

This enables support for sparse irq, esp for msi/msi-x. the irq
number will be bus/dev/fn + 12bit. You may need if you have lots of
cards supports msi-x installed.


If you don't know what to do here, say Y.

+------

 Compiling...


+----------------
 time make -j6 bzImage modules
#
#real    9m29.556s
#user    17m10.396s
#sys     1m5.056s
+----------------

 Booting ...


+-------
Linux dcu-router-ng 2.6.27-bisect #6 SMP PREEMPT Thu Nov 20 14:25:40 CET 2008 i686 GNU/Linux
+-------

 The output from /proc/interrupts changed, very weird!
 BUT eth3 does use a "PCI-MSI-edge" interrupt.

 Guess this is a GOOD state even though it looks weird.

 Unloading NIU driver also GOOD.


+---------
cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
0x0:        124          1          0          0   IO-APIC-edge      timer
0x1:          1          0          0          1   IO-APIC-edge      i8042
0x3:          2          1          2          2   IO-APIC-edge      serial
0x8:          1          0          0          1   IO-APIC-edge      rtc
0x9:          0          0          0          0   IO-APIC-fasteoi   acpi
0xc:          0          1          2          1   IO-APIC-edge      i8042
0x10:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb1, ehci_hcd:usb6
0x11:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
0x12:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
0x6000fe:        288        289        290        290   PCI-MSI-edge      cciss0
0x16:         23         24         23         23   IO-APIC-fasteoi   uhci_hcd:usb5
0xb00100:          0          0          0          0   PCI-MSI-edge      eth2
0xb000ff:          0          0          0          0   PCI-MSI-edge      eth2
0xb000fe:          0          0          0          0   PCI-MSI-edge      eth2
0xb000fd:          0          0          0          0   PCI-MSI-edge      eth2
0xb000fc:          0          0          0          0   PCI-MSI-edge      eth2
0xb000fb:          0          0          0          0   PCI-MSI-edge      eth2
0xb000fa:          0          0          0          0   PCI-MSI-edge      eth2
0xb000f9:          0          0          0          0   PCI-MSI-edge      eth2
0xb000f8:          0          0          0          0   PCI-MSI-edge      eth2
0xb000f7:          0          0          0          0   PCI-MSI-edge      eth2
0xb000f6:          0          0          0          0   PCI-MSI-edge      eth2
0xb000f5:          0          0          0          0   PCI-MSI-edge      eth2
0xb000f4:          0          0          0          0   PCI-MSI-edge      eth2
0xb01100:          0          0          0          0   PCI-MSI-edge      eth3
0xb010ff:          0          0          0          0   PCI-MSI-edge      eth3
0xb010fe:          0          0          0          0   PCI-MSI-edge      eth3
0xb010fd:          0          0          0          0   PCI-MSI-edge      eth3
0xb010fc:          0          0          0          0   PCI-MSI-edge      eth3
0xb010fb:          0          0          0          0   PCI-MSI-edge      eth3
0xb010fa:          0          0          0          0   PCI-MSI-edge      eth3
0xb010f9:          0          0          0          0   PCI-MSI-edge      eth3
0xb010f8:          0          0          0          0   PCI-MSI-edge      eth3
0xb010f7:          0          0          0          0   PCI-MSI-edge      eth3
0xb010f6:          0          0          0          0   PCI-MSI-edge      eth3
0x13:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
0x300100:        210        210        210        208   PCI-MSI-edge      eth0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:       3630       3265       3103       2711   Local timer interrupts
RES:         34        226         12        417   Rescheduling interrupts
CAL:         89         55         90         78   Function call interrupts
TLB:        253        205        311        267   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0
+---------

 Guess it a GOOD situation...


+------
 git-bisect good
Bisecting: 44 revisions left to test after this
[ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI
+------

 Compiling ...


+--------
 time make -j6 bzImage modules
#real    9m28.062s
#user    17m7.492s
#sys     1m4.248s
+--------

 Installing ...


 Booting ...


+------
Linux dcu-router-ng 2.6.27-bisect #7 SMP PREEMPT Thu Nov 20 14:52:45 CET 2008 i686 GNU/Linux
+------

 Still looks GOOD (/proc/interrupts still looks weird).

 And rmmod NIU driver GOOD.


+------
git-bisect good
Bisecting: 22 revisions left to test after this
[922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4
+------

 Compiling ...


+--------
 time make -j6 bzImage modules
#real    0m34.622s
#user    0m41.139s
#sys     0m5.812s
+--------

 Install ...


 Booting ...


+-----
Linux dcu-router-ng 2.6.27-bisect #8 SMP PREEMPT Thu Nov 20 15:04:11 CET 2008 i686 GNU/Linux
+-----

 Looks GOOD, and /proc/interrupts changed again! Now the interrupts
 are not i HEX anymore, but in decimal, but still strange/large
 numbers for MSI.


 Unloading NIU driver GOOD.


+------
 cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
  0:        124          0          0          0   IO-APIC-edge      timer
  1:          0          0          1          1   IO-APIC-edge      i8042
  3:          2          2          1          2   IO-APIC-edge      serial
  8:          0          0          1          1   IO-APIC-edge      rtc
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          1          2          1          0   IO-APIC-edge      i8042
 16:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb1, ehci_hcd:usb6
 17:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
 18:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
6291710:        828        821        828        823   PCI-MSI-edge      cciss0
 22:         23         24         24         22   IO-APIC-fasteoi   uhci_hcd:usb5
11534592:          0          0          0          0   PCI-MSI-edge      eth2
11534591:          0          0          0          0   PCI-MSI-edge      eth2
11534590:          0          0          0          0   PCI-MSI-edge      eth2
11534589:          0          0          0          0   PCI-MSI-edge      eth2
11534588:          0          0          0          0   PCI-MSI-edge      eth2
11534587:          0          0          0          0   PCI-MSI-edge      eth2
11534586:          0          0          0          0   PCI-MSI-edge      eth2
11534585:          0          0          0          0   PCI-MSI-edge      eth2
11534584:          0          0          0          0   PCI-MSI-edge      eth2
11534583:          0          0          0          0   PCI-MSI-edge      eth2
11534582:          0          0          0          0   PCI-MSI-edge      eth2
11534581:          0          0          0          0   PCI-MSI-edge      eth2
11534580:          0          0          0          0   PCI-MSI-edge      eth2
11538688:          0          0          0          0   PCI-MSI-edge      eth3
11538687:          0          0          0          0   PCI-MSI-edge      eth3
11538686:          0          0          0          0   PCI-MSI-edge      eth3
11538685:          0          0          0          0   PCI-MSI-edge      eth3
11538684:          0          0          0          0   PCI-MSI-edge      eth3
11538683:          0          0          0          0   PCI-MSI-edge      eth3
11538682:          0          0          0          0   PCI-MSI-edge      eth3
11538681:          0          0          0          0   PCI-MSI-edge      eth3
11538680:          0          0          0          0   PCI-MSI-edge      eth3
11538679:          0          0          0          0   PCI-MSI-edge      eth3
11538678:          0          0          0          0   PCI-MSI-edge      eth3
 19:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
3145984:       9993       9994       9987       9993   PCI-MSI-edge      eth0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      10075      11448       8001       8787   Local timer interrupts
RES:        297         17        349         26   Rescheduling interrupts
CAL:        173        189         95        173   Function call interrupts
TLB:        299        259        330        345   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0
+------

+-------
 git-bisect good
Bisecting: 11 revisions left to test after this
[a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal
+-------

 Compiling


+--------
 time make -j6 bzImage modules
#real    9m30.767s
#user    17m10.808s
#sys     1m6.388s
+--------

 Installing ...


 Booting ...


+-----
Linux dcu-router-ng 2.6.27-bisect #9 SMP PREEMPT Thu Nov 20 15:28:17 CET 2008 i686 GNU/Linux
+-----

 BAD kernel version, max IRQ is 34.  And eth3 got assigned a
 IO-APIC-fasteoi shared with uhci_hcd:usb2.


 Also BAD unloading of NIU driver.


 BUG is some where in between:


  git log 922402f15a85f7a064926eb1db68cc52bc4d4a91..a1aca5de08a0cb840a90fb3f729a5940f8d21185 | grep ^commit | wc -l

 11 commits



+-------
git-bisect bad
Bisecting: 5 revisions left to test after this
[3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig
+-------

 Compiling...


+--------
 time make -j6 bzImage modules
+--------

 Install ...


 Booting


+------
Linux dcu-router-ng 2.6.27-bisect #10 SMP PREEMPT Thu Nov 20 15:56:10 CET 2008 i686 GNU/Linux
+------

 BAD kernel.


 BAD rmmod NIU driver.


+---------
git bisect bad
Bisecting: 2 revisions left to test after this
[4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out
+---------

 Compiling...


+------
 time make -j6 bzImage modules
#real    7m23.814s
#user    12m15.718s
#sys     0m42.183s
+------

 Config change prompting:


+-----
 Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) Y
+-----

 Installing ...


 Booting ...


+-------
Linux dcu-router-ng 2.6.27-bisect #11 SMP PREEMPT Thu Nov 20 16:19:29 CET 2008 i686 GNU/Linux
+-------

 GOOD!!!


+--------
 cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
  0:        124          0          0          0   IO-APIC-edge      timer
  1:          0          0          1          1   IO-APIC-edge      i8042
  3:          2          2          2          2   IO-APIC-edge      serial
  8:          1          0          0          1   IO-APIC-edge      rtc
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          1          2          1          0   IO-APIC-edge      i8042
 16:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb1, ehci_hcd:usb6
 17:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
 18:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
6291710:        285        288        293        287   PCI-MSI-edge      cciss0
11534592:          0          0          0          0   PCI-MSI-edge      eth2
11534591:          0          0          0          0   PCI-MSI-edge      eth2
11534590:          0          0          0          0   PCI-MSI-edge      eth2
11534589:          0          0          0          0   PCI-MSI-edge      eth2
11534588:          0          0          0          0   PCI-MSI-edge      eth2
11534587:          0          0          0          0   PCI-MSI-edge      eth2
11534586:          0          0          0          0   PCI-MSI-edge      eth2
11534585:          0          0          0          0   PCI-MSI-edge      eth2
11534584:          0          0          0          0   PCI-MSI-edge      eth2
11534583:          0          0          0          0   PCI-MSI-edge      eth2
11534582:          0          0          0          0   PCI-MSI-edge      eth2
11534581:          0          0          0          0   PCI-MSI-edge      eth2
11534580:          0          0          0          0   PCI-MSI-edge      eth2
 22:         23         24         23         23   IO-APIC-fasteoi   uhci_hcd:usb5
11538688:          0          0          0          0   PCI-MSI-edge      eth3
11538687:          0          0          0          0   PCI-MSI-edge      eth3
11538686:          0          0          0          0   PCI-MSI-edge      eth3
11538685:          0          0          0          0   PCI-MSI-edge      eth3
11538684:          0          0          0          0   PCI-MSI-edge      eth3
11538683:          0          0          0          0   PCI-MSI-edge      eth3
11538682:          0          0          0          0   PCI-MSI-edge      eth3
11538681:          0          0          0          0   PCI-MSI-edge      eth3
11538680:          0          0          0          0   PCI-MSI-edge      eth3
11538679:          0          0          0          0   PCI-MSI-edge      eth3
11538678:          0          0          0          0   PCI-MSI-edge      eth3
 19:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
3145984:        244        242        238        241   PCI-MSI-edge      eth0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:       3715       3104       2853       2542   Local timer interrupts
RES:         88         52        280        258   Rescheduling interrupts
CAL:         76         75         93         59   Function call interrupts
TLB:        245        241        312        283   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0
+---------


+---------
 git-bisect good
Bisecting: 1 revisions left to test after this
[7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n
+----------

 Compiling ...


+------
 time make -j6 bzImage modules
+------

 Install...


 Boot ...


+-------
Linux dcu-router-ng 2.6.27-bisect #12 SMP PREEMPT Thu Nov 20 16:33:11 CET 2008 i686 GNU/Linux
+-------

+--------
 cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
  0:        124          0          0          0   IO-APIC-edge      timer
  1:          0          0          1          1   IO-APIC-edge      i8042
  3:          1          2          2          2   IO-APIC-edge      serial
  8:          2          0          0          0   IO-APIC-edge      rtc
  9:          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          1          2          1          0   IO-APIC-edge      i8042
 16:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb1, ehci_hcd:usb6
 17:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
 18:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
6291710:        268        269        268        267   PCI-MSI-edge      cciss0
11534592:          0          0          0          0   PCI-MSI-edge      eth2
11534591:          0          0          0          0   PCI-MSI-edge      eth2
11534590:          0          0          0          0   PCI-MSI-edge      eth2
11534589:          0          0          0          0   PCI-MSI-edge      eth2
11534588:          0          0          0          0   PCI-MSI-edge      eth2
11534587:          0          0          0          0   PCI-MSI-edge      eth2
11534586:          0          0          0          0   PCI-MSI-edge      eth2
11534585:          0          0          0          0   PCI-MSI-edge      eth2
11534584:          0          0          0          0   PCI-MSI-edge      eth2
11534583:          0          0          0          0   PCI-MSI-edge      eth2
11534582:          0          0          0          0   PCI-MSI-edge      eth2
11534581:          0          0          0          0   PCI-MSI-edge      eth2
11534580:          0          0          0          0   PCI-MSI-edge      eth2
11538688:          0          0          0          0   PCI-MSI-edge      eth3
11538687:          0          0          0          0   PCI-MSI-edge      eth3
11538686:          0          0          0          0   PCI-MSI-edge      eth3
11538685:          0          0          0          0   PCI-MSI-edge      eth3
11538684:          0          0          0          0   PCI-MSI-edge      eth3
11538683:          0          0          0          0   PCI-MSI-edge      eth3
11538682:          0          0          0          0   PCI-MSI-edge      eth3
11538681:          0          0          0          0   PCI-MSI-edge      eth3
11538680:          0          0          0          0   PCI-MSI-edge      eth3
11538679:          0          0          0          0   PCI-MSI-edge      eth3
11538678:          0          0          0          0   PCI-MSI-edge      eth3
 19:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
 22:         25         23         24         25   IO-APIC-fasteoi   uhci_hcd:usb5
3145984:        175        174        176        178   PCI-MSI-edge      eth0
NMI:          0          0          0          0   Non-maskable interrupts
LOC:       3508       2902       2765       2489   Local timer interrupts
RES:        238         35        461          6   Rescheduling interrupts
CAL:         61         90         59         81   Function call interrupts
TLB:        257        220        299        300   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
SPU:          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0
+--------

 GOOD.


+----------
git-bisect good
3235e936c0cc3589309280b6f59e5096779adae3 is first bad commit
commit 3235e936c0cc3589309280b6f59e5096779adae3
Author: Thomas Gleixner <tglx@linutronix.de>
Date:   Wed Oct 15 13:16:00 2008 +0200

    x86: remove sparse irq from Kconfig


    This code is not ready yet.


    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

:040000 040000 6043e32465556e828de0fbb6aa497b277239af01 2dd75ba207990d83a3a4c7b7b16abccfe2d5e10d M    arch

+--------


 Found bad commit:

   3235e936c0cc3589309280b6f59e5096779adae3




Git bisect LOG

~~~~~~~~~~~~~~

+-------
git-bisect log
git-bisect start
# good: [3fa8749e584b55f1180411ab1b51117190bac1e5] Linux 2.6.27
git-bisect good 3fa8749e584b55f1180411ab1b51117190bac1e5
# bad: [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
# good: [af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2
git-bisect good af5c2bd16ac2e5688c3bf46ea1f95112d696d294
# good: [36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git-bisect good 36ac1d2f323f8bf8bc10c25b88f617657720e241
# good: [1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen()
git-bisect good 1aece34833721d64eb33fc15cd923c727296d3d3
# good: [1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
git-bisect good 1d9a8a47d659f053abeca9ece45651b4d94780c8
# bad: [dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files
git-bisect bad dd3a1db900f2a215a7d7dd71b836e149a6cf5fed
# good: [db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function
git-bisect good db4b5525caafd846ec20f95afbc6403c792e22cf
# good: [ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI
git-bisect good ba374c9baef910fbc5373541d98c50f15e82c3f8
# good: [922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4
git-bisect good 922402f15a85f7a064926eb1db68cc52bc4d4a91
# bad: [a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal
git-bisect bad a1aca5de08a0cb840a90fb3f729a5940f8d21185
# bad: [3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig
git-bisect bad 3235e936c0cc3589309280b6f59e5096779adae3
# good: [4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out
git-bisect good 4c66a73f0796dacc2ff0d4af75794ec843ceb3d1
# good: [7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n
git-bisect good 7ef0c30dbf96a8d9a234e90c248eb19df3c031be
+-------

Email

~~~~~

 To:

  Thomas Gleixner <tglx@linutronix.de>
  David Miller <davem@davemloft.net>, 
  Jesper Dangaard Brouer <jdb@comx.dk>, 
  netdev <netdev@vger.kernel.org>, 
  linux-kernel@vger.kernel.org, 
  Robert Olsson <Robert.Olsson@data.slu.se>

 Subj.:

  Regression: Bisected, IRQ and MSI allocations screwed without sparse irq


 Hi Thomas Gleixner,


 I have bisected a regression to your commit

 3235e936c0cc3589309280b6f59e5096779adae3, "x86: remove sparse irq
 from Kconfig".


 Its actually not necessary your fault, as your commit simply removes
 the config option HAVE_SPARSE_IRQ.  This revels the bug / regression
 I'm exposted to.


 Guess I should bisect again to find the exact faulty commit, but I'm
 rather sick of bisecting at the moment, and though you might have a
 better idea whats going wrong.  I would rather spend my time
 performance tuning the multiqueue routing code...


 [The regression]:


 During my testing of the Sun Neptune based NICs.  On kernel 2.6.27 I
 get really good performance (900-1200kpps) compared to 2.6.28 (davem
 git net-2.6).


 The cause of this problem (tracked down together with Robert Olsson)
 is that on 2.6.28 I have a lot less IRQs available.  It seems max 34
 IRQs.  Due the reduced number of IRQs the NIU driver cannot get
 enough IRQs to the interfaces, and starts to use "IO-APIC" based
 IRQs.


 On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge".  BUT
 my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with
 the usb driver.  That my performance problem on 2.6.28.

 [Other related bugs]:

 Is that unloading the "niu" driver will give a kernel BUG.

\
 
 \ /
  Last update: 2008-11-20 20:51    [from the cache]
©2003-2008