lkml.org 
[lkml]   [2008]   [Nov]   [20]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRegression: Bisected, IRQ and MSI allocations screwed without sparse irq
    Hi Thomas Gleixner,

    I have bisected a regression to your commit
    3235e936c0cc3589309280b6f59e5096779adae3,
    "x86: remove sparse irq from Kconfig".

    Its actually not necessary your fault, as your commit simply removes
    the config option HAVE_SPARSE_IRQ. This revels the bug / regression
    I'm exposted to.

    Guess I should bisect again to find the exact faulty commit, but I'm
    rather sick of bisecting at the moment, and though you might have a
    better idea whats going wrong. I would rather spend my time
    performance tuning the multiqueue routing code...

    [The regression]:

    During my testing of the Sun Neptune based NICs. On kernel 2.6.27 I
    get really good performance (900-1200kpps) compared to 2.6.28 (davem
    git net-2.6).

    The cause of this problem (tracked down together with Robert Olsson)
    is that on 2.6.28 I have a lot less IRQs available. It seems max 34
    IRQs. Due the reduced number of IRQs the NIU driver cannot get
    enough IRQs to the interfaces, and starts to use "IO-APIC" based
    IRQs.

    On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge". BUT
    my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with
    the usb driver. That my performance problem on 2.6.28.

    [Other related bugs]:
    Is that unloading the "niu" driver will give a kernel BUG during
    deallocation og MSI interrupts. (See dmesg output below if interested)

    (I have attached full bisect history)

    Cheers,
    Jesper Brouer

    --
    -------------------------------------------------------------------
    MSc. Master of Computer Science
    Dept. of Computer Science, University of Copenhagen
    Author of http://www.adsl-optimizer.dk
    -------------------------------------------------------------------


    On Wed, 19 Nov 2008, David Miller wrote:
    > From: Jesper Dangaard Brouer <hawk@diku.dk>
    > Date: Wed, 19 Nov 2008 23:58:12 +0100 (CET)
    >
    >> Well that was not the real cause of the performance loss. Because
    >> on kernel 2.6.27 I get really good performance (900-1200kpps)
    >> compared to 2.6.28 (git net-2.6).
    >>
    >> The cause of this problem (tracked down together with Robert Olsson)
    >> is that on 2.6.28 I have a lot less IRQs available. It seems max 34
    >> IRQs.
    >>
    >> Due the reduced number of IRQs the NIU driver cannot get enough IRQs
    >> to the interfaces, and starts to use "IO-APIC" based IRQs.
    >
    > This is almost certainly related to the driver unload bug.
    >
    > I know you ran into unbuildable/unbootable kernels during a bisect,
    > but you really need to track down this regression.


    ------------[ cut here ]------------
    kernel BUG at drivers/pci/msi.c:632!
    invalid opcode: 0000 [#1] PREEMPT SMP
    Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo
    niu(-)

    Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5
    EIP: 0060:[<c021ecac>] EFLAGS: 00010286 CPU: 2
    EIP is at msi_free_irqs+0xdc/0xe0
    EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500
    ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8
    DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
    Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000)
    Stack:
    f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000
    f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858
    f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4
    Call Trace:
    [<c021ecb8>] ? msix_free_all_irqs+0x8/0x10
    [<c021ef41>] ? pci_disable_msix+0x31/0x40
    [<f809d3f8>] ? niu_pci_remove_one+0x88/0x8a [niu]
    [<c0219c29>] ? pci_device_remove+0x19/0x40
    [<c026b0d4>] ? __device_release_driver+0x54/0x80
    [<c026b197>] ? driver_detach+0x97/0xa0
    [<c026a475>] ? bus_remove_driver+0x75/0xa0
    [<c026b609>] ? driver_unregister+0x39/0x40
    [<c0219e51>] ? pci_unregister_driver+0x21/0x80
    [<f809a0ad>] ? niu_exit+0xd/0x10 [niu]
    [<c0145d74>] ? sys_delete_module+0x114/0x1d0
    [<c016810a>] ? remove_vma+0x3a/0x50
    [<c0168c29>] ? do_munmap+0x189/0x1e0
    [<c0103229>] ? sysenter_do_call+0x12/0x21
    [<c0330000>] ? quirk_disable_msi+0x30/0x50
    Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b
    14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe
    55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55
    EIP: [<c021ecac>] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8
    ---[ end trace f72de2e283920207 ]---
    ~~ -*-text-*-

    -------------------------------------------------------
    Bisecting IRQ change:
    What change reduced the IRQs
    -------------------------------------------------------
    Jesper Dangaard Brouer (jdb@comx.dk)
    -------------------------------------------------------
    $LastChangedRevision: 786 $
    $Date: 2008-11-20 20:44:51 +0100 (Thu, 20 Nov 2008) $
    -------------------------------------------------------

    git clone
    ~~~~~~~~~

    +---------
    cd /var/kernels/git/davem
    git clone git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git net-2.6-bisect-irqs
    +---------

    Description / Reason to find
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    During my testing of the Sun Neptune based NICs.

    On kernel 2.6.27 I get really good performance (900-1200kpps)
    compared to 2.6.28 (git net-2.6).

    The cause of this problem (tracked down together with Robert Olsson)
    is that on 2.6.28 I have a lot less IRQs available. It seems max 34
    IRQs.

    Due the reduced number of IRQs the NIU driver cannot get enough IRQs
    to the interfaces, and starts to use "IO-APIC" based IRQs.

    On kernel 2.6.28:

    My eth2 is using 10 IRQs all "PCI-MSI-edge".

    BUT my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared
    with the usb driver...

    Think thats must be my performance problem on 2.6.28.

    Known: Good and bad
    ~~~~~~~~~~~~~~~~~~~

    GOOD:
    git bisect good v2.6.27

    BAD:
    git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43

    [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] #Merge branch
    'tracing-v28-for-linus' of
    git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip


    HiSTORY:
    ~~~~~~~~

    +--------
    cd /var/kernels/git/davem/net-2.6-bisect-irqs/
    git bisect start
    git bisect good v2.6.27
    +--------

    +--------------
    git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
    Bisecting: 3220 revisions left to test after this
    [af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2
    +--------------

    CONFIG_LOCALVERSION="-bisect"

    +-------------
    cp ../net-2.6-bisect/.config .
    script make_oldconfig_01
    make oldconfig
    exit
    #Script done, file is make_oldconfig_01
    +-------------

    +----------------
    time make -j6 bzImage modules
    #
    #real 9m22.739s
    #user 16m56.776s
    #sys 1m4.672s
    +----------------

    Booted kernel: GOOD: irqs and (niu rmmod good)

    +----------------
    git bisect good
    Bisecting: 1614 revisions left to test after this
    [36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
    +----------------

    Compiling:

    +----------------
    time make -j6 bzImage modules
    +----------------

    Booted kernel: GOOD: irqs and (niu rmmod good)

    +-----------
    git bisect good
    Bisecting: 807 revisions left to test after this
    [1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen()
    +-----------

    Compiling...

    +----------------
    time make -j6 bzImage modules
    #real 10m1.561s
    #user 17m23.293s
    #sys 1m5.744s
    +----------------

    Installing...

    Booted kernel:

    +----
    dcu-router-ng:~# uname -a
    Linux dcu-router-ng 2.6.27-bisect #3 SMP PREEMPT Thu Nov 20 12:33:02 CET 2008 i686 GNU/Linux
    +----

    Results: GOOD: irqs and (niu rmmod good)

    +------
    git bisect good
    Bisecting: 403 revisions left to test after this
    [1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
    +------

    Compiling...

    +----------------
    time make -j6 bzImage modules
    #real 10m9.371s
    #user 17m21.781s
    #sys 1m6.052s
    +----------------

    Installing...

    Booting ...

    +-------
    dcu-router-ng:~# uname -a
    Linux dcu-router-ng 2.6.27-bisect #4 SMP PREEMPT Thu Nov 20 12:50:39 CET 2008 i686 GNU/Linux
    +-------

    Results: GOOD: irqs and (niu rmmod good)


    +-------
    git-bisect good
    Bisecting: 223 revisions left to test after this
    [dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files
    +-------

    +----------------
    time make -j6 bzImage modules
    +----------------

    Booting ...

    +--------
    Linux dcu-router-ng 2.6.27-bisect #5 SMP PREEMPT Thu Nov 20 13:58:34 CET 2008 i686 GNU/Linux
    +--------

    Results: BAD: irqs and (niu rmmod also BAD)

    +-------
    cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3
    0: 125 0 0 0 IO-APIC-edge timer
    1: 0 0 1 1 IO-APIC-edge i8042
    3: 2 1 2 2 IO-APIC-edge serial
    8: 0 2 0 0 IO-APIC-edge rtc
    9: 0 0 0 0 IO-APIC-fasteoi acpi
    12: 1 2 1 0 IO-APIC-edge i8042
    16: 103 108 108 112 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6, eth0
    17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
    18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4, eth3
    20: 0 0 0 0 PCI-MSI-edge eth2
    21: 0 0 0 0 PCI-MSI-edge eth2
    22: 24 23 23 23 IO-APIC-fasteoi uhci_hcd:usb5, eth2
    23: 0 0 0 0 PCI-MSI-edge eth2
    24: 0 0 0 0 PCI-MSI-edge eth2
    25: 0 0 0 0 PCI-MSI-edge eth2
    26: 0 0 0 0 PCI-MSI-edge eth2
    27: 0 0 0 0 PCI-MSI-edge eth2
    28: 0 0 0 0 PCI-MSI-edge eth2
    29: 0 0 0 0 PCI-MSI-edge eth2
    30: 0 0 0 0 PCI-MSI-edge eth2
    31: 0 0 0 0 PCI-MSI-edge eth2
    32: 0 0 0 0 PCI-MSI-edge eth2
    34: 271 268 268 264 PCI-MSI-edge cciss0
    NMI: 0 0 0 0 Non-maskable interrupts
    LOC: 3301 2970 2594 2389 Local timer interrupts
    RES: 28 560 6 13 Rescheduling interrupts
    CAL: 50 104 99 62 Function call interrupts
    TLB: 241 224 287 279 TLB shootdowns
    TRM: 0 0 0 0 Thermal event interrupts
    SPU: 0 0 0 0 Spurious interrupts
    ERR: 0
    MIS: 0
    +-------

    OUTPUT "rmmod niu" (gives segfault) and "dmesg"

    +-------
    ------------[ cut here ]------------
    kernel BUG at drivers/pci/msi.c:632!
    invalid opcode: 0000 [#1] PREEMPT SMP
    Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo niu(-)

    Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5
    EIP: 0060:[<c021ecac>] EFLAGS: 00010286 CPU: 2
    EIP is at msi_free_irqs+0xdc/0xe0
    EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500
    ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8
    DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
    Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000)
    Stack:
    f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000
    f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858
    f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4
    Call Trace:
    [<c021ecb8>] ? msix_free_all_irqs+0x8/0x10
    [<c021ef41>] ? pci_disable_msix+0x31/0x40
    [<f809d3f8>] ? niu_pci_remove_one+0x88/0x8a [niu]
    [<c0219c29>] ? pci_device_remove+0x19/0x40
    [<c026b0d4>] ? __device_release_driver+0x54/0x80
    [<c026b197>] ? driver_detach+0x97/0xa0
    [<c026a475>] ? bus_remove_driver+0x75/0xa0
    [<c026b609>] ? driver_unregister+0x39/0x40
    [<c0219e51>] ? pci_unregister_driver+0x21/0x80
    [<f809a0ad>] ? niu_exit+0xd/0x10 [niu]
    [<c0145d74>] ? sys_delete_module+0x114/0x1d0
    [<c016810a>] ? remove_vma+0x3a/0x50
    [<c0168c29>] ? do_munmap+0x189/0x1e0
    [<c0103229>] ? sysenter_do_call+0x12/0x21
    [<c0330000>] ? quirk_disable_msi+0x30/0x50
    Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b 14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe 55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55
    EIP: [<c021ecac>] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8
    ---[ end trace f72de2e283920207 ]---
    +-------

    +------
    git-bisect bad
    Bisecting: 89 revisions left to test after this
    [db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function
    +------

    Related config change? (make oldconfig)

    +------
    script make_oldconfig_02
    make oldconfig
    Script done, file is make_oldconfig_02
    +------

    +------
    Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) ? ?Y

    This enables support for sparse irq, esp for msi/msi-x. the irq
    number will be bus/dev/fn + 12bit. You may need if you have lots of
    cards supports msi-x installed.

    If you don't know what to do here, say Y.
    +------

    Compiling...

    +----------------
    time make -j6 bzImage modules
    #
    #real 9m29.556s
    #user 17m10.396s
    #sys 1m5.056s
    +----------------

    Booting ...

    +-------
    Linux dcu-router-ng 2.6.27-bisect #6 SMP PREEMPT Thu Nov 20 14:25:40 CET 2008 i686 GNU/Linux
    +-------

    The output from /proc/interrupts changed, very weird!
    BUT eth3 does use a "PCI-MSI-edge" interrupt.

    Guess this is a GOOD state even though it looks weird.

    Unloading NIU driver also GOOD.

    +---------
    cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3
    0x0: 124 1 0 0 IO-APIC-edge timer
    0x1: 1 0 0 1 IO-APIC-edge i8042
    0x3: 2 1 2 2 IO-APIC-edge serial
    0x8: 1 0 0 1 IO-APIC-edge rtc
    0x9: 0 0 0 0 IO-APIC-fasteoi acpi
    0xc: 0 1 2 1 IO-APIC-edge i8042
    0x10: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
    0x11: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
    0x12: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    0x6000fe: 288 289 290 290 PCI-MSI-edge cciss0
    0x16: 23 24 23 23 IO-APIC-fasteoi uhci_hcd:usb5
    0xb00100: 0 0 0 0 PCI-MSI-edge eth2
    0xb000ff: 0 0 0 0 PCI-MSI-edge eth2
    0xb000fe: 0 0 0 0 PCI-MSI-edge eth2
    0xb000fd: 0 0 0 0 PCI-MSI-edge eth2
    0xb000fc: 0 0 0 0 PCI-MSI-edge eth2
    0xb000fb: 0 0 0 0 PCI-MSI-edge eth2
    0xb000fa: 0 0 0 0 PCI-MSI-edge eth2
    0xb000f9: 0 0 0 0 PCI-MSI-edge eth2
    0xb000f8: 0 0 0 0 PCI-MSI-edge eth2
    0xb000f7: 0 0 0 0 PCI-MSI-edge eth2
    0xb000f6: 0 0 0 0 PCI-MSI-edge eth2
    0xb000f5: 0 0 0 0 PCI-MSI-edge eth2
    0xb000f4: 0 0 0 0 PCI-MSI-edge eth2
    0xb01100: 0 0 0 0 PCI-MSI-edge eth3
    0xb010ff: 0 0 0 0 PCI-MSI-edge eth3
    0xb010fe: 0 0 0 0 PCI-MSI-edge eth3
    0xb010fd: 0 0 0 0 PCI-MSI-edge eth3
    0xb010fc: 0 0 0 0 PCI-MSI-edge eth3
    0xb010fb: 0 0 0 0 PCI-MSI-edge eth3
    0xb010fa: 0 0 0 0 PCI-MSI-edge eth3
    0xb010f9: 0 0 0 0 PCI-MSI-edge eth3
    0xb010f8: 0 0 0 0 PCI-MSI-edge eth3
    0xb010f7: 0 0 0 0 PCI-MSI-edge eth3
    0xb010f6: 0 0 0 0 PCI-MSI-edge eth3
    0x13: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
    0x300100: 210 210 210 208 PCI-MSI-edge eth0
    NMI: 0 0 0 0 Non-maskable interrupts
    LOC: 3630 3265 3103 2711 Local timer interrupts
    RES: 34 226 12 417 Rescheduling interrupts
    CAL: 89 55 90 78 Function call interrupts
    TLB: 253 205 311 267 TLB shootdowns
    TRM: 0 0 0 0 Thermal event interrupts
    SPU: 0 0 0 0 Spurious interrupts
    ERR: 0
    MIS: 0
    +---------

    Guess it a GOOD situation...

    +------
    git-bisect good
    Bisecting: 44 revisions left to test after this
    [ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI
    +------

    Compiling ...

    +--------
    time make -j6 bzImage modules
    #real 9m28.062s
    #user 17m7.492s
    #sys 1m4.248s
    +--------

    Installing ...

    Booting ...

    +------
    Linux dcu-router-ng 2.6.27-bisect #7 SMP PREEMPT Thu Nov 20 14:52:45 CET 2008 i686 GNU/Linux
    +------

    Still looks GOOD (/proc/interrupts still looks weird).

    And rmmod NIU driver GOOD.

    +------
    git-bisect good
    Bisecting: 22 revisions left to test after this
    [922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4
    +------

    Compiling ...

    +--------
    time make -j6 bzImage modules
    #real 0m34.622s
    #user 0m41.139s
    #sys 0m5.812s
    +--------

    Install ...

    Booting ...

    +-----
    Linux dcu-router-ng 2.6.27-bisect #8 SMP PREEMPT Thu Nov 20 15:04:11 CET 2008 i686 GNU/Linux
    +-----

    Looks GOOD, and /proc/interrupts changed again! Now the interrupts
    are not i HEX anymore, but in decimal, but still strange/large
    numbers for MSI.

    Unloading NIU driver GOOD.

    +------
    cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3
    0: 124 0 0 0 IO-APIC-edge timer
    1: 0 0 1 1 IO-APIC-edge i8042
    3: 2 2 1 2 IO-APIC-edge serial
    8: 0 0 1 1 IO-APIC-edge rtc
    9: 0 0 0 0 IO-APIC-fasteoi acpi
    12: 1 2 1 0 IO-APIC-edge i8042
    16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
    17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
    18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    6291710: 828 821 828 823 PCI-MSI-edge cciss0
    22: 23 24 24 22 IO-APIC-fasteoi uhci_hcd:usb5
    11534592: 0 0 0 0 PCI-MSI-edge eth2
    11534591: 0 0 0 0 PCI-MSI-edge eth2
    11534590: 0 0 0 0 PCI-MSI-edge eth2
    11534589: 0 0 0 0 PCI-MSI-edge eth2
    11534588: 0 0 0 0 PCI-MSI-edge eth2
    11534587: 0 0 0 0 PCI-MSI-edge eth2
    11534586: 0 0 0 0 PCI-MSI-edge eth2
    11534585: 0 0 0 0 PCI-MSI-edge eth2
    11534584: 0 0 0 0 PCI-MSI-edge eth2
    11534583: 0 0 0 0 PCI-MSI-edge eth2
    11534582: 0 0 0 0 PCI-MSI-edge eth2
    11534581: 0 0 0 0 PCI-MSI-edge eth2
    11534580: 0 0 0 0 PCI-MSI-edge eth2
    11538688: 0 0 0 0 PCI-MSI-edge eth3
    11538687: 0 0 0 0 PCI-MSI-edge eth3
    11538686: 0 0 0 0 PCI-MSI-edge eth3
    11538685: 0 0 0 0 PCI-MSI-edge eth3
    11538684: 0 0 0 0 PCI-MSI-edge eth3
    11538683: 0 0 0 0 PCI-MSI-edge eth3
    11538682: 0 0 0 0 PCI-MSI-edge eth3
    11538681: 0 0 0 0 PCI-MSI-edge eth3
    11538680: 0 0 0 0 PCI-MSI-edge eth3
    11538679: 0 0 0 0 PCI-MSI-edge eth3
    11538678: 0 0 0 0 PCI-MSI-edge eth3
    19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
    3145984: 9993 9994 9987 9993 PCI-MSI-edge eth0
    NMI: 0 0 0 0 Non-maskable interrupts
    LOC: 10075 11448 8001 8787 Local timer interrupts
    RES: 297 17 349 26 Rescheduling interrupts
    CAL: 173 189 95 173 Function call interrupts
    TLB: 299 259 330 345 TLB shootdowns
    TRM: 0 0 0 0 Thermal event interrupts
    SPU: 0 0 0 0 Spurious interrupts
    ERR: 0
    MIS: 0
    +------

    +-------
    git-bisect good
    Bisecting: 11 revisions left to test after this
    [a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal
    +-------

    Compiling

    +--------
    time make -j6 bzImage modules
    #real 9m30.767s
    #user 17m10.808s
    #sys 1m6.388s
    +--------

    Installing ...

    Booting ...

    +-----
    Linux dcu-router-ng 2.6.27-bisect #9 SMP PREEMPT Thu Nov 20 15:28:17 CET 2008 i686 GNU/Linux
    +-----

    BAD kernel version, max IRQ is 34. And eth3 got assigned a
    IO-APIC-fasteoi shared with uhci_hcd:usb2.

    Also BAD unloading of NIU driver.

    BUG is some where in between:

    git log 922402f15a85f7a064926eb1db68cc52bc4d4a91..a1aca5de08a0cb840a90fb3f729a5940f8d21185 | grep ^commit | wc -l

    11 commits


    +-------
    git-bisect bad
    Bisecting: 5 revisions left to test after this
    [3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig
    +-------

    Compiling...

    +--------
    time make -j6 bzImage modules
    +--------

    Install ...

    Booting

    +------
    Linux dcu-router-ng 2.6.27-bisect #10 SMP PREEMPT Thu Nov 20 15:56:10 CET 2008 i686 GNU/Linux
    +------

    BAD kernel.

    BAD rmmod NIU driver.

    +---------
    git bisect bad
    Bisecting: 2 revisions left to test after this
    [4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out
    +---------

    Compiling...

    +------
    time make -j6 bzImage modules
    #real 7m23.814s
    #user 12m15.718s
    #sys 0m42.183s
    +------

    Config change prompting:

    +-----
    Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) Y
    +-----

    Installing ...

    Booting ...

    +-------
    Linux dcu-router-ng 2.6.27-bisect #11 SMP PREEMPT Thu Nov 20 16:19:29 CET 2008 i686 GNU/Linux
    +-------

    GOOD!!!

    +--------
    cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3
    0: 124 0 0 0 IO-APIC-edge timer
    1: 0 0 1 1 IO-APIC-edge i8042
    3: 2 2 2 2 IO-APIC-edge serial
    8: 1 0 0 1 IO-APIC-edge rtc
    9: 0 0 0 0 IO-APIC-fasteoi acpi
    12: 1 2 1 0 IO-APIC-edge i8042
    16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
    17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
    18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    6291710: 285 288 293 287 PCI-MSI-edge cciss0
    11534592: 0 0 0 0 PCI-MSI-edge eth2
    11534591: 0 0 0 0 PCI-MSI-edge eth2
    11534590: 0 0 0 0 PCI-MSI-edge eth2
    11534589: 0 0 0 0 PCI-MSI-edge eth2
    11534588: 0 0 0 0 PCI-MSI-edge eth2
    11534587: 0 0 0 0 PCI-MSI-edge eth2
    11534586: 0 0 0 0 PCI-MSI-edge eth2
    11534585: 0 0 0 0 PCI-MSI-edge eth2
    11534584: 0 0 0 0 PCI-MSI-edge eth2
    11534583: 0 0 0 0 PCI-MSI-edge eth2
    11534582: 0 0 0 0 PCI-MSI-edge eth2
    11534581: 0 0 0 0 PCI-MSI-edge eth2
    11534580: 0 0 0 0 PCI-MSI-edge eth2
    22: 23 24 23 23 IO-APIC-fasteoi uhci_hcd:usb5
    11538688: 0 0 0 0 PCI-MSI-edge eth3
    11538687: 0 0 0 0 PCI-MSI-edge eth3
    11538686: 0 0 0 0 PCI-MSI-edge eth3
    11538685: 0 0 0 0 PCI-MSI-edge eth3
    11538684: 0 0 0 0 PCI-MSI-edge eth3
    11538683: 0 0 0 0 PCI-MSI-edge eth3
    11538682: 0 0 0 0 PCI-MSI-edge eth3
    11538681: 0 0 0 0 PCI-MSI-edge eth3
    11538680: 0 0 0 0 PCI-MSI-edge eth3
    11538679: 0 0 0 0 PCI-MSI-edge eth3
    11538678: 0 0 0 0 PCI-MSI-edge eth3
    19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
    3145984: 244 242 238 241 PCI-MSI-edge eth0
    NMI: 0 0 0 0 Non-maskable interrupts
    LOC: 3715 3104 2853 2542 Local timer interrupts
    RES: 88 52 280 258 Rescheduling interrupts
    CAL: 76 75 93 59 Function call interrupts
    TLB: 245 241 312 283 TLB shootdowns
    TRM: 0 0 0 0 Thermal event interrupts
    SPU: 0 0 0 0 Spurious interrupts
    ERR: 0
    MIS: 0
    +---------


    +---------
    git-bisect good
    Bisecting: 1 revisions left to test after this
    [7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n
    +----------

    Compiling ...

    +------
    time make -j6 bzImage modules
    +------

    Install...

    Boot ...

    +-------
    Linux dcu-router-ng 2.6.27-bisect #12 SMP PREEMPT Thu Nov 20 16:33:11 CET 2008 i686 GNU/Linux
    +-------

    +--------
    cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3
    0: 124 0 0 0 IO-APIC-edge timer
    1: 0 0 1 1 IO-APIC-edge i8042
    3: 1 2 2 2 IO-APIC-edge serial
    8: 2 0 0 0 IO-APIC-edge rtc
    9: 0 0 0 0 IO-APIC-fasteoi acpi
    12: 1 2 1 0 IO-APIC-edge i8042
    16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
    17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
    18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    6291710: 268 269 268 267 PCI-MSI-edge cciss0
    11534592: 0 0 0 0 PCI-MSI-edge eth2
    11534591: 0 0 0 0 PCI-MSI-edge eth2
    11534590: 0 0 0 0 PCI-MSI-edge eth2
    11534589: 0 0 0 0 PCI-MSI-edge eth2
    11534588: 0 0 0 0 PCI-MSI-edge eth2
    11534587: 0 0 0 0 PCI-MSI-edge eth2
    11534586: 0 0 0 0 PCI-MSI-edge eth2
    11534585: 0 0 0 0 PCI-MSI-edge eth2
    11534584: 0 0 0 0 PCI-MSI-edge eth2
    11534583: 0 0 0 0 PCI-MSI-edge eth2
    11534582: 0 0 0 0 PCI-MSI-edge eth2
    11534581: 0 0 0 0 PCI-MSI-edge eth2
    11534580: 0 0 0 0 PCI-MSI-edge eth2
    11538688: 0 0 0 0 PCI-MSI-edge eth3
    11538687: 0 0 0 0 PCI-MSI-edge eth3
    11538686: 0 0 0 0 PCI-MSI-edge eth3
    11538685: 0 0 0 0 PCI-MSI-edge eth3
    11538684: 0 0 0 0 PCI-MSI-edge eth3
    11538683: 0 0 0 0 PCI-MSI-edge eth3
    11538682: 0 0 0 0 PCI-MSI-edge eth3
    11538681: 0 0 0 0 PCI-MSI-edge eth3
    11538680: 0 0 0 0 PCI-MSI-edge eth3
    11538679: 0 0 0 0 PCI-MSI-edge eth3
    11538678: 0 0 0 0 PCI-MSI-edge eth3
    19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
    22: 25 23 24 25 IO-APIC-fasteoi uhci_hcd:usb5
    3145984: 175 174 176 178 PCI-MSI-edge eth0
    NMI: 0 0 0 0 Non-maskable interrupts
    LOC: 3508 2902 2765 2489 Local timer interrupts
    RES: 238 35 461 6 Rescheduling interrupts
    CAL: 61 90 59 81 Function call interrupts
    TLB: 257 220 299 300 TLB shootdowns
    TRM: 0 0 0 0 Thermal event interrupts
    SPU: 0 0 0 0 Spurious interrupts
    ERR: 0
    MIS: 0
    +--------

    GOOD.

    +----------
    git-bisect good
    3235e936c0cc3589309280b6f59e5096779adae3 is first bad commit
    commit 3235e936c0cc3589309280b6f59e5096779adae3
    Author: Thomas Gleixner <tglx@linutronix.de>
    Date: Wed Oct 15 13:16:00 2008 +0200

    x86: remove sparse irq from Kconfig

    This code is not ready yet.

    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

    :040000 040000 6043e32465556e828de0fbb6aa497b277239af01 2dd75ba207990d83a3a4c7b7b16abccfe2d5e10d M arch
    +--------

    Found bad commit:
    3235e936c0cc3589309280b6f59e5096779adae3



    Git bisect LOG
    ~~~~~~~~~~~~~~

    +-------
    git-bisect log
    git-bisect start
    # good: [3fa8749e584b55f1180411ab1b51117190bac1e5] Linux 2.6.27
    git-bisect good 3fa8749e584b55f1180411ab1b51117190bac1e5
    # bad: [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
    git-bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
    # good: [af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2
    git-bisect good af5c2bd16ac2e5688c3bf46ea1f95112d696d294
    # good: [36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
    git-bisect good 36ac1d2f323f8bf8bc10c25b88f617657720e241
    # good: [1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen()
    git-bisect good 1aece34833721d64eb33fc15cd923c727296d3d3
    # good: [1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
    git-bisect good 1d9a8a47d659f053abeca9ece45651b4d94780c8
    # bad: [dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files
    git-bisect bad dd3a1db900f2a215a7d7dd71b836e149a6cf5fed
    # good: [db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function
    git-bisect good db4b5525caafd846ec20f95afbc6403c792e22cf
    # good: [ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI
    git-bisect good ba374c9baef910fbc5373541d98c50f15e82c3f8
    # good: [922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4
    git-bisect good 922402f15a85f7a064926eb1db68cc52bc4d4a91
    # bad: [a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal
    git-bisect bad a1aca5de08a0cb840a90fb3f729a5940f8d21185
    # bad: [3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig
    git-bisect bad 3235e936c0cc3589309280b6f59e5096779adae3
    # good: [4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out
    git-bisect good 4c66a73f0796dacc2ff0d4af75794ec843ceb3d1
    # good: [7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n
    git-bisect good 7ef0c30dbf96a8d9a234e90c248eb19df3c031be
    +-------

    Email
    ~~~~~

    To:
    Thomas Gleixner <tglx@linutronix.de>
    David Miller <davem@davemloft.net>,
    Jesper Dangaard Brouer <jdb@comx.dk>,
    netdev <netdev@vger.kernel.org>,
    linux-kernel@vger.kernel.org,
    Robert Olsson <Robert.Olsson@data.slu.se>

    Subj.:
    Regression: Bisected, IRQ and MSI allocations screwed without sparse irq


    Hi Thomas Gleixner,

    I have bisected a regression to your commit
    3235e936c0cc3589309280b6f59e5096779adae3, "x86: remove sparse irq
    from Kconfig".

    Its actually not necessary your fault, as your commit simply removes
    the config option HAVE_SPARSE_IRQ. This revels the bug / regression
    I'm exposted to.

    Guess I should bisect again to find the exact faulty commit, but I'm
    rather sick of bisecting at the moment, and though you might have a
    better idea whats going wrong. I would rather spend my time
    performance tuning the multiqueue routing code...

    [The regression]:

    During my testing of the Sun Neptune based NICs. On kernel 2.6.27 I
    get really good performance (900-1200kpps) compared to 2.6.28 (davem
    git net-2.6).

    The cause of this problem (tracked down together with Robert Olsson)
    is that on 2.6.28 I have a lot less IRQs available. It seems max 34
    IRQs. Due the reduced number of IRQs the NIU driver cannot get
    enough IRQs to the interfaces, and starts to use "IO-APIC" based
    IRQs.

    On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge". BUT
    my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with
    the usb driver. That my performance problem on 2.6.28.

    [Other related bugs]:
    Is that unloading the "niu" driver will give a kernel BUG.

    \
     
     \ /
      Last update: 2008-11-20 20:51    [W:0.079 / U:91.612 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site