lkml.org 
[lkml]   [2015]   [Dec]   [3]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device
    On Thu, Dec 03, 2015 at 11:55:53AM -0700, Alex Williamson wrote:
    > On Thu, 2015-12-03 at 10:22 -0800, Yunhong Jiang wrote:
    > > When assigning a VFIO device to a KVM guest with low latency requirement, it
    > > is better to handle the interrupt in the hard interrupt context, to reduce
    > > the context switch to/from the IRQ thread.
    > >
    > > Based on discussion on https://lkml.org/lkml/2015/10/26/764, the VFIO msi
    > > interrupt is changed to use request_threaded_irq(). The primary interrupt
    > > handler tries to set the guest interrupt atomically. If it fails to achieve
    > > it, a threaded interrupt handler will be invoked.
    > >
    > > The irq_bypass manager is extended for this purpose. The KVM eventfd will
    > > provide a irqbypass consumer to handle the interrupt at hard interrupt
    > > context. The producer will invoke the consumer's handler then.
    >
    > Do you have any performance data? Thanks,

    Sorry, I should include the data on the commit messages.

    The test:
    Launch a VM with a FPGA device, which triggers an interrpt every 1ms. The VM
    is launched on a isolated CPU with NONHZ_FULL enabled.
    Two data are collected. On the guest, a special program will check the
    latency from the time the interrupt been triggered to the time the interrupt
    received by guest IRS. On the host, I use the perf to collect the perf data.

    The performance Data with the patch:

    Host perf data:
    [root@otcnfv02 test-tools]# perf kvm stat record -C 34 sleep 10
    [root@otcnfv02 test-tools]# perf kvm stat report
    Analyze events for all VMs, all VCPUs:
    VM-EXIT Samples Samples% Time% Min Time Max Time
    Avg time
    EXTERNAL_INTERRUPT 9997 98.55% 99.31% 1.73us 17.07us
    2.09us ( +- 0.21% )
    PAUSE_INSTRUCTION 127 1.25% 0.51% 0.69us 1.20us
    0.84us ( +- 1.34% )
    MSR_WRITE 20 0.20% 0.18% 1.62us 3.21us
    1.93us ( +- 3.95% )

    Guest data:
    [nfv@rt-test-1 ~]$ ./run-int-test.sh
    Latency is Min: 3.74us Max: 20.08us Avg: 4.49us
    No of interrupts = 74995

    The performance data without the patch:
    Host perf data:
    [root@otcnfv02 test-tools]# perf kvm stat record -C 34 sleep 10
    [root@otcnfv02 test-tools]# perf kvm stat report
    Analyze events for all VMs, all VCPUs:
    VM-EXIT Samples Samples% Time% Min Time Max Time
    Avg time
    PAUSE_INSTRUCTION 141136 87.74% 50.39% 0.69us 8.51us
    0.77us ( +- 0.07% )
    EXTERNAL_INTERRUPT 19701 12.25% 49.59% 2.31us 15.93us
    5.46us ( +- 0.12% )
    MSR_WRITE 24 0.01% 0.02% 1.51us 2.22us
    1.91us ( +- 1.91% )

    Notice:
    The EXTERNAL_INTERRUPT VMExit is different w/ the patch (9997) and w/o the
    patch (19701). It is because with threaded IRQ, the NOHZ_FULL it not working
    because two threads on the pCPU, thus we have both the FPGA device interrupt
    and the tick timer interrupt. After calculation, the average time for the
    FPGA device interrupt is 4.72 us.

    Guest data:
    [nfv@rt-test-1 ~]$ ./run-int-test.sh
    Latency is Min: 6.70us Max: 50.38us Avg: 7.44us
    No of interrupts = 42639

    Thanks
    --jyh

    >
    > Alex
    >


    \
     
     \ /
      Last update: 2015-12-04 00:01    [W:2.701 / U:0.032 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site