lkml.org 
[lkml]   [2019]   [Feb]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    Date
    From
    SubjectRe: [patch V5 4/8] nvme-pci: Simplify interrupt allocation
    On Thu, 14 Feb 2019 20:47:59 +0000,
    Thomas Gleixner <tglx@linutronix.de> wrote:
    >
    > From: Ming Lei <ming.lei@redhat.com>
    >
    > The NVME PCI driver contains a tedious mechanism for interrupt
    > allocation, which is necessary to adjust the number and size of interrupt
    > sets to the maximum available number of interrupts which depends on the
    > underlying PCI capabilities and the available CPU resources.
    >
    > It works around the former short comings of the PCI and core interrupt
    > allocation mechanims in combination with interrupt sets.
    >
    > The PCI interrupt allocation function allows to provide a maximum and a
    > minimum number of interrupts to be allocated and tries to allocate as
    > many as possible. This worked without driver interaction as long as there
    > was only a single set of interrupts to handle.
    >
    > With the addition of support for multiple interrupt sets in the generic
    > affinity spreading logic, which is invoked from the PCI interrupt
    > allocation, the adaptive loop in the PCI interrupt allocation did not
    > work for multiple interrupt sets. The reason is that depending on the
    > total number of interrupts which the PCI allocation adaptive loop tries
    > to allocate in each step, the number and the size of the interrupt sets
    > need to be adapted as well. Due to the way the interrupt sets support was
    > implemented there was no way for the PCI interrupt allocation code or the
    > core affinity spreading mechanism to invoke a driver specific function
    > for adapting the interrupt sets configuration.
    >
    > As a consequence the driver had to implement another adaptive loop around
    > the PCI interrupt allocation function and calling that with maximum and
    > minimum interrupts set to the same value. This ensured that the
    > allocation either succeeded or immediately failed without any attempt to
    > adjust the number of interrupts in the PCI code.
    >
    > The core code now allows drivers to provide a callback to recalculate the
    > number and the size of interrupt sets during PCI interrupt allocation,
    > which in turn allows the PCI interrupt allocation function to be called
    > in the same way as with a single set of interrupts. The PCI code handles
    > the adaptive loop and the interrupt affinity spreading mechanism invokes
    > the driver callback to adapt the interrupt set configuration to the
    > current loop value. This replaces the adaptive loop in the driver
    > completely.
    >
    > Implement the NVME specific callback which adjusts the interrupt sets
    > configuration and remove the adaptive allocation loop.
    >
    > [ tglx: Simplify the callback further and restore the dropped adjustment of
    > number of sets ]
    >
    > Signed-off-by: Ming Lei <ming.lei@redhat.com>
    > Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
    >
    > ---
    > drivers/nvme/host/pci.c | 108 ++++++++++++------------------------------------
    > 1 file changed, 28 insertions(+), 80 deletions(-)
    >
    > --- a/drivers/nvme/host/pci.c
    > +++ b/drivers/nvme/host/pci.c
    > @@ -2041,41 +2041,32 @@ static int nvme_setup_host_mem(struct nv
    > return ret;
    > }
    >
    > -/* irq_queues covers admin queue */
    > -static void nvme_calc_io_queues(struct nvme_dev *dev, unsigned int irq_queues)
    > +/*
    > + * nirqs is the number of interrupts available for write and read
    > + * queues. The core already reserved an interrupt for the admin queue.
    > + */
    > +static void nvme_calc_irq_sets(struct irq_affinity *affd, unsigned int nrirqs)
    > {
    > - unsigned int this_w_queues = write_queues;
    > -
    > - WARN_ON(!irq_queues);
    > -
    > - /*
    > - * Setup read/write queue split, assign admin queue one independent
    > - * irq vector if irq_queues is > 1.
    > - */
    > - if (irq_queues <= 2) {
    > - dev->io_queues[HCTX_TYPE_DEFAULT] = 1;
    > - dev->io_queues[HCTX_TYPE_READ] = 0;
    > - return;
    > - }
    > + struct nvme_dev *dev = affd->priv;
    > + unsigned int nr_read_queues;
    >
    > /*
    > - * If 'write_queues' is set, ensure it leaves room for at least
    > - * one read queue and one admin queue
    > - */
    > - if (this_w_queues >= irq_queues)
    > - this_w_queues = irq_queues - 2;
    > -
    > - /*
    > - * If 'write_queues' is set to zero, reads and writes will share
    > - * a queue set.
    > - */
    > - if (!this_w_queues) {
    > - dev->io_queues[HCTX_TYPE_DEFAULT] = irq_queues - 1;
    > - dev->io_queues[HCTX_TYPE_READ] = 0;
    > - } else {
    > - dev->io_queues[HCTX_TYPE_DEFAULT] = this_w_queues;
    > - dev->io_queues[HCTX_TYPE_READ] = irq_queues - this_w_queues - 1;
    > - }
    > + * If only one interrupt is available, combine write and read
    > + * queues. If 'write_queues' is set, ensure it leaves room for at
    > + * least one read queue.

    [Full disclaimer: I only have had two coffees this morning, and it is
    only at the fourth that my brain is able to kick in...]

    I don't know much about NVME, but I feel like there is a small
    disconnect between the code and the above comment, which says "leave
    room for at least one read queue"...

    > + */
    > + if (nrirqs == 1)
    > + nr_read_queues = 0;
    > + else if (write_queues >= nrirqs)
    > + nr_read_queues = nrirqs - 1;

    ... while this seem to ensure that we carve out one write queue out of
    the irq set. It looks like a departure from the original code, which
    would set nr_read_queues to 1 in that particular case.

    Thanks,

    M.

    --
    Jazz is not dead, it just smell funny.

    \
     
     \ /
      Last update: 2019-02-15 10:25    [W:5.970 / U:0.004 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site