lkml.org 
[lkml]   [2015]   [Jul]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [Ksummit-discuss] [TECH TOPIC] IRQ affinity
On 07/15/2015 11:19 AM, Keith Busch wrote:
> On Wed, 15 Jul 2015, Bart Van Assche wrote:
>> * With blk-mq and scsi-mq optimal performance can only be achieved if
>> the relationship between MSI-X vector and NUMA node does not change
>> over time. This is necessary to allow a blk-mq/scsi-mq driver to
>> ensure that interrupts are processed on the same NUMA node as the
>> node on which the data structures for a communication channel have
>> been allocated. However, today there is no API that allows
>> blk-mq/scsi-mq drivers and irqbalanced to exchange information
>> about the relationship between MSI-X vector ranges and NUMA nodes.
>
> We could have low-level drivers provide blk-mq the controller's irq
> associated with a particular h/w context, and the block layer can provide
> the context's cpumask to irqbalance with the smp affinity hint.
>
> The nvme driver already uses the hwctx cpumask to set hints, but this
> doesn't seems like it should be a driver responsibility. It currently
> doesn't work correctly anyway with hot-cpu since blk-mq could rebalance
> the h/w contexts without syncing with the low-level driver.
>
> If we can add this to blk-mq, one additional case to consider is if the
> same interrupt vector is used with multiple h/w contexts. Blk-mq's cpu
> assignment needs to be aware of this to prevent sharing a vector across
> NUMA nodes.

Exactly. I may have promised to do just that at the last LSF/MM
conference, just haven't done it yet. The point is to share the mask,
I'd ideally like to take it all the way where the driver just asks for a
number of vecs through a nice API that takes care of all this. Lots of
duplicated code in drivers for this these days, and it's a mess.

--
Jens Axboe



\
 
 \ /
  Last update: 2015-07-15 19:41    [W:2.058 / U:0.056 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site