Messages in this thread | | | Subject | Re: [PATCH] nvme: Acknowledge completion queue on each iteration | From | Sinan Kaya <> | Date | Tue, 18 Jul 2017 14:52:26 -0400 |
| |
On 7/18/2017 10:36 AM, Keith Busch wrote: > On Mon, Jul 17, 2017 at 07:07:00PM -0400, okaya@codeaurora.org wrote: >> Maybe, I need to understand the design better. I was curious why completion >> and submission queues were protected by a single lock causing lock >> contention. > Ideally the queues are tied to CPUs, so you couldn't have one thread > submitting to a particular queue-pair while another thread is reaping > completions from it. Such a setup wouldn't get lock contention.
I do see that the NVMe driver is creating a completion interrupt on each CPU core for the completions. No problems with that.
However, I don't think you can guarantee that there will always be a single CPU core targeting one submission queue especially with asynchronous IO.
Lock contention counters from CONFIG_LOCK_STAT are pointing to nvmeq->lock in my FIO tests.
Did I miss something?
> > Some machines have so many CPUs, though, that sharing hardware queues > is required. We've experimented with separate submission and completion > locks for such cases, but I've never seen an improved performance as a > result. >
I have also experimented with multiple locks with no significant gains. However, I was curious if somebody else had a better implementation than mine.
-- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.
| |