Messages in this thread Patch in this message | | | Subject | Re: [PATCH 1/4] softirq: implement IRQ flood detection mechanism | From | Sagi Grimberg <> | Date | Fri, 20 Sep 2019 13:45:30 -0700 |
| |
>>> Sagi, >>> >>> Sorry it took a while to bring my system back online. >>> >>> With the patch, the IOPS is about the same drop with the 1st patch. I think >> the excessive context switches are causing the drop in IOPS. >>> >>> The following are captured by "perf sched record" for 30 seconds during >> tests. >>> >>> "perf sched latency" >>> With patch: >>> fio:(82) | 937632.706 ms | 1782255 | avg: 0.209 ms | max: 63.123 >> ms | max at: 768.274023 s >>> >>> without patch: >>> fio:(82) |2348323.432 ms | 18848 | avg: 0.295 ms | max: 28.446 >> ms | max at: 6447.310255 s >> >> Without patch means the proposed hard-irq patch? > > It means the current upstream code without any patch. But It's prone to soft lockup. > > Ming's proposed hard-irq patch gets similar results to "without patch", however it fixes the soft lockup.
Thanks for the clarification.
The problem with what Ming is proposing in my mind (and its an existing problem that exists today), is that nvme is taking precedence over anything else until it absolutely cannot hog the cpu in hardirq.
In the thread Ming referenced a case where today if the cpu core has a net softirq activity it cannot make forward progress. So with Ming's suggestion, net softirq will eventually make progress, but it creates an inherent fairness issue. Who said that nvme completions should come faster then the net rx/tx or another I/O device (or hrtimers or sched events...)?
As much as I'd like nvme to complete as soon as possible, I might have other activities in the system that are as important if not more. So I don't think we can solve this with something that is not cooperative or fair with the rest of the system.
>> If we are context switching too much, it means the soft-irq operation is not >> efficient, not necessarily the fact that the completion path is running in soft- >> irq.. >> >> Is your kernel compiled with full preemption or voluntary preemption? > > The tests are based on Ubuntu 18.04 kernel configuration. Here are the parameters: > > # CONFIG_PREEMPT_NONE is not set > CONFIG_PREEMPT_VOLUNTARY=y > # CONFIG_PREEMPT is not set
I see, so it still seems that irq_poll_softirq is still not efficient in reaping completions. reaping the completions on its own is pretty much the same in hard and soft irq, so its really the scheduling part that is creating the overhead (which does not exist in hard irq).
Question: when you test with without the patch (completions are coming in hard-irq), do the fio threads that run on the cpu cores that are assigned to the cores that are handling interrupts get substantially lower throughput than the rest of the fio threads? I would expect that the fio threads that are running on the first 32 cores to get very low iops (overpowered by the nvme interrupts) and the rest doing much more given that nvme has almost no limits to how much time it can spend on processing completions.
If need_resched() is causing us to context switch too aggressively, does changing that to local_softirq_pending() make things better? -- diff --git a/lib/irq_poll.c b/lib/irq_poll.c index d8eab563fa77..05d524fcaf04 100644 --- a/lib/irq_poll.c +++ b/lib/irq_poll.c @@ -116,7 +116,7 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) /* * If softirq window is exhausted then punt. */ - if (need_resched()) + if (local_softirq_pending()) break; } -- Although, this can potentially cause other threads from making forward progress.. If it is better, perhaps we also need a time limit as well.
Perhaps we should add statistics/tracing on how many completions we are reaping per invocation...
| |