Messages in this thread | | | Date | Wed, 24 Jan 2018 23:49:07 -0800 | From | vcaputo@pengaru ... | Subject | Re: [REGRESSION] (>= v4.12) IO w/dmcrypt causing audio underruns |
| |
On Fri, Jan 19, 2018 at 11:57:32AM +0100, Enric Balletbo Serra wrote: > Hi Vito, > > 2018-01-17 23:48 GMT+01:00 <vcaputo@pengaru.com>: > > On Mon, Dec 18, 2017 at 10:25:33AM +0100, Enric Balletbo Serra wrote: > >> Hi Vito, > >> > >> 2017-12-01 22:33 GMT+01:00 <vcaputo@pengaru.com>: > >> > On Wed, Nov 29, 2017 at 10:39:19AM -0800, vcaputo@pengaru.com wrote: > >> >> Hello, > >> >> > >> >> Recently I noticed substantial audio dropouts when listening to MP3s in > >> >> `cmus` while doing big and churny `git checkout` commands in my linux git > >> >> tree. > >> >> > >> >> It's not something I've done much of over the last couple months so I > >> >> hadn't noticed until yesterday, but didn't remember this being a problem in > >> >> recent history. > >> >> > >> >> As there's quite an accumulation of similarly configured and built kernels > >> >> in my grub menu, it was trivial to determine approximately when this began: > >> >> > >> >> 4.11.0: no dropouts > >> >> 4.12.0-rc7: dropouts > >> >> 4.14.0-rc6: dropouts (seem more substantial as well, didn't investigate) > >> >> > >> >> Watching top while this is going on in the various kernel versions, it's > >> >> apparent that the kworker behavior changed. Both the priority and quantity > >> >> of running kworker threads is elevated in kernels experiencing dropouts. > >> >> > >> >> Searching through the commit history for v4.11..v4.12 uncovered: > >> >> > >> >> commit a1b89132dc4f61071bdeaab92ea958e0953380a1 > >> >> Author: Tim Murray <timmurray@google.com> > >> >> Date: Fri Apr 21 11:11:36 2017 +0200 > >> >> > >> >> dm crypt: use WQ_HIGHPRI for the IO and crypt workqueues > >> >> > >> >> Running dm-crypt with workqueues at the standard priority results in IO > >> >> competing for CPU time with standard user apps, which can lead to > >> >> pipeline bubbles and seriously degraded performance. Move to using > >> >> WQ_HIGHPRI workqueues to protect against that. > >> >> > >> >> Signed-off-by: Tim Murray <timmurray@google.com> > >> >> Signed-off-by: Enric Balletbo i Serra <enric.balletbo@collabora.com> > >> >> Signed-off-by: Mike Snitzer <snitzer@redhat.com> > >> >> > >> >> --- > >> >> > >> >> Reverting a1b8913 from 4.14.0-rc6, my current kernel, eliminates the > >> >> problem completely. > >> >> > >> >> Looking at the diff in that commit, it looks like the commit message isn't > >> >> even accurate; not only is the priority of the dmcrypt workqueues being > >> >> changed - they're also being made "CPU intensive" workqueues as well. > >> >> > >> >> This combination appears to result in both elevated scheduling priority and > >> >> greater quantity of participant worker threads effectively starving any > >> >> normal priority user task under periods of heavy IO on dmcrypt volumes. > >> >> > >> >> I don't know what the right solution is here. It seems to me we're lacking > >> >> the appropriate mechanism for charging CPU resources consumed on behalf of > >> >> user processes in kworker threads to the work-causing process. > >> >> > >> >> What effectively happens is my normal `git` user process is able to > >> >> greatly amplify what share of CPU it takes from the system by generating IO > >> >> on what happens to be a high-priority CPU-intensive storage volume. > >> >> > >> >> It looks potentially complicated to fix properly, but I suspect at its core > >> >> this may be a fairly longstanding shortcoming of the page cache and its > >> >> asynchronous design. Something that has been exacerbated substantially by > >> >> the introduction of CPU-intensive storage subsystems like dmcrypt. > >> >> > >> >> If we imagine the whole stack simplified, where all the IO was being done > >> >> synchronously in-band, and the dmcrypt kernel code simply ran in the > >> >> IO-causing process context, it would be getting charged to the calling > >> >> process and scheduled accordingly. The resource accounting and scheduling > >> >> problems all emerge with the page cache, buffered IO, and async background > >> >> writeback in a pool of unrelated worker threads, etc. That's how it > >> >> appears to me anyways... > >> >> > >> >> The system used is a X61s Thinkpad 1.8Ghz with 840 EVO SSD, lvm on dmcrypt. > >> >> The kernel .config is attached in case it's of interest. > >> >> > >> >> Thanks, > >> >> Vito Caputo > >> > > >> > > >> > > >> > Ping... > >> > > >> > Could somebody please at least ACK receiving this so I'm not left wondering > >> > if my mails to lkml are somehow winding up flagged as spam, thanks! > >> > >> Sorry I did not notice your email before you ping me directly. It's > >> interesting that issue, though we didn't notice this problem. It's a > >> bit far since I tested this patch but I'll setup the environment again > >> and do more tests to understand better what is happening. > >> > > > > Any update on this? > > > > I did not reproduce the issue for now. Can you try what happens if you > remove the WQ_CPU_INTENSIVE in the kcryptd_io workqueue? > > - cc->io_queue = alloc_workqueue("kcryptd_io", WQ_HIGHPRI | > WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM, 1); > cc->io_queue = alloc_workqueue("kcryptd_io", WQ_HIGHPRI | WQ_MEM_RECLAIM, 1); >
FYI I also tried just removing WQ_HIGHPRI, retaining WQ_CPU_INTENSIVE, also bad results.
So far just reverting a1b8913 has been the best solution.
I haven't studied the dmcrypt code, is there reason to observe the effects of these changes on both the workqueues touched by a1b8913 instead of just kcryptd_io?
Regards, Vito Caputo
| |