lkml.org 
[lkml]   [2014]   [Jun]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [bisected] pre-3.16 regression on open() scalability
On Thu, Jun 19, 2014 at 07:24:18AM +0200, Mike Galbraith wrote:
> On Wed, 2014-06-18 at 21:19 -0700, Paul E. McKenney wrote:
> > On Wed, Jun 18, 2014 at 08:38:16PM -0700, Andi Kleen wrote:
> > > On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote:
> > > > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote:
> > > > >
> > > > > I still think it's totally the wrong direction to pollute so
> > > > > many fast paths with this obscure debugging check workaround
> > > > > unconditionally.
> > > >
> > > > OOM prevention should count for something, I would hope.
> > >
> > > OOM in what scenario? This is getting bizarre.
> >
> > On the bizarre part, at least we agree on something. ;-)
> >
> > CONFIG_NO_HZ_FULL booted with at least one nohz_full CPU. Said CPU
> > gets into the kernel and stays there, not necessarily generating RCU
> > callbacks. The other CPUs are very likely generating RCU callbacks.
> > Because the nohz_full CPU is in the kernel, and because there are no
> > scheduling-clock interrupts on that CPU, grace periods do not complete.
> > Eventually, the callbacks from the other CPUs (and perhaps also some
> > from the nohz_full CPU, for that matter) OOM the machine.
> >
> > Now this scenario constitutes an abuse of CONFIG_NO_HZ_FULL, because it
> > is intended for CPUs that execute either in userspace (in which case
> > those CPUs are in extended quiescent states so that RCU can happily
> > ignore them) or for real-time workloads with low CPU untilization (in
> > which case RCU sees them go idle, which is also a quiescent state).
> > But that won't stop people from abusing their kernels and complaining
> > when things break.
>
> IMHO, those people can keep the pieces.
>
> I don't even enable RCU_BOOST in -rt kernels, because that safety net
> has a price. The instant Joe User picks up the -rt shovel, it's his
> grave, and he gets to do the digging. Instead of trying to save his
> bacon, I hand him a slightly better shovel, let him prioritize all
> kthreads including workqueues. Joe can dig all he wants to, and it's on
> him, I just make sure he has the means to bury himself properly :)

One of the nice things about NO_HZ_FULL is that it should reduce the
need for RCU_BOOST. One of the purposes of RCU_BOOST is for the guy
who has an infinite-loop bug in a high-priority RT-priority process,
because such processes can starve out low-priority RCU readers. But
with a properly configured NO_HZ_FULL system, the low-priority processes
aren't sharing CPUs with the RT-priority processes.

In fact, it might be worth making RCU_BOOST depend on !NO_HZ_FULL in -rt.

> > This same thing can also happen without CONFIG_NO_HZ full, though
> > the system has to work a bit harder. In this case, the CPU looping
> > in the kernel has scheduling-clock interrupts, but if all it does
> > is cond_resched(), RCU is never informed of any quiescent states.
> > The whole point of this patch is to make those cond_resched() calls,
> > which are quiescent states, visible to RCU.
> >
> > > If something keeps looping forever in the kernel creating
> > > RCU callbacks without any real quiescent states it's simply broken.
> >
> > I could get behind that. But by that definition, there is a lot of
> > breakage in the current kernel, especially as we move to larger CPU
> > counts.
>
> Not only larger CPU counts: skipping the -rq clock update on wakeup
> (cycle saving optimization) turned out to be deadly to boxen with a
> zillion disks because our wakeup latency can be so incredibly horrible
> that falsely attributing wakeup latency to the next task to run
> (watchdog) resulted in it being throttled for long enough that big IO
> boxen panicked during boot.
>
> The root cause of that wasn't the optimization, the root was the
> horrific amounts of time we can spend locked up in the kernel.

Completely agreed!

Thanx, Paul



\
 
 \ /
  Last update: 2014-06-19 20:41    [W:1.047 / U:0.348 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site