Messages in this thread | | | Date | Thu, 19 Jun 2014 11:14:04 -0700 | From | "Paul E. McKenney" <> | Subject | Re: [bisected] pre-3.16 regression on open() scalability |
| |
On Thu, Jun 19, 2014 at 07:24:18AM +0200, Mike Galbraith wrote: > On Wed, 2014-06-18 at 21:19 -0700, Paul E. McKenney wrote: > > On Wed, Jun 18, 2014 at 08:38:16PM -0700, Andi Kleen wrote: > > > On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote: > > > > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > > > > > > > I still think it's totally the wrong direction to pollute so > > > > > many fast paths with this obscure debugging check workaround > > > > > unconditionally. > > > > > > > > OOM prevention should count for something, I would hope. > > > > > > OOM in what scenario? This is getting bizarre. > > > > On the bizarre part, at least we agree on something. ;-) > > > > CONFIG_NO_HZ_FULL booted with at least one nohz_full CPU. Said CPU > > gets into the kernel and stays there, not necessarily generating RCU > > callbacks. The other CPUs are very likely generating RCU callbacks. > > Because the nohz_full CPU is in the kernel, and because there are no > > scheduling-clock interrupts on that CPU, grace periods do not complete. > > Eventually, the callbacks from the other CPUs (and perhaps also some > > from the nohz_full CPU, for that matter) OOM the machine. > > > > Now this scenario constitutes an abuse of CONFIG_NO_HZ_FULL, because it > > is intended for CPUs that execute either in userspace (in which case > > those CPUs are in extended quiescent states so that RCU can happily > > ignore them) or for real-time workloads with low CPU untilization (in > > which case RCU sees them go idle, which is also a quiescent state). > > But that won't stop people from abusing their kernels and complaining > > when things break. > > IMHO, those people can keep the pieces. > > I don't even enable RCU_BOOST in -rt kernels, because that safety net > has a price. The instant Joe User picks up the -rt shovel, it's his > grave, and he gets to do the digging. Instead of trying to save his > bacon, I hand him a slightly better shovel, let him prioritize all > kthreads including workqueues. Joe can dig all he wants to, and it's on > him, I just make sure he has the means to bury himself properly :)
One of the nice things about NO_HZ_FULL is that it should reduce the need for RCU_BOOST. One of the purposes of RCU_BOOST is for the guy who has an infinite-loop bug in a high-priority RT-priority process, because such processes can starve out low-priority RCU readers. But with a properly configured NO_HZ_FULL system, the low-priority processes aren't sharing CPUs with the RT-priority processes.
In fact, it might be worth making RCU_BOOST depend on !NO_HZ_FULL in -rt.
> > This same thing can also happen without CONFIG_NO_HZ full, though > > the system has to work a bit harder. In this case, the CPU looping > > in the kernel has scheduling-clock interrupts, but if all it does > > is cond_resched(), RCU is never informed of any quiescent states. > > The whole point of this patch is to make those cond_resched() calls, > > which are quiescent states, visible to RCU. > > > > > If something keeps looping forever in the kernel creating > > > RCU callbacks without any real quiescent states it's simply broken. > > > > I could get behind that. But by that definition, there is a lot of > > breakage in the current kernel, especially as we move to larger CPU > > counts. > > Not only larger CPU counts: skipping the -rq clock update on wakeup > (cycle saving optimization) turned out to be deadly to boxen with a > zillion disks because our wakeup latency can be so incredibly horrible > that falsely attributing wakeup latency to the next task to run > (watchdog) resulted in it being throttled for long enough that big IO > boxen panicked during boot. > > The root cause of that wasn't the optimization, the root was the > horrific amounts of time we can spend locked up in the kernel.
Completely agreed!
Thanx, Paul
| |