Messages in this thread | | | Date | Wed, 27 Jan 2021 13:43:39 -0500 | From | Joel Fernandes <> | Subject | Re: [PATCH] sched/fair: Rate limit calls to update_blocked_averages() for NOHZ |
| |
Hi Vincent,
On Mon, Jan 25, 2021 at 03:42:41PM +0100, Vincent Guittot wrote: > On Fri, 22 Jan 2021 at 20:10, Joel Fernandes <joel@joelfernandes.org> wrote: > > On Fri, Jan 22, 2021 at 05:56:22PM +0100, Vincent Guittot wrote: > > > On Fri, 22 Jan 2021 at 16:46, Joel Fernandes (Google) > > > <joel@joelfernandes.org> wrote: > > > > > > > > On an octacore ARM64 device running ChromeOS Linux kernel v5.4, I found > > > > that there are a lot of calls to update_blocked_averages(). This causes > > > > the schedule loop to slow down to taking upto 500 micro seconds at > > > > times (due to newidle load balance). I have also seen this manifest in > > > > the periodic balancer. > > > > > > > > Closer look shows that the problem is caused by the following > > > > ingredients: > > > > 1. If the system has a lot of inactive CGroups (thanks Dietmar for > > > > suggesting to inspect /proc/sched_debug for this), this can make > > > > __update_blocked_fair() take a long time. > > > > > > Inactive cgroups are removed from the list so they should not impact > > > the duration > > > > I meant blocked CGroups. According to this code, a cfs_rq can be partially > > decayed and not have any tasks running on it but its load needs to be > > decayed, correct? That's what I meant by 'inactive'. I can reword it to > > 'blocked'. > > How many blocked cgroups have you got ?
I put a counter in for_each_leaf_cfs_rq_safe() { } to count how many times this loop runs per new idle balance. When the problem happens I see this loop run 35-40 times (for one single instance of newidle balance). So in total there are at least these many cfs_rq load updates.
I also see that new idle balance can be called 200-500 times per second.
> > > > * There can be a lot of idle CPU cgroups. Don't let fully > > * decayed cfs_rqs linger on the list. > > */ > > if (cfs_rq_is_decayed(cfs_rq)) > > list_del_leaf_cfs_rq(cfs_rq); > > > > > > 2. The device has a lot of CPUs in a cluster which causes schedutil in a > > > > shared frequency domain configuration to be slower than usual. (the load > > > > > > What do you mean exactly by it causes schedutil to be slower than usual ? > > > > sugov_next_freq_shared() is order number of CPUs in the a cluster. This > > system is a 6+2 system with 6 CPUs in a cluster. schedutil shared policy > > frequency update needs to go through utilization of other CPUs in the > > cluster. I believe this could be adding to the problem but is not really > > needed to optimize if we can rate limit the calls to update_blocked_averages > > to begin with. > > Qais mentioned half of the time being used by > sugov_next_freq_shared(). Are there any frequency changes resulting in > this call ?
I do not see a frequency update happening at the time of the problem. However note that sugov_iowait_boost() does run even if frequency is not being updated. IIRC, this function is also not that light weight and I am not sure if it is a good idea to call this that often.
> > > > average updates also try to update the frequency in schedutil). > > > > > > > > 3. The CPU is running at a low frequency causing the scheduler/schedutil > > > > code paths to take longer than when running at a high CPU frequency. > > > > > > Low frequency usually means low utilization so it should happen that much. > > > > It happens a lot as can be seen with schbench. It is super easy to reproduce. > > Happening a lot in itself is not a problem if there is nothing else to > do so it's not a argument in itself
It is a problem - it shows up in the preempt off critical section latency tracer. Are you saying its Ok for preemption to be disabled on system for 500 micro seconds? That hurts real-time applications (audio etc).
> So why is it a problem for you ? You are mentioning newly idle load > balance so I assume that your root problem is the scheduling delay > generated by the newly idle load balance which then calls > update_blocked_averages
Yes, the new idle balance is when I see it happen quite often. I do see it happen with other load balance as well, but it not that often as those LB don't run as often as new idle balance.
> > rate limiting the call to update_blocked_averages() will only reduce > the number of time it happens but it will not prevent it to happen.
Sure, but soft real-time issue can tolerate if the issue does not happen very often. In this case though, it is frequent.
> IIUC, your real problem is that newidle_balance is running whereas a > task is about to wake up on the cpu and we don't abort quickly during > this load_balance
Yes.
> so we could also try to abort earlier in case of newly idle load balance
I think interrupts are disabled when the load balance runs, so there's no way for say an audio interrupt to even run in order to wake up a task. How would you know to abort the new idle load balance?
Could you elaborate more also on the drawback of the rate limiting patch we posted? Do you see a side effect?
> > > > sometimes, which seems overkill. > > > > > > > > schbench shows a clear improvement with the change: > > > > > > Have you got more details about your test setup ? > > > which platform ? > > > which kernel ? > > > > I mentioned in the commit message it is a v5.4 kernel. > > I was not sure if the tests results done with this kernel because we > usually ask for results against mainline to make sure you are not > facing a problem that has solved since v5.4 has been released
Ok, yes I have a userspace up and running only on 5.4 kernel unfortunately. I was hoping that is recent enough for this debug.
thanks,
- Joel
| |