[lkml]   [2017]   [May]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Patch in this message
Subject[PATCH v2 for-4.12-fixes 1/2] sched/fair: Use task_groups instead of leaf_cfs_rq_list to walk all cfs_rqs
Currently, rq->leaf_cfs_rq_list is a traversal ordered list of all
live cfs_rqs which have ever been active on the CPU; unfortunately,
this makes update_blocked_averages() O(total number of CPU cgroups)
which isn't scalable at all.

The next patch will make rq->leaf_cfs_rq_list only contain the cfs_rqs
which are currently active. In preparation, this patch converts users
which need to traverse all cfs_rqs to use task_groups list instead.

task_groups list is protected by its own lock and allows RCU protected
traversal and the order of operations guarantees that all online
cfs_rqs will be visited, but holding rq->lock won't protect against
iterating an already unregistered cfs_rq. However, the operations of
the two users that get converted - update_runtime_enabled() and
unthrottle_offline_cfs_rqs() - should be safe to perform on already
dead cfs_rqs, so adding rcu read protection around them should be

Note that print_cfs_stats() is not converted. The next patch will
change its behavior to print out only active cfs_rqs, which is
intended as there's not much point in printing out idle cfs_rqs.

v2: Dropped strong synchronization around removal and left
print_cfs_stats() unchanged as suggested by Peterz.

Signed-off-by: Tejun Heo <>
Cc: Ingo Molnar <>
Cc: Peter Zijlstra <>
Cc: Mike Galbraith <>
Cc: Paul Turner <>
Cc: Chris Mason <>

Updated the patch - pasted in your version and updated the description
accordingly. Please feel free to use this one or whichever you like.

Based on mainline and stable is cc'd.


kernel/sched/fair.c | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4644,22 +4644,32 @@ static void destroy_cfs_bandwidth(struct

static void __maybe_unused update_runtime_enabled(struct rq *rq)
- struct cfs_rq *cfs_rq;
+ struct task_group *tg;

- for_each_leaf_cfs_rq(rq, cfs_rq) {
- struct cfs_bandwidth *cfs_b = &cfs_rq->tg->cfs_bandwidth;
+ lockdep_assert_held(&rq->lock);
+ rcu_read_lock();
+ list_for_each_entry_rcu(tg, &task_groups, list) {
+ struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
+ struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)];

cfs_rq->runtime_enabled = cfs_b->quota != RUNTIME_INF;
+ rcu_read_unlock();

static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
- struct cfs_rq *cfs_rq;
+ struct task_group *tg;
+ lockdep_assert_held(&rq->lock);
+ rcu_read_lock();
+ list_for_each_entry_rcu(tg, &task_groups, list) {
+ struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)];

- for_each_leaf_cfs_rq(rq, cfs_rq) {
if (!cfs_rq->runtime_enabled)

@@ -4677,6 +4687,7 @@ static void __maybe_unused unthrottle_of
if (cfs_rq_throttled(cfs_rq))
+ rcu_read_unlock();

 \ /
  Last update: 2017-05-10 21:18    [W:0.092 / U:5.648 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site