Messages in this thread | | | Date | Tue, 04 Nov 2008 09:34:44 -0500 | From | Gregory Haskins <> | Subject | Re: RT sched: cpupri_vec lock contention with def_root_domain and no load balance |
| |
Gregory Haskins wrote: > Peter Zijlstra wrote: > >> On Mon, 2008-11-03 at 15:07 -0600, Dimitri Sivanich wrote: >> >> >>> When load balancing gets switched off for a set of cpus via the >>> sched_load_balance flag in cpusets, those cpus wind up with the >>> globally defined def_root_domain attached. The def_root_domain is >>> attached when partition_sched_domains calls detach_destroy_domains(). >>> A new root_domain is never allocated or attached as a sched domain >>> will never be attached by __build_sched_domains() for the non-load >>> balanced processors. >>> >>> The problem with this scenario is that on systems with a large number >>> of processors with load balancing switched off, we start to see the >>> cpupri->pri_to_cpu->lock in the def_root_domain becoming contended. >>> This starts to become much more apparent above 8 waking RT threads >>> (with each RT thread running on it's own cpu, blocking and waking up >>> continuously). >>> >>> I'm wondering if this is, in fact, the way things were meant to work, >>> or should we have a root domain allocated for each cpu that is not to >>> be part of a sched domain? Note the the def_root_domain spans all of >>> the non-load-balanced cpus in this case. Having it attached to cpus >>> that should not be load balancing doesn't quite make sense to me. >>> >>> >> It shouldn't be like that, each load-balance domain (in your case a >> single cpu) should get its own root domain. Gregory? >> >> > > Yeah, this sounds broken. I know that the root-domain code was being > developed coincident to some upheaval with the cpuset code, so I suspect > something may have been broken from the original intent. I will take a > look. > > -Greg > >
After thinking about it some more, I am not quite sure what to do here. The root-domain code was really designed to be 1:1 with a disjoint cpuset. In this case, it sounds like all the non-balanced cpus are still in one default cpuset. In that case, the code is correct to place all those cores in the singleton def_root_domain. The question really is: How do we support the sched_load_balance flag better?
I suppose we could go through the scheduler code and have it check that flag before consulting the root-domain. Another alternative is to have the sched_load_balance=false flag create a disjoint cpuset. Any thoughts?
-Greg
[unhandled content-type:application/pgp-signature] | |