Messages in this thread | | | Date | Mon, 24 Nov 2008 13:46:14 -0800 | From | Max Krasnyansky <> | Subject | Re: RT sched: cpupri_vec lock contention with def_root_domain and no load balance |
| |
Li Zefan wrote: > Max Krasnyansky wrote: >> Dimitri Sivanich wrote: >>> kernel: CPU3 root domain e0000069ecb20000 >>> kernel: CPU3 attaching sched-domain: >>> kernel: domain 0: span 3 level NODE >>> kernel: groups: 3 >>> kernel: CPU2 root domain e000006884a00000 >>> kernel: CPU2 attaching sched-domain: >>> kernel: domain 0: span 2 level NODE >>> kernel: groups: 2 >>> kernel: CPU1 root domain e000006884a20000 >>> kernel: CPU1 attaching sched-domain: >>> kernel: domain 0: span 1 level NODE >>> kernel: groups: 1 >>> kernel: CPU0 root domain e000006884a40000 >>> kernel: CPU0 attaching sched-domain: >>> kernel: domain 0: span 0 level NODE >>> kernel: groups: 0 >>> >>> Which is the way sched_load_balance is supposed to work. You need to set >>> sched_load_balance=0 for all cpusets containing any cpu you want to disable >>> balancing on, otherwise some balancing will happen. >> It won't be much of a balancing in this case because this just one cpu per >> domain. >> In other words no that's not how it supposed to work. There is code in >> cpu_attach_domain() that is supposed to remove redundant levels >> (sd_degenerate() stuff). There is an explicit check in there for numcpus == 1. >> btw The reason you got a different result that I did is because you have a >> NUMA box where is mine is UMA. I was able to reproduce the problem though by >> enabling multi-core scheduler. In which case I also get one redundant domain >> level CPU, with a single CPU in it. >> So we definitely need to fix this. I'll try to poke around tomorrow and figure >> out why redundant level is not dropped. >> > > You were not using latest kernel, were you? > > There was a bug in sd degenerate code, and it has already been fixed: > http://lkml.org/lkml/2008/11/8/10 Ah, makes sense. The funny part is that I did see the patch before but completely forgot about it :).
>>> So when we do that for just par3, we get the following: >>> echo 0 > par3/cpuset.sched_load_balance >>> kernel: cpusets: rebuild ndoms 3 >>> kernel: cpuset: domain 0 cpumask >>> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0 >>> 0000000,00000000,00000000,00000000,0 >>> kernel: cpuset: domain 1 cpumask >>> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0 >>> 0000000,00000000,00000000,00000000,0 >>> kernel: cpuset: domain 2 cpumask >>> 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0 >>> 0000000,00000000,00000000,00000000,0 >>> kernel: CPU3 root domain default >>> kernel: CPU3 attaching NULL sched-domain. >>> >>> So the def_root_domain is now attached for CPU 3. And we do have a NULL >>> sched-domain, which we expect for a cpu with load balancing turned off. If >>> we turn sched_load_balance off ('0') on each of the other cpusets (par0-2), >>> each of those cpus would also have a NULL sched-domain attached. >> Ok. This one is a bug in cpuset.c:generate_sched_domains(). Sched domain >> generator in cpusets should not drop domains with single cpu in them when >> sched_load_balance==0. I'll look at that tomorrow too. >> > > Do you mean the correct behavior should be as following? > kernel: cpusets: rebuild ndoms 4 Yes.
> But why do you think this is a bug? In generate_sched_domains(), cpusets with > sched_load_balance==0 will be skippped: > > list_add(&top_cpuset.stack_list, &q); > while (!list_empty(&q)) { > ... > if (is_sched_load_balance(cp)) { > csa[csn++] = cp; > continue; > } > ... > } > > Correct me if I misunderstood your point. The problem is that all cpus in cpusets with sched_load_balance==0 end up in the default root_domain which causes lock contention. We can fix it either in sched.c:partition_sched_domains() or in cpusets.c:generate_sched_domains(). I'd rather fix cpusets because sched.c fix will be sub-optimal. See my answer to Greg on the same thread. Basically the scheduler code would have to allocate a root_domain for each CPU even on transitional states. So I'd rather fix cpusets to generate domain for each non-overlapping cpuset regardless of the sched_load_balance flag.
Max
| |