Messages in this thread |  | | | Date | Thu, 6 Mar 2008 07:47:29 -0600 | | From | Paul Jackson <> | | Subject | Re: [RFC/PATCH] cpuset: cpuset irq affinities |
| |
Peter wrote: > How about we make this in-kernel boot set, that by default contains all > IRQs, all unbounded kthreads and all of user-space.
If I understood your proposal, the /cgroup/irq cpuset is rather an odd cpuset -- it seems to be just used to place the 'system' irqs on the specified CPUs. This is not the usual use of cpusets, which are to associate a set of tasks with some resources. In your example, the 'tasks' file in /cgroup/irqs is probably empty, right?
And then you have to argue that certain incompatibilities this causes are tolerable:
> (Upgrading a kernel would require distributing some new userspace > anyway, right? - and we could offer a .config option to disable the boot > set for those who do upgrade kernels without upgrading user-space).
No ... we normally do our best not to force apps to change source code, or where possible not even force them to recompile, to run on new kernels.
> So by providing a .config option for strict backward compatibility,
That sort of alternative is useless when dealing with the major distros. They choose one config for half or all of their entire market (perhaps distinguishing personal PC from server, but no more). They enable everything that doesn't break something they care about.
> > How does all this interact with /proc/irq/N/smp_affinity? > > Much the same way the cpuset cpus_allowed interacts with a task's > cpus_allowed. That is, cs->cpus_allowed is a mask on top of the provided > affinity.
Ok - something like that could be done, if we had to, just as a cpusets 'mems' masks mbind and set_mempolicy nodemasks, and a cpusets 'cpus' masks sched_setaffinity cpumasks.
Could be ... I suppose ... if we had to ... however ...
I suspect that the reason you had the odd /cgroup/irqs cpuset is that it wasn't clear what to do with the irqs of cpusets that had both: 1) overlapping cpus, and 2) specified different lists of irqs.
In my view, you are trying to: A] force the <<irq, cpu>, Boolean> mapping into linear lists, and B] then trying to force those linear lists into the cpuset hierarchy.
With each capability that we've added to cpusets, beginning with CPUs and Memory Nodes themselves, and more recently with sched_domains, I've strived to keep cpusets fully hierarchical and nestable, supporting arbitrary combinations of overlapping CPUs and Memory Nodes.
What we have here, in this cpuset-irq proposal, I claim, is another example of trying to put in a tree what is essentially flat.
No ... that last paragraph is wrong. It's worse than that.
The mapping of <cpu, irq> pairs to Boolean ("can we direct this irq to this CPU - yes or no?") is not flat; it's a two-dimensional matrix of Boolean.
So you're [A] squinting out the left eye, flattening the <irq, cpu> map to a linear list; then [B] squinting out the right eye and flattening the cpuset hierarchy into a flat world; and then exclaiming "Gee, these two worlds have similar shape, so let's join them!"
Why, why, why? Why such insanity?
What in tarnation are you trying to do, that's painful or impossible to do, with what we have now?
The only opportunity I've sensed in all this so far is some sort of Karnaugh-map factoring of the <<irq, cpu>, Boolean> matrix, as an improved representation of the simple, brute force, low level /proc/irq/N/smp_affinity map. Sets of irqs that held a common usage pattern could be named, and sets of CPUs that had similar irq needs could be named (or named lists of existing cpusets might be a suitable proxy for some, not all, such sets), and then this map could be represented using these named sets. Mind you, I don't see that big a gain from presenting a second way of looking at what we can already see by the current, simple-minded, way, because the elegance of being able to group together common clusters (e.g., the set of CPUs that all get the same set of RealTime IRQ's) of the existing smp_affinity map is offset by the added complexities of having to deal with two ways of saying the same thing. But perhaps I'm missing some real opportunity here to gain significant leverage on some problem. If that's so, then perhaps some new cgroup would be appropriate, representing this new interface, pairing named sets of irqs to sets with sets of CPUs. And perhaps, for -some- system configurations, it would make sense to use the cgroup capability to mount multiple cgroup subsystems together, in a common mount. Perhaps ... I haven't seen what would be sufficient motivation to justify such an effort, but I can imagine that such could succeed, if some need, as yet unforseen by myself, existed.
-- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.940.382.4214
|  |