Messages in this thread | | | Date | Thu, 31 Mar 2011 06:50:14 -0500 | From | Robin Holt <> | Subject | Re: [RFC 2/2] Make x86 calibrate_delay run in parallel. |
| |
On Thu, Mar 31, 2011 at 11:57:05AM +0200, Ingo Molnar wrote: > > * Robin Holt <holt@sgi.com> wrote: > > > On Thu, Mar 31, 2011 at 08:58:05AM +0200, Ingo Molnar wrote: > > > > > > * Ingo Molnar <mingo@elte.hu> wrote: > > > > > > > > > > > * Yinghai Lu <yinghai@kernel.org> wrote: > > > > > > > > > On Tue, Dec 14, 2010 at 5:58 PM, <Robin@sgi.com> wrote: > > > > > > > > > > > > On a 4096 cpu machine, we noticed that 318 seconds were taken for bringing > > > > > > up the cpus. By specifying lpj=<value>, we reduced that to 75 seconds. > > > > > > Andi Kleen suggested we rework the calibrate_delay calls to run in > > > > > > parallel. With that code in place, a test boot of the same machine took > > > > > > 61 seconds to bring the cups up. I am not sure how we beat the lpj= > > > > > > case, but it did outperform. > > > > > > > > > > > > One thing to note is the total BogoMIPS value is also consistently higher. > > > > > > I am wondering if this is an effect with the cores being in performance > > > > > > mode. I did notice that the parallel calibrate_delay calls did cause the > > > > > > fans on the machine to ramp up to full speed where the normal sequential > > > > > > calls did not cause them to budge at all. > > > > > > > > > > please check attached patch, that could calibrate correctly. > > > > > > > > > > Thanks > > > > > > > > > > Yinghai > > > > > > > > > [PATCH -v2] x86: Make calibrate_delay run in parallel. > > > > > > > > > > On a 4096 cpu machine, we noticed that 318 seconds were taken for bringing > > > > > up the cpus. By specifying lpj=<value>, we reduced that to 75 seconds. > > > > > Andi Kleen suggested we rework the calibrate_delay calls to run in > > > > > parallel. > > > > > > > > The risk wit that suggestion is that it will spectacularly miscalibrate on > > > > hyperthreading systems. > > > > I am not trying to be argumentative. I never got an understanding of > > what was going wrong with that earlier patch and am hoping for some > > understanding now. > > Well, if calibrate_delay() calls run in parallel then different hyperthreads > will impact each other. > > > Why does it spectacularly miscalibrate? Can anything be done to correct > > that miscalibration? Doesn't this patch indicate another problem with > > the calibration for hotplug cpus? Isn't there already a problem if > > you boot a cpu normally, then hot-remove a hyperthread of a cpu, run a > > userland task which fully loads up all the cores on that socket, then > > hot-add that hyperthread back in? If the lpj value is that volatile, > > what value does it really have? > > The typical CPU hotplug usecase is suspend/resume, where we bring down the CPUs > in a more or less controlled manner. > > Yes, you could achieve something similar by frobbing /sys/*/*/online but that's > a big difference to *always* running the calibration loops in parallel. > > I'd argue for the opposite direction: only calibrate a physical CPU once per > CPU per bootup - this would also make CPU hotplug faster btw. > > ( Virtual CPUs (KVM, etc.) need a recalibration per bringup, because the new > CPU could be running on different hardware - but that's a detail: 4096 UV > CPUs are not in this category. ) > > Really, there's no good reason why every CPU should be calibrated on a system > running identical CPUs, right? Mixed-frequency systems are rather elusive on > x86.
I had argued initially for calibrating a single core per socket earlier. I do not remember who indicated that would not work, but I do recall something about some AMD hardware possibly not having the same frequency for all cores. Do know any details about any offering where the individual cores on a socket can have different lpj values (other than calculation noise)?
Robin -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |