Messages in this thread | | | Date | Thu, 3 Mar 2016 16:18:03 -0500 | From | Paul Gortmaker <> | Subject | Re: runtime regression with "x86/mm/pat: Emulate PAT when it is disabled" |
| |
[runtime regression with "x86/mm/pat: Emulate PAT when it is disabled"] On 03/03/2016 (Thu 15:59) Paul Gortmaker wrote:
> So, the yocto folks moved from 4.1 to 4.4 and one of their automated > qemu x86-32 boot tests started failing. None of the yocto details seem > to matter since I offered to help and I've repropduced it using 100% > mainline kernels and a generic distro toolchain as well. > > The test case is slightly complicated, in that it relies on uvesafb > being modular, and so one has to juggle modules within an ext4 image > that qemu boots from. We tried making uvesafb builtin, but that made > the issue magically vanish. Given PAT, this isn't too surprising. > > Richard did the preliminary investigation and analysis, and from that I > did a bisect, and found the commit in $SUBJECT to be the root cause, as > per the discussion here: > > http://lists.openembedded.org/pipermail/openembedded-core/2016-March/118397.html > > I'd mentioned the above to bpetkov on IRC and after confirming it was > still an issue on 4.5-rc6, he'd asked if I had a portable reproducer. > > Not sure how complicated that would be, I set out to make one from my > build. With a little LD_PRELOAD type magic and ensuring all the qemu > components are in ./ I have one that runs on an otherwise qemu-free > x86-64 box. > > The stand alone reproducer is here; launched in 00-runme: > > http://openlinux.wrs.com/pat-splat/reproducer.tar.bz2
Apologies, I'd used an internal DNS abbreviation here that isn't global. Replace the wrs with windriver and everything should be good.
P. --
> > It is nothing fancy, just a generic yocto build of "sato" (gfx enabled > rootfs). When it "works" it boots to a UI touchscreen interface. When > it fails, you get a black screen with a blinking cursor (as seen in > "vncviewer localhost:0"). > > Upon failure, you can do <Ctrl>-<Alt>-<2> to get to a passwd-less root > login ; there you can run dmesg and see the splat. The image is > currently using 4.5-rc6 ; but any kernel can be inserted; "make > modules_install INSTALL_MOD_PATH=here" and then populating those modules > from "here" into /lib/modules of the loopback mounted image. And of > course updating the bzImage on the qemu cmdline. Currently it > contains a bzImage and modules for 4.5-rc6 as I last tested that. > > Also note that vncviewer will disconnect when it goes from early boot > 80x25 to a higer res gfx mode; just reconnect and continue observing the > target. > > I've ruled out yocto kernel changes, and yocto toolchain -- but maybe it > is a qemu issue this commit triggers ; who knows at this point. > > Since I've NFI what component(s) cause this, I wanted to have the qemu > binary, all libraries etc as part of the reproducer and nothing left to > chance, and I've tested the reproducer on an ancient dual core w/o vmx > and w/o any qemu binaries installed. Bruce also tested it on a slightly > more modern dual socket xeon with vmx and confirmed it failed there.. > > Inside there is a 00-runme ; mostly a copy of qemu args the yocto > automated tests were using. There is also everything the qemu binaries > need to run ; toplevel dir is noisy since qemu only looks in ./ it > seems. There is also an ext4.img ; as mentioned earlier, this only > happens when uvesafb.ko is a module, so one has to loopback mount that > image and repopulate /lib/modules/ for each boot test/bisect step. > > I've also included 00-bisect.txt as the output of git bisect log. And > there is also 00-configs/ dir that has the ".config" kernel file for > each build (dir names are "git describe" in here for easy correlation) > done for the bisect (plus the latest mainline build). The failing commit > in the subject is v4.1-rc5-22-g9cd25aac1f44 . > > My contribution here is largely a bisect that can be relied on and > providing a portable reproducer of the regression; I am by no means a > PAT expert ; Richard invested more time into actually understanding the > problem than I did, so I'm going to totally throw him under the bus on > this when it comes to considering the ultimate root cause and possible > fixes. :) > > Paul. > --
| |