Messages in this thread | ![/](/images/icornerl.gif) | | From | Kyle Huey <> | Date | Fri, 21 Aug 2020 20:03:16 -0700 | Subject | Re: [REGRESSION] x86/cpu fsgsbase breaks TLS in 32 bit rr tracees on a 64 bit system |
| |
On Fri, Aug 21, 2020 at 7:53 PM Andy Lutomirski <luto@amacapital.net> wrote: > > > > > On Aug 21, 2020, at 2:33 PM, Kyle Huey <me@kylehuey.com> wrote: > > > > On Fri, Aug 21, 2020 at 1:08 PM Bae, Chang Seok > > <chang.seok.bae@intel.com> wrote: > >> > >> > >>>> On Aug 20, 2020, at 21:41, Kyle Huey <me@kylehuey.com> wrote: > >>> > >>> On the x86-64 5.9-rc1 TLS is completely broken in 32 bit tracees when > >>> running under rr[0]. Booting the kernel with `nofsgsbase` fixes it and > >>> I bisected to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=673903495c85137791d5820d690229efe09c8f7b. > >>> > >>> STR: > >>> 1. Build rr from source by > >>> a. git clone https://github.com/mozilla/rr > >>> b. mkdir rr/obj > >>> c. cd rr/obj > >>> d. cmake .. > >>> e. make -j16 > >>> 2. Run the simple 32 bit tracee outside of rr with `./bin/simple_32`. > >>> It should print a message and exit cleanly. > >>> 3. Run it under rr with `./bin/rr ./bin/simple_32`. > >>> > >>> It should behave the same way, but with fsgsbase enabled it will > >>> segfault. The `simple_32` binary is a simple "hello world" type > >>> program but it does link to pthreads, so pre-main code attempts to > >>> access TLS variables. > >>> > >>> The interplay between 32 bit and 64 bit TLS is dark magic to me > >>> unfortunately so this is all the useful information I have. > >> > >> As I run it and collect the ptrace logs, it starts to set FSBASE with > >> some numbers, e.g. 140632147826496, and then later attempts to set GS > >> with 99 through putreg(), not putreg32(). > >> > >> With FSGSBASE, the FS/GS base is decoupled from the selector. Andy > >> made putreg32() to retain the old behavior, fetching FS/GS base > >> according to the index, but the putreg() does not do. So, rr probably > >> relies on the old behavior as observed to setting the GS index only. > >> But it was previously considered to be okay with this comment [1]: > >> > >> "Our modifications to fs/gs/fs_base/gs_base are always either a) > >> setting values that the kernel set during recording to make them > >> happen during replay or b) emulating PTRACE_SET_REGS that a tracee > >> ptracer tried to set on another tracee. Either way I think the > >> effects are going to be the same as what would happen if the > >> program were run without rr." > >> > >> It is not straightforward to go into the rr internal to me. Robert, > >> any thought? > > > > Hmm. When we are running a 32 bit tracee in a 64 bit build of rr we > > internally convert between the 32 bit and 64 bit user_regs_structs[0] > > at the ptrace boundary. This does not preserve the fs/gsbase (because > > there is no fs/gsbase in the 32 bit user_regs_struct, of course). > > > > 40c45904f818c1f6555294ca27afc5fda4f09e68 added magic for a 32 bit > > tracer tracing a 32 bit tracee on a 64 bit kernel, but it looks like a > > 64 bit tracer tracing a 32 bit tracee on a 64 bit kernel *is* now > > expected to preserve the fs/gsbase values (or die, in our case). > > > > Is that correct? > > I was certainly not expecting rr to do this, and I thought I had asked in advance. What exact ptrace() calls are you doing here? Is this POKEUSER or something else? Breaking rr is at least impolite, and I’d like to fix this.
I believe we are PTRACE_GETREGSing and later PTRACE_SETREGSing, but doing the latter with garbage for fs/gs_base for a 32 bit tracee. That didn't used to matter (because those values were completely ignored for a 32 bit tracee) but now it does.
There's a good case that that's our fault and I'm happy to spend my "don't break userspace" points somewhere else ;)
- Kyle
| ![\](/images/icornerr.gif) |