[lkml]   [2012]   [Feb]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: ftrace_enabled set to 1 on bootup, slow downs with CONFIG_FUNCTION_TRACER in virt environments?
On Tue, 2012-02-14 at 10:29 -0500, Konrad Rzeszutek Wilk wrote: 
> Hey,
> I was running some benchmarks (netserver/netperf) where the init script just launched
> the netserver and nothing else and was concerned to see the performance not up to par.
> This was an HVM guest running with PV drivers.
> If I compile the kernel without CONFIG_FUNCTION_TRACER it is much better

There is a known performance degrade of 1 or 2% with function tracing
enabled, on some work loads. Anything more that needs to be

Did you also keep FRAME_POINTERS enabled? FUNCTION_TRACER selects frame
pointers which can also slow down the system.

> - but it was
> my understanding that the tracing code does not impact the machine unless it is enabled.
> And when I inserted a bunch of print_dump_bytes I do see instructions such as
> e8 6a 90 60 e1 get replaced with 66 66 66 90 so I see the the instructions getting
> patched over.

Right on boot up (and module load) the calls do get changed to nops. Now
note that there's some calls that do not get changed at boot up, but the
most recent scripts/recordmcount.c should change them to nops at compile
> To get a better feel for this I tried this on baremetal, and (this is going
> to sound a bit round-about way, but please bear with me), I was working on making
> the pte_flags be paravirt (so it is a function instead of being a macro) and noticed
> that on on an AMD A8-3850, with a CONFIG_PARAVIRT and CONFIG_FUNCTION_TRACER and
> running kernelbench it would run slower than without CONFIG_FUNCTION_TRACER.

Have you tried what the difference is between !CONFIG_PARAVIRT and with

> I am not really sure what the problem is, but based on those experiments
> four things come to my mind:
> - Lots of nops and we choke the CPU instruction decoder with 20-30 bytes
> of 'nop', so the CPU is stalling waiting for some real instructions.

But the nop is only placed at the beginning of functions.

> - The compiler has choosen to compile most of the paravirt instructions as
> functions making the call to mcount (which gets patched over), but the
> end result is that we have an extra 'call' in the chain.

You mean that we get a lot more functions because the compiler made them
functions? Maybe we should add "notrace" to all paravirt functions? Then
they wont have the calls or nops.

> - Somehow the low-level para-virt (like the assembler ones) calls don't get
> patched over and still end up calling mcount? (but I really doubt that is the
> case - but you never know).

We only live patch code in a white list of sections. But with the latest
scripts/recordmcount.c, as I stated above, the ones that don't get
patched at boot up, should be patched at compile time. But that still
keeps the nops there.

> - Something else?
> My thought was to crash the kernel as it is up and running and look at the
> diassembled core to see what the instructions end up looking to get a further feel
> for this. But before I go with this are there some other ideas of what I should look
> for?

You can just look at the objdump of vmlinux, as the recordmcount.c would
have already patched the code that is not whitelisted, and you can also
see if things are function calls.

-- Steve

> Thanks!
> Note: The "working on making the pte_flags be paravirt" patches are here:
> if you are interested.

 \ /
  Last update: 2012-02-15 18:47    [W:0.583 / U:0.200 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site