Messages in this thread | | | Date | Sat, 9 Feb 2008 08:01:44 +0100 | From | Ingo Molnar <> | Subject | Re: [git pull] latency tracer |
| |
* Ingo Molnar <mingo@elte.hu> wrote:
> It does include one very interesting new feature that deserves to be > mentioned outside of the shortlog: 'dynamic ftrace' - which is a > transparent kernel-image-patcher mechanism that lazily patches out > mcount callsites from all functions that get executed. [if tracing is > disabled] These patched out callsites are remembered, and are patched > back in when tracing is enabled. > > This technique does not just accelelerate the "tracing disabled" case > enormously, we were in fact unable to measure _any_ performance > difference (within noise) between an mcount-enabled dyn-ftrace [but > tracing-disabled] and a vanilla kernel (!), on modern CPUs.
hot off the presses: Steve has just finished doing a complete set of lmbench and hbackbench runs (on 2.8GHz Xeons, dual socket). The comparison results are at:
http://rostedt.homelinux.com/dyn-ftrace-lmbench/lmbench-table
the results show zero lmbench/hackbench overhead from dyn-ftrace (with thousands of functions instrumented) apart of the usual lmbench jitter.
The core kernel results for example [these used to be rather sensitive on mcount overhead]:
Host OS Mhz null null open slct sig sig fork exec sh call I/O stat clos TCP inst hndl proc proc proc --------- ------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ftrace-jm Linux 2.6.24 2783 0.65 0.74 4.74 6.96 5.28 0.76 4.29 290. 866. 2626 ftrace-no Linux 2.6.24 2784 0.65 0.74 4.75 7.14 5.31 0.76 3.92 282. 837. 2591 no-patche Linux 2.6.24 2786 0.64 0.71 4.86 7.10 5.34 0.78 3.89 283. 829. 2572 patched-n Linux 2.6.24 2784 0.64 0.71 4.72 6.98 5.35 0.78 3.94 284. 832. 2582
"ftrace-jm": dyn-ftrace, 2-byte JMPs patched in the mcount callsite "ftrace-no": dyn-ftrace, 5-byte NOPs patched in (devel patch, not in this pull request) "no-patche": no patches [vanilla Linus -git kernel] "patched-n": patched with the ftrace tree, all tracer options off
i can see an outlier in the "Create" results - that's noise i think because the patched+options-off kernel is in essence the same as vanilla. "File reread" is noisy and unreliable as ever. 16-task ctx-switch results are noisy too (especially the 64k working set ones) - this is a HT system hence the multi-threaded results are fundamentally noisy due to sibling interaction.
The hackbench results look good too.
enabling full tracing of all the tens of thousands of kernel functions makes things measurably slower - but that is expected. (filtered patching of a user-selected list of function names is an upcoming feature)
the full raw results are at:
http://rostedt.homelinux.com/dyn-ftrace-lmbench/
Ingo
| |