[lkml]   [2010]   [Feb]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    Patch in this message
    Subject[PATCH 1/7] ftrace: Remove memory barriers from NMI code when not needed
    From: Steven Rostedt <>

    The code in stop_machine that modifies the kernel text has a bit
    of logic to handle the case of NMIs. stop_machine does not prevent
    NMIs from executing, and if an NMI were to trigger on another CPU
    as the modifying CPU is changing the NMI text, a GPF could result.

    To prevent the GPF, the NMI calls ftrace_nmi_enter() which may
    modify the code first, then any other NMIs will just change the
    text to the same content which will do no harm. The code that
    stop_machine called must wait for NMIs to finish while it changes
    each location in the kernel. That code may also change the text
    to what the NMI changed it to. The key is that the text will never
    change content while another CPU is executing it.

    To make the above work, the call to ftrace_nmi_enter() must also
    do a smp_mb() as well as atomic_inc(). But for applications like
    perf that require a high number of NMIs for profiling, this can have
    a dramatic effect on the system. Not only is it doing a full memory
    barrier on both nmi_enter() as well as nmi_exit() it is also
    modifying a global variable with an atomic operation. This kills
    performance on large SMP machines.

    Since the memory barriers are only needed when ftrace is in the
    process of modifying the text (which is seldom), this patch
    adds a "modifying_code" variable that gets set before stop machine
    is executed and cleared afterwards.

    The NMIs will check this variable and store it in a per CPU
    "save_modifying_code" variable that it will use to check if it
    needs to do the memory barriers and atomic dec on NMI exit.

    Acked-by: Peter Zijlstra <>
    Signed-off-by: Steven Rostedt <>
    arch/x86/kernel/ftrace.c | 26 ++++++++++++++++++++++++++
    1 files changed, 26 insertions(+), 0 deletions(-)

    diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
    index 3096892..605ef19 100644
    --- a/arch/x86/kernel/ftrace.c
    +++ b/arch/x86/kernel/ftrace.c
    @@ -30,14 +30,32 @@


    + * modifying_code is set to notify NMIs that they need to use
    + * memory barriers when entering or exiting. But we don't want
    + * to burden NMIs with unnecessary memory barriers when code
    + * modification is not being done (which is most of the time).
    + *
    + * A mutex is already held when ftrace_arch_code_modify_prepare
    + * and post_process are called. No locks need to be taken here.
    + *
    + * Stop machine will make sure currently running NMIs are done
    + * and new NMIs will see the updated variable before we need
    + * to worry about NMIs doing memory barriers.
    + */
    +static int modifying_code __read_mostly;
    +static DEFINE_PER_CPU(int, save_modifying_code);
    int ftrace_arch_code_modify_prepare(void)
    + modifying_code = 1;
    return 0;

    int ftrace_arch_code_modify_post_process(void)
    + modifying_code = 0;
    return 0;
    @@ -149,6 +167,11 @@ static void ftrace_mod_code(void)

    void ftrace_nmi_enter(void)
    + __get_cpu_var(save_modifying_code) = modifying_code;
    + if (!__get_cpu_var(save_modifying_code))
    + return;
    if (atomic_inc_return(&nmi_running) & MOD_CODE_WRITE_FLAG) {
    @@ -160,6 +183,9 @@ void ftrace_nmi_enter(void)

    void ftrace_nmi_exit(void)
    + if (!__get_cpu_var(save_modifying_code))
    + return;
    /* Finish all executions before clearing nmi_running */

     \ /
      Last update: 2010-02-25 20:47    [W:0.043 / U:268.576 seconds]
    ©2003-2017 Jasper Spaans. hosted at Digital OceanAdvertise on this site