Messages in this thread Patch in this message | | | From | Denys Vlasenko <> | Subject | [PATCH 2/4] x86/asm/entry/64: do not TRACE_IRQS fast SYSRET64 path | Date | Wed, 25 Mar 2015 18:18:14 +0100 |
| |
SYSRET code path has a small irq-off block. On this code path, TRACE_IRQS_ON can't be called right before interrupts are enabled for real, we can't clobber registers there. So current code does it earlier, in a safe place.
But with this, TRACE_IRQS_OFF/ON frames just two fast instructions, which is ridiculous: now most of irq-off block is _outside_ of the framing.
Do the same thing that we do on SYSCALL entry: do not track this irq-off block, it is very small to ever cause noticeable irq latency.
Be careful: make sure that "jnz int_ret_from_sys_call_irqs_off" now does invoke TRACE_IRQS_OFF - move int_ret_from_sys_call_irqs_off label before TRACE_IRQS_OFF.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com> CC: Linus Torvalds <torvalds@linux-foundation.org> CC: Steven Rostedt <rostedt@goodmis.org> CC: Ingo Molnar <mingo@kernel.org> CC: Borislav Petkov <bp@alien8.de> CC: "H. Peter Anvin" <hpa@zytor.com> CC: Andy Lutomirski <luto@amacapital.net> CC: Oleg Nesterov <oleg@redhat.com> CC: Frederic Weisbecker <fweisbec@gmail.com> CC: Alexei Starovoitov <ast@plumgrid.com> CC: Will Drewry <wad@chromium.org> CC: Kees Cook <keescook@chromium.org> CC: x86@kernel.org CC: linux-kernel@vger.kernel.org --- arch/x86/kernel/entry_64.S | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S index 40f7760..11b7339 100644 --- a/arch/x86/kernel/entry_64.S +++ b/arch/x86/kernel/entry_64.S @@ -268,8 +268,11 @@ system_call_fastpath: * Has incompletely filled pt_regs. */ LOCKDEP_SYS_EXIT + /* + * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, + * it is too small to ever cause noticeable irq latency. + */ DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF /* * We must check ti flags with interrupts (or at least preemption) @@ -283,10 +286,7 @@ system_call_fastpath: jnz int_ret_from_sys_call_irqs_off /* Go to the slow path */ CFI_REMEMBER_STATE - /* - * sysretq will re-enable interrupts: - */ - TRACE_IRQS_ON + RESTORE_C_REGS_EXCEPT_RCX_R11 movq RIP(%rsp),%rcx CFI_REGISTER rip,rcx @@ -297,6 +297,7 @@ system_call_fastpath: * 64bit SYSRET restores rip from rcx, * rflags from r11 (but RF and VM bits are forced to 0), * cs and ss are loaded from MSRs. + * Restoration of rflags re-enables interrupts. */ USERGS_SYSRET64 @@ -345,8 +346,8 @@ tracesys_phase2: */ GLOBAL(int_ret_from_sys_call) DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF int_ret_from_sys_call_irqs_off: + TRACE_IRQS_OFF movl $_TIF_ALLWORK_MASK,%edi /* edi: mask to check */ GLOBAL(int_with_check) -- 1.8.1.4
| |