lkml.org 
[lkml]   [2017]   [Dec]   [16]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    Date
    From
    Subject[patch V149 43/50] x86/mm: Optimize RESTORE_CR3
    From: Peter Zijlstra <peterz@infradead.org>

    Most NMI/paranoid exceptions will not in fact change pagetables and would
    thus not require TLB flushing, however RESTORE_CR3 uses flushing CR3
    writes.

    Restores to kernel PCIDs can be NOFLUSH, because we explicitly flush the
    kernel mappings and now that we track which user PCIDs need flushing we can
    avoid those too when possible.

    This does mean RESTORE_CR3 needs an additional scratch_reg, luckily both
    sites have plenty available.

    Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
    Signed-off-by: Ingo Molnar <mingo@kernel.org>
    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
    Cc: Andy Lutomirski <luto@kernel.org>
    Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
    Cc: Borislav Petkov <bp@alien8.de>
    Cc: Brian Gerst <brgerst@gmail.com>
    Cc: Dave Hansen <dave.hansen@linux.intel.com>
    Cc: David Laight <David.Laight@aculab.com>
    Cc: Denys Vlasenko <dvlasenk@redhat.com>
    Cc: Eduardo Valentin <eduval@amazon.com>
    Cc: Greg KH <gregkh@linuxfoundation.org>
    Cc: H. Peter Anvin <hpa@zytor.com>
    Cc: Josh Poimboeuf <jpoimboe@redhat.com>
    Cc: Juergen Gross <jgross@suse.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Will Deacon <will.deacon@arm.com>
    Cc: aliguori@amazon.com
    Cc: daniel.gruss@iaik.tugraz.at
    Cc: hughd@google.com
    Cc: keescook@google.com
    ---
    arch/x86/entry/calling.h | 30 ++++++++++++++++++++++++++++--
    arch/x86/entry/entry_64.S | 4 ++--
    2 files changed, 30 insertions(+), 4 deletions(-)

    --- a/arch/x86/entry/calling.h
    +++ b/arch/x86/entry/calling.h
    @@ -281,8 +281,34 @@ For 32-bit we have the following convent
    .Ldone_\@:
    .endm

    -.macro RESTORE_CR3 save_reg:req
    +.macro RESTORE_CR3 scratch_reg:req save_reg:req
    ALTERNATIVE "jmp .Lend_\@", "", X86_BUG_CPU_SECURE_MODE_PTI
    +
    + ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID
    +
    + /*
    + * KERNEL pages can always resume with NOFLUSH as we do
    + * explicit flushes.
    + */
    + bt $X86_CR3_PTI_SWITCH_BIT, \save_reg
    + jnc .Lnoflush_\@
    +
    + /*
    + * Check if there's a pending flush for the user ASID we're
    + * about to set.
    + */
    + movq \save_reg, \scratch_reg
    + andq $(0x7FF), \scratch_reg
    + bt \scratch_reg, THIS_CPU_user_pcid_flush_mask
    + jnc .Lnoflush_\@
    +
    + btr \scratch_reg, THIS_CPU_user_pcid_flush_mask
    + jmp .Lwrcr3_\@
    +
    +.Lnoflush_\@:
    + SET_NOFLUSH_BIT \save_reg
    +
    +.Lwrcr3_\@:
    /*
    * The CR3 write could be avoided when not changing its value,
    * but would require a CR3 read *and* a scratch register.
    @@ -301,7 +327,7 @@ For 32-bit we have the following convent
    .endm
    .macro SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg:req save_reg:req
    .endm
    -.macro RESTORE_CR3 save_reg:req
    +.macro RESTORE_CR3 scratch_reg:req save_reg:req
    .endm

    #endif
    --- a/arch/x86/entry/entry_64.S
    +++ b/arch/x86/entry/entry_64.S
    @@ -1294,7 +1294,7 @@ ENTRY(paranoid_exit)
    testl %ebx, %ebx /* swapgs needed? */
    jnz .Lparanoid_exit_no_swapgs
    TRACE_IRQS_IRETQ
    - RESTORE_CR3 save_reg=%r14
    + RESTORE_CR3 scratch_reg=%rbx save_reg=%r14
    SWAPGS_UNSAFE_STACK
    jmp .Lparanoid_exit_restore
    .Lparanoid_exit_no_swapgs:
    @@ -1736,7 +1736,7 @@ ENTRY(nmi)
    movq $-1, %rsi
    call do_nmi

    - RESTORE_CR3 save_reg=%r14
    + RESTORE_CR3 scratch_reg=%r15 save_reg=%r14

    testl %ebx, %ebx /* swapgs needed? */
    jnz nmi_restore

    \
     
     \ /
      Last update: 2017-12-16 22:39    [W:4.293 / U:0.072 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site