Messages in this thread Patch in this message | | | Date | Thu, 10 Oct 2013 08:27:39 -0400 | From | "Steven Rostedt <> (by way of Steven Rostedt <>) (by way of Steven Rostedt" <> | Subject | [RFC][PATCH] x86: Lazy disabling of interrupts |
| |
[ Resending, as somehow Claws email, removed the quotes from "H. Peter Anvin", and that prevented LKML from receiving this ]
*** NOT FOR INCLUSION ***
What this does --------------
There's several locations in the kernel that disable interrupts and enable them rather quickly. Most likely an interrupt will not happen during this time frame. Instead of actually disabling interrupts, set a flag instead, and if an interrupt were to come in, it would see the flag set and return (keeping interrupts disabled for real). When the flag is cleared, it checks if an interrupt came in and if it did it simulates that interrupt.
Rational -------- I noticed in function tracing that disabling interrupts is quite expensive. To measure this, I ran the stack tracer and several runs of hackbench:
trace-cmd stack --start for i in `seq 10` ; do time ./hackbench 100; done &> output
The stack tracer uses function tracing to examine every function's stack as the function is executed. If it finds a stack larger than the last max stack, it records it. But most of the time it just does the check and returns. To do this safely (using per cpu variables), it disables preemption:
kernel/trace/trace_stack.c: stack_trace_call()
preempt_disable_notrace(); [...] check_stack(ip, &stack); [...] preempt_enable_notrace();
Most of the time, check_stack() just returns without doing anything as it is unlikely to hit a new max (it does very seldom), and shouldn't be an issue in the benchmarks.
Then I changed this code do be:
kernel/trace/trace_stack.c: stack_trace_call()
local_irq_save(flags); [...] check_stack(ip, &stack); [...] local_irq_restore(flags);
And ran the test again. This caused a very large performance hit.
Running on: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (4 cores HT enabled)
Here's the differences:
With preempt disable (10 runs):
Time from hackbench: avg=2.0462 std=0.181487189630563
System time (from time): avg=10.5879 std=0.862181477416443
With irq disable (10 runs):
Time from hackbench: avg=2.7082 std=0.12304308188598
System time (from time): avg=14.6807 std=0.313856814487116
A 32% performance hit when using irq disabling told me that this is something we could improve on in normal activities. That is, avoid disabling interrupts when possible. For the last couple of weeks I decided to implement a "lazy irq disable" to do this.
The Setup ---------
I only had to touch four functions that deal with interrupts:
o native_irq_enable() o native_irq_disable() o native_save_fl() o native_restore_fl()
As these are the basis for all other C functions that disable interrupts (ie. local_irq_save(), local_irq_disable(), spin_lock_irq(), etc) just modifying them made it much easier to implement.
I added raw_* versions of each that do the real enabling and disabling. Basically, the raw_* versions are what they currently do today.
Per CPU -------
I added a couple of per cpu variables:
o lazy_irq_disabled_flags o lazy_irq_func o lazy_irq_vector o lazy_irq_on
The lazy_irq_disabled_flags holds the state of the system. The flags are:
DISABLED - When set, irqs are considered disabled (whether they are for real or not).
TEMP_DISABLE - Set when coming from a trap or other assembly that disables interrupts to let the native_irq_enable() know that interrupts are really disabled, and enable them as well.
IDLE - Used to tell the native_* functions that we are going idle and to continue to do real interrupt disabling/enabling.
REAL_DISABLE - Set by interrupts themselves. When interrupts are running, (this includes softirqs), we enable and disable interrupts normally. No lazy disabling is done from interrupt context.
The lazy_irq_func holds the interrupt function that was to trigger when we were in lazy irq disabled mode with interrupts enabled. Explained below.
The lazy_irq_vector holds the orig_rax, which is the vector that the interrupt handler needs to know what interrupt vector was triggered. Saved for the same use as lazy_irq_func is.
Because preempt_disable is currently a task flag, we need a per_cpu version of it for the lazy irq disabling. When irqs are disabled, the process requires that preemption is also disabled, and we need to do this with a per_cpu flag. For now, lazy_irq_on is used, and acts just like preempt_count for preventing scheduling from taking place.
The Process -----------
Here's the basic idea of what happens.
When native_irq_disable() is called, if any flag but DISABLED is set, then real interrupts are disabled. Otherwise, if DISABLED is already set, then nothing needs to be done. The DISABLED flag gets set, and at that moment if an interrupt comes in, it wont call the handler.
If an interrupt comes in when DISABLED is set, it updates the lazy_irq_func and lazy_irq_vector and returns. But before calling iretq, it clears the X86_EFLAGS_IF bit in the flags location of the stack to keep interrupts disabled when returning. This prevents any other interrupt from coming in. At this moment, interrupts are disabled like they would be on a non lazy irq disabled system.
When native_irq_enable() is called, if a flag other than DISABLED is set then it checks if lazy_irq_func is set, if it is, it will simulate the irq, otherwise it just enables interrupts. If DISABLED is set then it clears the DISABLED flag, then checks if lazy_irq_func is set. If lazy_irq_func is set, then we know that an interrupt came in and disabled interrupts for real. We don't need to worry about a race with new interrupts as interrupts are disabled. Just clearing the flag and then doing the check is safe. If an interrupt came in after we cleared the flag (assuming no interrupt came in before, because that would have disabled interrupts), it would run the interrupt handler normally, and not set lazy_irq_func.
When lazy_irq_func is set, interrupts must have been disabled (bug if not). Then we simulate the interrupt. This is done by software creating the interrupt stack frame, changing the flags to re-enable interrupts, and then calling the interrupt handler that was saved by lazy_irq_func (adding the saved vector to the stack as well). When the interrupt handler returns, a jmp to ret_from_intr is called, which will do the same processing as a normal interrupt would do. As EFLAGS was updated to re-enable interrupt when it does the iretq, interrupts would then be atomically enabled.
Specialty Processing -------------------
Mostly this works well, but there were a few areas that needed some extra work.
Switch To ---------
The switch_to code was a bit problematic, as for some reason (I don't know why), flags are saved on the prev stack, and restored from the next stack. I would assume that gcc would not be depending on flags after as asm() call, which switch_to does. But this causes problems as we don't disable interrupts unless a interrupt comes in. One could come in just before the switch, and then after the switch interrupts can be enabled again.
To avoid issues, the flags for next are changed to always disable interrupts and sets the TEMP flag to let the next native_enable_irq() know interrupts are really disabled.
Return From Fork ----------------
Return from fork does a popf with interrupts disabled. Just to be safe, we keep interrupts disabled and set the TEMP flag when calling schedule_tail().
Traps -----
This was also a pain. As a trap can happen in interrupt context, kernel context, or user context. Basically, it can happen in any context. Here we use the TEMP flag again, and just keep interrupts disabled when entering the trap. But if the trap may not enable interrupts so we need to check if the TEMP flag is still set when exiting the trap.
We also need to update the regs->eflags to show interrupts disabled if the DISABLED flag is set. That's because traps may check this as well and we need to make sure traps do the right decisions based on these flags. Instead of changing all locations that check these flags, just update them.
I found it best to just keep the TEMP flag set if the DISABLED flag is set and return with interrupts disabled (no need to touch flags, as they already) were set on entry of the trap. If the trap enabled interrupts when the interrupts were disabled on entry, that would be bad normally, so I don't check for that case.
Idle ----
Idle was also a bit of a pain, as it disables interrupts when calling into the hardware, and the hardware will allow an interrupt to happen and return. To solve this, I added some functions that would check the state of the lazy_irq_disable and if a pending interrupt was there, just call the interrupt and not do the idle. Otherwise, set the IDLE flag and remove all other flags, as well as disable interrupts for real. When the IDLE flag is set, the native_irq_enable/disable() functions will just do the raw_ versions, until the IDLE flag gets cleared.
Results -------
Actually this was quite disappointing. After spending several days hacking this, and finally getting it running stable on bare metal, I was able to do some bench marks.
Doing the same thing with the stack tracer, the patched code for interrupts disabled was:
With irq disable (10 runs):
Time from hackbench: avg=2.3455 std=0.106322622240049
System time (from time): avg=12.306 std=0.568022886862844
Which is just a 14% slowdown compared to a 32% slowdown that the normal irq disabling had. This looks good right?
Well, unfortunately, not so much :-( The problem here is that we improved an unrealistic case. The stack tracer with interrupts disabling stresses the irqs disabled for ever single function called in the kernel. That's not normal operation.
Disabling stack tracer and running hackbench normally again, we have:
Unpatched:
Time from hackbench: avg=1.0657 std=0.0533488519089212
System time (from time): avg=4.2248 std=0.150524416624015
Patched:
Time from hackbench: avg=1.0523 std=0.046519888219986
System time (from time): avg=4.21 std=0.214256855199548
Yeah, it improved a little, but as we can see from the standard deviation, the difference is within the noise.
Now maybe hackbench isn't the best benchmark to be testing this with. Other benchmarks should be used. But I've already spent too much time on this, and even though I got it working, it needs a lot of clean up if it is even worth doing. Unless there's real world benchmarks out there that shows us that this makes a huge difference, this work may be just chalked up as an academic exercise, which actually wasn't a waste of time as I now understand x86 infrastructure a little bit more.
Hey, when you learn from code you wrote, even if it's never used by anyone, it is still worth doing just for that extra bit of knowledge you received. Knowledge does not come cheap.
Summary -------
Although the extreme case shows a nice improvement, I'm skeptical if it is worth doing for real world applications. But that said, I'm posting the code here as well as in my git repo. I'll give my SOB thus that anyone that wants to take it can build on it as long as they give me credit for what I've done.
My git repo is here: But note, the commits in the repo are not stages of patches. It's a hodgepodge of states the code went through. The good, the bad, the ugly (mostly the ugly). Thus, you can see where I screwed up and had to rewrite the code. Every time I got something working (or thought I got something working), I committed it. The end result here had a little clean up so those reading the patch wont be so confused.
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-rt.git
Branch: x86/irq-soft-disable-v4
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b32ebf9..789f691 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -17,6 +17,10 @@ config X86_64 depends on 64BIT select X86_DEV_DMA_OPS +config LAZY_IRQ_DISABLE + def_bool y + depends on 64BIT + ### Arch settings config X86 def_bool y diff --git a/arch/x86/include/asm/irqflags.h b/arch/x86/include/asm/irqflags.h index bba3cf8..9d089f4 100644 --- a/arch/x86/include/asm/irqflags.h +++ b/arch/x86/include/asm/irqflags.h @@ -3,12 +3,42 @@ #include <asm/processor-flags.h> +#undef CONFIG_LAZY_IRQ_DEBUG + +#define LAZY_IRQ_DISABLED_BIT 0 +#define LAZY_IRQ_TEMP_DISABLE_BIT 1 +#define LAZY_IRQ_IDLE_BIT 2 +#define LAZY_IRQ_REAL_DISABLE_BIT 3 + +#define LAZY_IRQ_FL_DISABLED (1 << LAZY_IRQ_DISABLED_BIT) +#define LAZY_IRQ_FL_TEMP_DISABLE (1 << LAZY_IRQ_TEMP_DISABLE_BIT) +#define LAZY_IRQ_FL_IDLE (1 << LAZY_IRQ_IDLE_BIT) +#define LAZY_IRQ_FL_REAL_DISABLE (1 << LAZY_IRQ_REAL_DISABLE_BIT) + #ifndef __ASSEMBLY__ +#include <linux/kernel.h> + +#ifdef CONFIG_LAZY_IRQ_DEBUG +void update_last_hard_enable(unsigned long addr); +void update_last_soft_enable(unsigned long addr); +void update_last_hard_disable(unsigned long addr); +void update_last_soft_disable(unsigned long addr); +void update_last_preempt_disable(unsigned long addr); +void update_last_preempt_enable(unsigned long addr); +#else +static inline void update_last_hard_enable(unsigned long addr) { } +static inline void update_last_soft_enable(unsigned long addr) { } +static inline void update_last_hard_disable(unsigned long addr) { } +static inline void update_last_soft_disable(unsigned long addr) { } +static inline void update_last_preempt_disable(unsigned long addr) { } +static inline void update_last_preempt_enable(unsigned long addr) { } +#endif + /* * Interrupt control: */ -static inline unsigned long native_save_fl(void) +static inline unsigned long raw_native_save_fl(void) { unsigned long flags; @@ -26,21 +56,32 @@ static inline unsigned long native_save_fl(void) return flags; } -static inline void native_restore_fl(unsigned long flags) +static inline void raw_native_restore_fl(unsigned long flags) { +#ifdef CONFIG_LAZY_IRQ_DEBUG + if ((raw_native_save_fl() ^ flags) & X86_EFLAGS_IF) { + if (flags & X86_EFLAGS_IF) + update_last_hard_enable((long)__builtin_return_address(0)); + else + update_last_hard_disable((long)__builtin_return_address(0)); + } +#endif + asm volatile("push %0 ; popf" : /* no output */ :"g" (flags) :"memory", "cc"); } -static inline void native_irq_disable(void) +static inline void raw_native_irq_disable(void) { asm volatile("cli": : :"memory"); + update_last_hard_disable((long)__builtin_return_address(0)); } -static inline void native_irq_enable(void) +static inline void raw_native_irq_enable(void) { + update_last_hard_enable((long)__builtin_return_address(0)); asm volatile("sti": : :"memory"); } @@ -54,8 +95,294 @@ static inline void native_halt(void) asm volatile("hlt": : :"memory"); } +#ifndef CONFIG_LAZY_IRQ_DISABLE +#define native_save_fl() raw_native_save_fl() +#define native_restore_fl(flags) raw_native_restore_fl(flags) +#define native_irq_disable() raw_native_irq_disable() +#define native_irq_enable() raw_native_irq_enable() +static inline lazy_irq_idle_enter(void) +{ + return 1; +} +static inline void lazy_irq_idle_exit(void) { } +static inline void print_lazy_debug(void) { } +static inline void print_lazy_irq(int line) { } +static inline void lazy_test_idle(void) { } +#else +#include <linux/bug.h> + +extern int lazy_irq_idle_enter(void); +extern void lazy_irq_idle_exit(void); + +void lazy_irq_bug(const char *file, int line, unsigned long flags, unsigned long raw); + +#ifdef CONFIG_LAZY_IRQ_DEBUG +static inline void do_preempt_disable(void) +{ + unsigned long val; + + asm volatile ("addq $1,%%gs:lazy_irq_on\n" + "movq %%gs:lazy_irq_on,%0\n" : "=r"(val) : : "memory"); + update_last_preempt_disable((long)__builtin_return_address(0)); +} + +static inline void do_preempt_enable(void) +{ + unsigned long val; + static int once; + + asm volatile ("movq %%gs:lazy_irq_on,%0\n" + "subq $1,%%gs:lazy_irq_on" : "=r"(val) : : "memory"); + if (!once && !val) { + once++; + lazy_irq_bug(__func__, __LINE__, val, val); + } + if (!once) + update_last_preempt_enable((long)__builtin_return_address(0)); +} + +void print_lazy_debug(void); +void print_lazy_irq(int line); +void lazy_test_idle(void); + +#else +/* + * As preempt_disable is still a task variable, we need to make + * it a per_cpu variable for our own purposes. This can be fixed + * when preempt_count becomes a per cpu variable. + */ +static inline void do_preempt_disable(void) +{ + asm volatile ("addq $1,%%gs:lazy_irq_on\n" : : : "memory"); +} + +static inline void do_preempt_enable(void) +{ + asm volatile ("subq $1,%%gs:lazy_irq_on" : : : "memory"); +} + +static inline void print_lazy_debug(void) { } +static inline void print_lazy_irq(int line) { } +static inline void lazy_test_idle(void) { } + +#endif /* CONFIG_LAZY_IRQ_DEBUG */ + +void lazy_irq_simulate(void *func); + +/* + * Unfortunatetly, due to include hell, we can't include percpu.h. + * Thus, we open code our fetching and changing of per cpu variables. + */ +static inline unsigned long get_lazy_irq_flags(void) +{ + unsigned long flags; + + asm volatile ("movq %%gs:lazy_irq_disabled_flags, %0" : "=r"(flags) :: ); + return flags; +} + +static inline void * get_lazy_irq_func(void) +{ + void *func; + + asm volatile ("movq %%gs:lazy_irq_func, %0" : "=r"(func) :: ); + return func; +} + +static inline unsigned long native_save_fl(void) +{ + unsigned long flags; + + /* + * It might be possible that if irqs are fully enabled + * we could migrate. But the result of this operation + * will be the same regardless if we move from one + * CPU to another. That is, if flags is not zero, we + * wont schedule, and we can only migrate if flags is + * zero, which means it will be zero after the migrate + * or scheduled back in. + */ + flags = get_lazy_irq_flags(); + + if (flags >> LAZY_IRQ_TEMP_DISABLE_BIT) + return raw_native_save_fl(); + + return flags & LAZY_IRQ_FL_DISABLED ? 0 : X86_EFLAGS_IF; +} + +/* + * Again, because of include hell, we can't include local.h, and + * we need to make sure we use a true "add" and "sub" that is + * atomic for the CPU. We can't have a load modify store, and + * I don't trust gcc enough to think it will do that for us. + */ +static inline void lazy_irq_sub(unsigned long val) +{ +#ifdef CONFIG_LAZY_IRQ_DEBUG + if (val > get_lazy_irq_flags()) + lazy_irq_bug(__func__, __LINE__, + get_lazy_irq_flags(), raw_native_save_fl()); #endif + asm volatile ("subq %0, %%gs:lazy_irq_disabled_flags" : : "r"(val) : "memory"); +} + +static inline void lazy_irq_add(unsigned long val) +{ + asm volatile ("addq %0, %%gs:lazy_irq_disabled_flags" : : "r"(val) : "memory"); +} + +static inline void lazy_irq_sub_temp(void) +{ + lazy_irq_sub(LAZY_IRQ_FL_TEMP_DISABLE); +} + +static inline void lazy_irq_add_temp(void) +{ + lazy_irq_add(LAZY_IRQ_FL_TEMP_DISABLE); +} + +static inline void lazy_irq_sub_disable(void) +{ + update_last_soft_enable((long)__builtin_return_address(0)); + lazy_irq_sub(LAZY_IRQ_FL_DISABLED); +} + +static inline void lazy_irq_add_disable(void) +{ + update_last_soft_disable((long)__builtin_return_address(0)); + lazy_irq_add(LAZY_IRQ_FL_DISABLED); +} + +static inline void native_irq_disable(void) +{ + unsigned long flags; + unsigned long raw; + + do_preempt_disable(); + flags = get_lazy_irq_flags(); + raw = raw_native_save_fl(); + + if (flags) { + /* Always disable for real not in lazy mode */ + if (flags >> LAZY_IRQ_TEMP_DISABLE_BIT) + raw_native_irq_disable(); + /* If flags is set, we already disabled preemption */ + do_preempt_enable(); + return; + } + +#ifdef CONFIG_LAZY_IRQ_DEBUG + if (!(raw & X86_EFLAGS_IF)) + lazy_irq_bug(__func__, __LINE__, flags, raw); +#endif + + lazy_irq_add_disable(); + /* Leave with preemption disabled */ +} + +static inline void native_irq_enable(void) +{ + unsigned long flags; + unsigned long raw; + void *func = NULL; + + flags = get_lazy_irq_flags(); + raw = raw_native_save_fl(); + + /* Do nothing if already enabled */ + if (!flags) + goto out; + + if (flags >> LAZY_IRQ_TEMP_DISABLE_BIT) { +#ifdef CONFIG_LAZY_IRQ_DEBUG + WARN_ON((flags & LAZY_IRQ_FL_IDLE) && (flags & LAZY_IRQ_FL_DISABLED)); + if ((flags & ~LAZY_IRQ_FL_IDLE) && raw_native_save_fl() & X86_EFLAGS_IF) + lazy_irq_bug(__func__, __LINE__, flags, raw); +#endif + /* + * If we temporary disabled interrupts, that means + * we did so from assembly, and we want to go back + * to lazy irq disable mode. + */ + if (flags & LAZY_IRQ_FL_TEMP_DISABLE) { + lazy_irq_sub_temp(); + /* + * If we are not in interrupt context, we need + * to enable irqs in lazy mode too when temp flag was set. + */ + if ((flags & ~LAZY_IRQ_FL_TEMP_DISABLE) == LAZY_IRQ_FL_DISABLED) + lazy_irq_sub_disable(); + } +#ifdef CONFIG_LAZY_IRQ_DEBUG + if (get_lazy_irq_flags() & LAZY_IRQ_FL_DISABLED) + lazy_irq_bug(__func__, __LINE__, flags, raw); +#endif + /* + * If func is set, then interrupts was disabled when coming + * in, or up to the point that we had the DISABLED flag set. + * We cleared it, so it is safe to read the func, as it only + * will be set when DISABLED flag set, and if that does happen + * interrupts will be disabled to prevent another interrupt + * coming in now. + */ + func = get_lazy_irq_func(); + if (func) + lazy_irq_simulate(func); /* enables interrupts */ + else + raw_native_irq_enable(); + goto out; + } + +#ifdef CONFIG_LAZY_IRQ_DEBUG + if (LAZY_IRQ_FL_DISABLED > get_lazy_irq_flags()) + lazy_irq_bug(__func__, __LINE__, flags, raw); +#endif + lazy_irq_sub_disable(); + /* + * Grab func *after* enabling lazy irqs, this prevents the race + * where we enable the lazy irq but a interrupt comes in when + * we do it and sets func. If an interrupt comes in after we + * clear the DISABLED flag, it will just run the interrupt normally. + */ + func = get_lazy_irq_func(); + + /* + * If func is set, then an interrupt came in when the DISABLED + * flag was set (it's no longer set), and interrupts will be + * really disabled because of that. In that case, we need to + * simulate the interrupt (which will enable interrupts too). + */ + if (func) { +#ifdef CONFIG_LAZY_IRQ_DEBUG + if (raw_native_save_fl() & X86_EFLAGS_IF) + lazy_irq_bug(__func__, __LINE__, flags, raw); +#endif + lazy_irq_simulate(func); + } + + do_preempt_enable(); +out: +#ifdef CONFIG_LAZY_IRQ_DEBUG + if (!(raw_native_save_fl() & X86_EFLAGS_IF)) { + printk("func=%pS flags=%lx\n", func, get_lazy_irq_flags()); + lazy_irq_bug(__func__, __LINE__, flags, raw); + } +#endif + return; +} + +static inline void native_restore_fl(unsigned long flags) +{ + if (flags & X86_EFLAGS_IF) + native_irq_enable(); + else + native_irq_disable(); +} +#endif /* CONFIG_LAZY_IRQ_DISABLE */ + +#endif /* !__ASSEMBLY__ */ + #ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else @@ -206,4 +533,5 @@ static inline int arch_irqs_disabled(void) # endif #endif /* __ASSEMBLY__ */ + #endif diff --git a/arch/x86/include/asm/switch_to.h b/arch/x86/include/asm/switch_to.h index 4ec45b3..d981812 100644 --- a/arch/x86/include/asm/switch_to.h +++ b/arch/x86/include/asm/switch_to.h @@ -1,6 +1,8 @@ #ifndef _ASM_X86_SWITCH_TO_H #define _ASM_X86_SWITCH_TO_H +#include <asm/irqflags.h> + struct task_struct; /* one of the stranger aspects of C forward declarations */ struct task_struct *__switch_to(struct task_struct *prev, struct task_struct *next); @@ -80,7 +82,25 @@ do { \ /* frame pointer must be last for get_wchan */ #define SAVE_CONTEXT "pushf ; pushq %%rbp ; movq %%rsi,%%rbp\n\t" -#define RESTORE_CONTEXT "movq %%rbp,%%rsi ; popq %%rbp ; popf\t" +#define RESTORE_CONTEXT "movq %%rbp,%%rsi ; popq %%rbp ; " LAZY_CONTEXT "popf\t" + +#ifdef CONFIG_LAZY_IRQ_DISABLE +/* + * When doing the context switch, the DISABLED flag should be set. + * But interrupts may not be disabled, and we may switch to having them + * disabled. Worse yet, they may be disabled and we are switching to having + * them enabled, and if we do that, a pending interrupt may be lost. + * The safest thing to do (for now) is to just set the TEMP flag and + * disable interrupts in the switch. This will cause the enabling to + * do the check for any interrupts that came in during the switch that + * we don't want to miss. + */ +#define LAZY_CONTEXT "andq $~(1<<9),(%%rsp); orq $" \ + __stringify(LAZY_IRQ_FL_TEMP_DISABLE) \ + ",%%gs:lazy_irq_disabled_flags\n\t" +#else +# define LAZY_CONTEXT +#endif #define __EXTRA_CLOBBER \ , "rcx", "rbx", "rdx", "r8", "r9", "r10", "r11", \ diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c index a698d71..93e0b42 100644 --- a/arch/x86/kernel/apic/hw_nmi.c +++ b/arch/x86/kernel/apic/hw_nmi.c @@ -72,6 +72,8 @@ arch_trigger_all_cpu_backtrace_handler(unsigned int cmd, struct pt_regs *regs) arch_spin_lock(&lock); printk(KERN_WARNING "NMI backtrace for cpu %d\n", cpu); + print_lazy_irq(__LINE__); + print_lazy_debug(); show_regs(regs); arch_spin_unlock(&lock); cpumask_clear_cpu(cpu, to_cpumask(backtrace_mask)); diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S index 1b69951..8201096 100644 --- a/arch/x86/kernel/entry_64.S +++ b/arch/x86/kernel/entry_64.S @@ -547,6 +547,11 @@ ENTRY(ret_from_fork) pushq_cfi $0x0002 popfq_cfi # reset kernel eflags +#ifdef CONFIG_LAZY_IRQ_DISABLE + /* Return from fork always disables interrupts for real. */ + orq $LAZY_IRQ_FL_TEMP_DISABLE, PER_CPU_VAR(lazy_irq_disabled_flags) +#endif + call schedule_tail # rdi: 'prev' task parameter GET_THREAD_INFO(%rcx) @@ -973,6 +978,144 @@ END(irq_entries_start) END(interrupt) .previous +#ifdef CONFIG_LAZY_IRQ_DISABLE + +#ifdef CONFIG_LAZY_IRQ_DEBUG + .macro LAZY_DEBUG int func=0 + pushq %rdi + pushq %rsi + movq 16(%rsp), %rsi + pushq %rdx + pushq %rcx + pushq %rax + pushq %r8 + pushq %r9 + pushq %r10 + pushq %r11 + pushq %rbx + pushq %rbp + pushq %r12 + pushq %r13 + pushq %r14 + pushq %r15 + movq $\int, %rdi + movq $\func, %rdx + call lazy_irq_debug + popq %r15 + popq %r14 + popq %r13 + popq %r12 + popq %rbp + popq %rbx + popq %r11 + popq %r10 + popq %r9 + popq %r8 + popq %rax + popq %rcx + popq %rdx + popq %rsi + popq %rdi + jmp 1f +1: + .endm +#if 0 + LAZY_DEBUG 0 +#endif +#endif /* CONFIG_LAZY_IRQ_DEBUG */ + + .macro LAZY_DISABLED_CHECK func + /* If in userspace, interrupts are always enabled */ + testl $3, 16(%rsp) /* CS is at offset 16 */ + jne 1f + bt $LAZY_IRQ_DISABLED_BIT, PER_CPU_VAR(lazy_irq_disabled_flags) + jnc 1f + pushq $\func + jmp irq_is_disabled +1: + .endm + + .macro LAZY_DISABLED_START + addq $LAZY_IRQ_FL_REAL_DISABLE, PER_CPU_VAR(lazy_irq_disabled_flags) + .endm + + .macro LAZY_DISABLED_DONE + subq $LAZY_IRQ_FL_REAL_DISABLE, PER_CPU_VAR(lazy_irq_disabled_flags) + .endm + + /* + * The lazy soft disabling of interrupts is for + * performance reasons, as enabling interrupts can + * have a cost. But when the hardware disables + * interrupts, it's rather pointless to use the soft + * disabling feature. + * + * When a trap is hit, interrupts are disabled. + * We set the TEMP flag to let the native_irq_enable() + * know to really enable interrupts. + */ + .macro LAZY_DISABLE_TRAP_ENTRY + testl $(~(LAZY_IRQ_FL_TEMP_DISABLE-1)), PER_CPU_VAR(lazy_irq_disabled_flags) + jne 1f + addq $LAZY_IRQ_FL_TEMP_DISABLE, PER_CPU_VAR(lazy_irq_disabled_flags) + + /* + * If interrupts are soft disabled, then make eflags disabled too. + * This is required because there's lots of places that read the + * regs->eflags to make decisions. Ideally, we should change all these + * places to test for the soft disable flag, but for now this is + * easier to do. But unfortunately, this is also the page fault handler + * which is going to kill all our efforts with the lazy irq disabling :( + */ + bt $LAZY_IRQ_DISABLED_BIT, PER_CPU_VAR(lazy_irq_disabled_flags) + jnc 1f + andq $~(1<<9), EFLAGS(%rsp) +1: + .endm + + .macro LAZY_DISABLE_TRAP_EXIT + /* + * If interrupts we soft disabled or really disabled, then we + * don't need to do anything. The TEMP flag will be set telling + * the next native_irq_enable() to enable interrupts for real. + * No need to enable them now. + * + * The trap really should not have cleared the TEMP flag, because + * that means it enabled interrupts when trapping from a interrupt + * disabled context, which would be really bad to do. + */ + bt $9, EFLAGS(%rsp) + jnc 1f + + /* + * EFLAGS has IRQs enabled, interrupts should be enabled both real + * and in lazy mode, just clear the TEMP flag if it is set + */ + bt $LAZY_IRQ_TEMP_DISABLE_BIT, PER_CPU_VAR(lazy_irq_disabled_flags) + jnc 1f + subq $LAZY_IRQ_FL_TEMP_DISABLE, PER_CPU_VAR(lazy_irq_disabled_flags) + +1: + .endm + +irq_is_disabled: + /* function is saved in stack */ + popq PER_CPU_VAR(lazy_irq_func) + /* Get the vector */ + popq PER_CPU_VAR(lazy_irq_vector) + + + andq $~(1 << 9), 16(%rsp) /* keep irqs disabled */ + jmp irq_return +#else + .macro LAZY_DISABLED_CHECK func + .endm +#define LAZY_DISABLED_START +#define LAZY_DISABLED_DONE +#define LAZY_DISABLE_TRAP_ENTRY +#define LAZY_DISABLE_TRAP_EXIT +#endif + /* * Interrupt entry/exit. * @@ -983,11 +1126,14 @@ END(interrupt) /* 0(%rsp): ~(interrupt number) */ .macro interrupt func + LAZY_DISABLED_CHECK \func /* reserve pt_regs for scratch regs and rbp */ subq $ORIG_RAX-RBP, %rsp CFI_ADJUST_CFA_OFFSET ORIG_RAX-RBP SAVE_ARGS_IRQ + LAZY_DISABLED_START call \func + LAZY_DISABLED_DONE .endm /* @@ -1124,6 +1270,12 @@ ENTRY(retint_kernel) jnc retint_restore_args bt $9,EFLAGS-ARGOFFSET(%rsp) /* interrupts off? */ jnc retint_restore_args +#ifdef CONFIG_LAZY_IRQ_DISABLE + /* Need to check our own preempt disabled variable */ + cmpl $0,PER_CPU_VAR(lazy_irq_on) + jnz retint_restore_args + orq $LAZY_IRQ_FL_TEMP_DISABLE, PER_CPU_VAR(lazy_irq_disabled_flags) +#endif call preempt_schedule_irq jmp exit_intr #endif @@ -1232,7 +1384,9 @@ ENTRY(\sym) DEFAULT_FRAME 0 movq %rsp,%rdi /* pt_regs pointer */ xorl %esi,%esi /* no error code */ + LAZY_DISABLE_TRAP_ENTRY call \do_sym + LAZY_DISABLE_TRAP_EXIT jmp error_exit /* %ebx: no swapgs flag */ CFI_ENDPROC END(\sym) @@ -1250,7 +1404,9 @@ ENTRY(\sym) TRACE_IRQS_OFF movq %rsp,%rdi /* pt_regs pointer */ xorl %esi,%esi /* no error code */ + LAZY_DISABLE_TRAP_ENTRY call \do_sym + LAZY_DISABLE_TRAP_EXIT jmp paranoid_exit /* %ebx: no swapgs flag */ CFI_ENDPROC END(\sym) @@ -1270,7 +1426,9 @@ ENTRY(\sym) movq %rsp,%rdi /* pt_regs pointer */ xorl %esi,%esi /* no error code */ subq $EXCEPTION_STKSZ, INIT_TSS_IST(\ist) + LAZY_DISABLE_TRAP_ENTRY call \do_sym + LAZY_DISABLE_TRAP_EXIT addq $EXCEPTION_STKSZ, INIT_TSS_IST(\ist) jmp paranoid_exit /* %ebx: no swapgs flag */ CFI_ENDPROC @@ -1289,7 +1447,9 @@ ENTRY(\sym) movq %rsp,%rdi /* pt_regs pointer */ movq ORIG_RAX(%rsp),%rsi /* get error code */ movq $-1,ORIG_RAX(%rsp) /* no syscall to restart */ + LAZY_DISABLE_TRAP_ENTRY call \do_sym + LAZY_DISABLE_TRAP_EXIT jmp error_exit /* %ebx: no swapgs flag */ CFI_ENDPROC END(\sym) @@ -1309,7 +1469,9 @@ ENTRY(\sym) movq %rsp,%rdi /* pt_regs pointer */ movq ORIG_RAX(%rsp),%rsi /* get error code */ movq $-1,ORIG_RAX(%rsp) /* no syscall to restart */ + LAZY_DISABLE_TRAP_ENTRY call \do_sym + LAZY_DISABLE_TRAP_EXIT jmp paranoid_exit /* %ebx: no swapgs flag */ CFI_ENDPROC END(\sym) @@ -1644,6 +1806,62 @@ ENTRY(error_exit) CFI_ENDPROC END(error_exit) +#ifdef CONFIG_LAZY_IRQ_DISABLE +/* + * Called with interrupts disabled. + * Returns enabling interrupts + * The calling function was kind enough to pass + * us the irq function and irq vector to use. + * + * This creates its own interrupt stack frame and + * then calls the interrupt as if the interrupt was + * called by hardware. It then returns via the normal + * interrupt return path which will enable interrupts + * with an iretq. + */ +ENTRY(native_simulate_irq) + CFI_STARTPROC + /* Save the current stack pointer */ + movq %rsp, %rcx + /* Save the stack frame as if we came from an interrupt */ + pushq_cfi $__KERNEL_DS + pushq_cfi %rcx + /* pop off the return addr for the return stack */ + subq $8, (%rsp) + pushfq_cfi + /* We want to return with interrupts enabled */ + addq $X86_EFLAGS_IF, (%rsp) + pushq_cfi $__KERNEL_CS + pushq_cfi (%rcx) + + ASM_CLAC + + /* Add the saved vector */ + pushq_cfi %rsi + + /* Function to call is in %rdi, but that will be clobbered */ + movq %rdi, %rcx + + /* Copied from interrupt macro */ + subq $ORIG_RAX-RBP, %rsp + CFI_ADJUST_CFA_OFFSET ORIG_RAX-RBP + SAVE_ARGS_IRQ + + addq $LAZY_IRQ_FL_REAL_DISABLE, PER_CPU_VAR(lazy_irq_disabled_flags) + + /* Call the triggered function */ + call *%rcx + + subq $LAZY_IRQ_FL_REAL_DISABLE, PER_CPU_VAR(lazy_irq_disabled_flags) + /* + * This will read our stack, and return + * enabling interrupts. + */ + jmp ret_from_intr + CFI_ENDPROC +END(native_simulate_irq) +#endif /* CONFIG_LAZY_IRQ_DISABLE */ + /* * Test if a given stack is an NMI stack or not. */ @@ -1874,7 +2092,9 @@ end_repeat_nmi: /* paranoidentry do_nmi, 0; without TRACE_IRQS_OFF */ movq %rsp,%rdi movq $-1,%rsi + LAZY_DISABLED_START call do_nmi + LAZY_DISABLED_DONE /* Did the NMI take a page fault? Restore cr2 if it did */ movq %cr2, %rcx diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 3a8185c..00ba667 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -363,3 +363,311 @@ void fixup_irqs(void) } } #endif + +#ifdef CONFIG_LAZY_IRQ_DISABLE +#include <linux/percpu.h> +#include <asm/local.h> + +/* Start out with real hard irqs disabled */ +DEFINE_PER_CPU(local_t, lazy_irq_disabled_flags) = LOCAL_INIT(LAZY_IRQ_FL_TEMP_DISABLE); +DEFINE_PER_CPU(void *, lazy_irq_func); +DEFINE_PER_CPU(unsigned long, lazy_irq_vector); +EXPORT_SYMBOL(lazy_irq_disabled_flags); +EXPORT_SYMBOL(lazy_irq_func); +EXPORT_SYMBOL(lazy_irq_vector); + +DEFINE_PER_CPU(unsigned long, lazy_irq_on); +EXPORT_SYMBOL(lazy_irq_on); + +#ifdef CONFIG_LAZY_IRQ_DEBUG + +static int update_data = 1; + +static DEFINE_PER_CPU(unsigned long, last_hard_enable); +static DEFINE_PER_CPU(unsigned long, last_soft_enable); +static DEFINE_PER_CPU(unsigned long, last_hard_disable); +static DEFINE_PER_CPU(unsigned long, last_soft_disable); +static DEFINE_PER_CPU(unsigned long, last_func); + +static DEFINE_PER_CPU(unsigned long, last_hard_enable_cnt); +static DEFINE_PER_CPU(unsigned long, last_soft_enable_cnt); +static DEFINE_PER_CPU(unsigned long, last_hard_disable_cnt); +static DEFINE_PER_CPU(unsigned long, last_soft_disable_cnt); +static DEFINE_PER_CPU(unsigned long, last_func_cnt); + +static DEFINE_PER_CPU(unsigned long, last_preempt_enable); +static DEFINE_PER_CPU(unsigned long, last_preempt_disable); +static DEFINE_PER_CPU(unsigned long, last_preempt_enable_cnt); +static DEFINE_PER_CPU(unsigned long, last_preempt_disable_cnt); + +atomic_t last_count = ATOMIC_INIT(0); + +#define UPDATE_LAST(type) \ + do { \ + if (update_data) { \ + this_cpu_write(last_##type, addr); \ + this_cpu_write(last_##type##_cnt, \ + atomic_inc_return(&last_count)); \ + } \ + } while (0) + +void notrace update_last_hard_enable(unsigned long addr) +{ + UPDATE_LAST(hard_enable); +} +EXPORT_SYMBOL(update_last_hard_enable); + +void notrace update_last_soft_enable(unsigned long addr) +{ + UPDATE_LAST(soft_enable); +} +EXPORT_SYMBOL(update_last_soft_enable); + +void notrace update_last_hard_disable(unsigned long addr) +{ + UPDATE_LAST(hard_disable); +} +EXPORT_SYMBOL(update_last_hard_disable); + +void notrace update_last_soft_disable(unsigned long addr) +{ + UPDATE_LAST(soft_disable); +} +EXPORT_SYMBOL(update_last_soft_disable); + + +void notrace update_last_preempt_disable(unsigned long addr) +{ + UPDATE_LAST(preempt_disable); +} +EXPORT_SYMBOL(update_last_preempt_disable); + +void notrace update_last_preempt_enable(unsigned long addr) +{ + UPDATE_LAST(preempt_enable); +} +EXPORT_SYMBOL(update_last_preempt_enable); + +void notrace update_last_func(unsigned long addr) +{ +// UPDATE_LAST(func); + if (update_data) { + this_cpu_write(last_func, addr); + this_cpu_write(last_func_cnt, atomic_read(&last_count)); +#if 0 + atomic_inc_return(&last_count)); +#endif + } +} + +void notrace print_lazy_debug(void) +{ + update_data = 0; + printk("Last hard enable: (%ld) %pS\n", + this_cpu_read(last_hard_enable_cnt), + (void *)this_cpu_read(last_hard_enable)); + printk("Last soft enable: (%ld) %pS\n", + this_cpu_read(last_soft_enable_cnt), + (void *)this_cpu_read(last_soft_enable)); + printk("Last hard disable: (%ld) %pS\n", + this_cpu_read(last_hard_disable_cnt), + (void *)this_cpu_read(last_hard_disable)); + printk("Last soft disable: (%ld) %pS\n", + this_cpu_read(last_soft_disable_cnt), + (void *)this_cpu_read(last_soft_disable)); + printk("Last func: (%ld) %pS\n", + this_cpu_read(last_func_cnt), + (void *)this_cpu_read(last_func)); + printk("Last preempt enable: (%ld) %pS\n", + this_cpu_read(last_preempt_enable_cnt), + (void *)this_cpu_read(last_preempt_enable)); + printk("Last preempt disable: (%ld) %pS\n", + this_cpu_read(last_preempt_disable_cnt), + (void *)this_cpu_read(last_preempt_disable)); + update_data = 1; +} + +void notrace print_lazy_irq(int line) +{ + update_data = 0; + printk("[%pS:%d] raw:%lx current:%lx flags:%lx\n", + __builtin_return_address(0), line, + raw_native_save_fl(), native_save_fl(), get_lazy_irq_flags()); + update_data = 1; +} + +asmlinkage void notrace lazy_irq_debug(long id, long err, void *func) +{ + update_data = 0; + printk("(%ld err=%lx f=%pS) flags=%lx vect=%lx func=%pS\n", id, ~err, func, + get_lazy_irq_flags(), + this_cpu_read(lazy_irq_vector), + this_cpu_read(lazy_irq_func)); + update_data = 1; +} + +void notrace +lazy_irq_bug(const char *file, int line, unsigned long flags, unsigned long raw) +{ + static int once; + + once = 1; + update_data = 0; + lazy_irq_add_temp(); + printk("FAILED HERE %s %d\n", file, line); + printk("flags=%lx init_raw=%lx\n", flags, raw); + printk("raw=%lx\n", raw_native_save_fl()); + print_lazy_debug(); + raw_native_irq_enable(); + update_data = 1; + BUG(); +} +EXPORT_SYMBOL(lazy_irq_bug); + +void notrace lazy_test_idle(void) +{ + unsigned long flags; + + flags = get_lazy_irq_flags(); + WARN_ON(!(flags & LAZY_IRQ_FL_IDLE)); + WARN_ON(flags & LAZY_IRQ_FL_DISABLED); +} + +#else +static inline void update_last_func(unsigned long addr) { } +#endif /* CONFIG_LAZY_IRQ_DEBUG */ + +#define BUG_ON_IRQS_ENABLED() \ + do { \ + BUG_ON(raw_native_save_fl() & X86_EFLAGS_IF); \ + } while (0) + +__init static int init_lazy_irqs(void) +{ + int cpu; + + return 0; + /* Only boot CPU needs irqs disabled */ + for_each_possible_cpu(cpu) { + if (cpu == smp_processor_id()) + continue; + local_set(&per_cpu(lazy_irq_disabled_flags, cpu), 0); + } + return 0; +} +early_initcall(init_lazy_irqs); + +unsigned long notrace lazy_irq_flags(void) +{ + return get_lazy_irq_flags(); +} + +/** + * native_simulate_irq - simulate an interrupt that triggered during lazy disable + * @func: The interrupt function to call. + * @orig_ax: The saved interrupt vector + * + * Defined in assembly, this function is used to simulate an interrupt + * that happened while the irq lazy disabling was in effect. + */ +extern void native_simulate_irq(void *func, unsigned long orig_ax); + +void notrace lazy_irq_simulate(void *func) +{ + this_cpu_write(lazy_irq_func, NULL); + + BUG_ON_IRQS_ENABLED(); + + update_last_func((unsigned long)func); + + native_simulate_irq(func, this_cpu_read(lazy_irq_vector)); +} +EXPORT_SYMBOL(lazy_irq_simulate); + +static inline void lazy_irq_sub_idle(void) +{ + lazy_irq_sub(LAZY_IRQ_FL_IDLE); +} + +static inline void lazy_irq_add_idle(void) +{ + lazy_irq_add(LAZY_IRQ_FL_IDLE); +} + +/** + * lazy_irq_idle_enter - handle lazy irq disabling entering idle + * + * When entering idle, we need to check if an interrupt came in, and + * if it did, then we should not go into the idle code. + * If no interrupt came in, we need to switch to a mode that + * we enable and disable interrupts for real, and turn off any + * lazy irq disable flags. The idle code is special as it can + * enter with interrupts disabled and leave with interrupts enabled + * via assembly. + */ +int notrace lazy_irq_idle_enter(void) +{ + unsigned long flags; + + flags = get_lazy_irq_flags(); + + /* + * If interrupts are hard coded off, then simply let the + * CPU do the work. + */ + if (flags >> LAZY_IRQ_REAL_DISABLE_BIT) { +#ifdef CONFIG_LAZY_IRQ_DEBUG + if (raw_native_save_fl() & X86_EFLAGS_IF) + lazy_irq_bug(__func__, __LINE__, flags, raw_native_save_fl()); + if (flags & LAZY_IRQ_FL_DISABLED) + lazy_irq_bug(__func__, __LINE__, flags, raw_native_save_fl()); +#endif + lazy_irq_add_idle(); + return 1; + } + + /* + * Note, if there's a pending interrupt, then on real hardware + * when the x86_idle() is called, it would trigger immediately. + * We need to imitate that. + * + * Disable interrupts for real, need this anyway, as interrupts + * would be enabled by the cpu idle. + */ + if (this_cpu_read(lazy_irq_func)) + BUG_ON_IRQS_ENABLED(); + + /* Disable for real to prevent any races */ + raw_native_irq_disable(); + + if (this_cpu_read(lazy_irq_func)) { + /* Process the interrupt and do not go idle */ + local_irq_enable(); + return 0; + } + + lazy_irq_add_idle(); + + flags = get_lazy_irq_flags(); + if (flags & LAZY_IRQ_FL_TEMP_DISABLE) + lazy_irq_sub_temp(); + /* Interrupts will be enabled exiting x86_idle() */ + BUG_ON(!(flags & LAZY_IRQ_FL_DISABLED)); + lazy_irq_sub_disable(); + return 1; +} + +void notrace lazy_irq_idle_exit(void) +{ + lazy_irq_sub_idle(); + BUG_ON(get_lazy_irq_flags() || get_lazy_irq_func()); +} + +#if 0 +EXPORT_SYMBOL(do_preempt_disable); +EXPORT_SYMBOL(do_preempt_enable); +EXPORT_SYMBOL(native_irq_disable); +EXPORT_SYMBOL(native_irq_enable); +#endif +#endif /* CONFIG_LAZY_IRQ_DISABLE */ + diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 83369e5..67c7ede 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -298,10 +298,13 @@ void arch_cpu_idle_dead(void) */ void arch_cpu_idle(void) { - if (cpuidle_idle_call()) - x86_idle(); - else - local_irq_enable(); + if (lazy_irq_idle_enter()) { + if (cpuidle_idle_call()) + x86_idle(); + else + local_irq_enable(); + lazy_irq_idle_exit(); + } } /* diff --git a/arch/x86/lib/thunk_64.S b/arch/x86/lib/thunk_64.S index a63efd6..7ab19e9 100644 --- a/arch/x86/lib/thunk_64.S +++ b/arch/x86/lib/thunk_64.S @@ -8,6 +8,20 @@ #include <linux/linkage.h> #include <asm/dwarf2.h> #include <asm/calling.h> +#include <asm/irqflags.h> + +#ifdef CONFIG_LAZY_IRQ_DISABLE + .macro LAZY_IRQ_ENTER + addq $LAZY_IRQ_FL_REAL_DISABLE, %gs:lazy_irq_disabled_flags + .endm + + .macro LAZY_IRQ_EXIT + subq $LAZY_IRQ_FL_REAL_DISABLE, %gs:lazy_irq_disabled_flags + .endm +#else +# define LAZY_IRQ_ENTER +# define LAZY_IRQ_EXIT +#endif /* rdi: arg1 ... normal C conventions. rax is saved/restored. */ .macro THUNK name, func, put_ret_addr_in_rdi=0 @@ -22,7 +36,9 @@ movq_cfi_restore 9*8, rdi .endif + LAZY_IRQ_ENTER call \func + LAZY_IRQ_EXIT jmp restore CFI_ENDPROC .endm diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c index fa6964d..da2f074 100644 --- a/drivers/idle/intel_idle.c +++ b/drivers/idle/intel_idle.c @@ -347,6 +347,7 @@ static int intel_idle(struct cpuidle_device *dev, unsigned int cstate; int cpu = smp_processor_id(); + lazy_test_idle(); cstate = (((eax) >> MWAIT_SUBSTATE_SIZE) & MWAIT_CSTATE_MASK) + 1; /* @@ -366,6 +367,7 @@ static int intel_idle(struct cpuidle_device *dev, if (!need_resched()) __mwait(eax, ecx); } + lazy_test_idle(); if (!(lapic_timer_reliable_states & (1 << (cstate)))) clockevents_notify(CLOCK_EVT_NOTIFY_BROADCAST_EXIT, &cpu); diff --git a/include/linux/debug_locks.h b/include/linux/debug_locks.h index 822c135..f883a74 100644 --- a/include/linux/debug_locks.h +++ b/include/linux/debug_locks.h @@ -26,8 +26,11 @@ extern int debug_locks_off(void); int __ret = 0; \ \ if (!oops_in_progress && unlikely(c)) { \ - if (debug_locks_off() && !debug_locks_silent) \ + if (debug_locks_off() && !debug_locks_silent) { \ + print_lazy_irq(__LINE__); \ + print_lazy_debug(); \ WARN(1, "DEBUG_LOCKS_WARN_ON(%s)", #c); \ + } \ __ret = 1; \ } \ __ret; \ diff --git a/kernel/watchdog.c b/kernel/watchdog.c index 1241d8c..712672a 100644 --- a/kernel/watchdog.c +++ b/kernel/watchdog.c @@ -301,6 +301,8 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) */ duration = is_softlockup(touch_ts); if (unlikely(duration)) { + static arch_spinlock_t lock = __ARCH_SPIN_LOCK_UNLOCKED; + /* * If a virtual machine is stopped by the host it can look to * the watchdog like a soft lockup, check to see if the host @@ -313,6 +315,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) if (__this_cpu_read(soft_watchdog_warn) == true) return HRTIMER_RESTART; + arch_spin_lock(&lock); printk(KERN_EMERG "BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n", smp_processor_id(), duration, current->comm, task_pid_nr(current)); @@ -322,6 +325,9 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) show_regs(regs); else dump_stack(); + arch_spin_unlock(&lock); + + trigger_all_cpu_backtrace(); if (softlockup_panic) panic("softlockup: hung tasks");
| |