Messages in this thread | | | Subject | IPI deadlock bug | Date | Tue, 8 Dec 1998 06:33:09 +0200 (IST) | From | Amnon Shiloh <> |
| |
Hello,
Here is a serious bug that can cause deadlocks on an SMP in the latest 2.1.1xx kernels:
The following conditions need to happen for the deadlock to appear: 1. main memory is short and in demand, so "kswapd" is very busy finding and swapping out pages. 2. Meanwhile, an interrupt is caught by another CPU, while running a process in user-mode, after which that process has a signal to process. (for example, a keyboard-interrupt that causes a SIGQUIT) 3. The signal-handling requires a non-trivial action, most commonly causing the process to die (call "do_exit"), or any other action that requires the kernel-lock ("lock_kernel()").
Because "do_signal" is called with interrupts disabled, the CPU does not respond to the multitude of invalidate requests generated for each page stolen by "kswapd", so "kswapd" waits for about 2 seconds per-page until giving up and printk'ing: "stuck on smp_invalidate_needed IPI wait (CPU#n)", and with heavy swapping and large memory, the number of pages can reach thousands, so it can be up to an hour during which the computer does not respond even to interrupts (it will emerge eventually if the operator was very patient and did not press RESET meanwhile, but with a wrong clock and lost network connections). During this time, the other CPU - the one that received the interrupt, is stuck on "lock_kernel", because "kswapd" (or any other process running "try_to_free_pages()") runs with a locked kernel.
The fix is VERY simple - in "arch/i386/kernel/entry.S":
ret_from_intr: GET_CURRENT(%ebx) movl EFLAGS(%esp),%eax # mix EFLAGS and CS movb CS(%esp),%al testl $(VM_MASK | 3),%eax # return to VM86 mode or non-supervisor? - jne ret_with_reschedule - jmp restore_all + je restore_all + sti + jmp ret_with_reschedule
ALIGN
Hope this is included in the forthcoming Linux kernels.
Amnon Shiloh - The MOSIX group, Hebrew University of Jerusalem.
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |