[lkml]   [2001]   [May]   [26]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Patch in this message
Subject[patch] severe softirq handling performance bug, fix, 2.4.5

i've been seeing really bad average TCP latencies on certain gigabit cards
(~300-400 microseconds instead of the expected 100-200 microseconds), ever
since softnet went into the main kernel, and never found a real
explanation for it, until today.

the problem always went away when i tried to use tcpdump or strace, so the
bug remained hidden and was hard to prove that it actually existed. (apart
from the bad lat_tcp numbers.) We found many related bugs, but this
problem remained. tcpdumps done on the network did not show any fault of
the TCP stack. The lat_tcp latencies fluctuated alot, but for certain
cards the latencies were stable, so i suspected some sort of hw problem.
The loopback networking device never showed these problems, which added to
the mystery.

the problem turned out to be a severe softirq handling bug in the x86

background: soft interrupts were introduced as a generic kernel framework
around January 2000, as part of the softnet networking-rewrite, that
predated the final scalability rewrite of the Linux TCP/IP networking
code. Soft interrupts have unique semantics, they can be best described as
'IRQ-triggered atomic system calls'. (unlike bottom halves, soft-IRQs do
not preempt kernel code.)

soft-IRQs, like their name suggest, are used from device interrupts ('hard
interrupts') to trigger 'background' work related to interrupts. Soft-IRQs
are triggered per-CPU, and they are supposed to execute whenever nothing
else is done by the kernel on that particular CPU. Softirqs are executed
with interrupts enabled, so hard interrupts can re-enable them while they
are executing. do_softirq() is a kernel function that returns with IRQs
disabled and at this point it's guaranteed that there are no more pending
softirqs for this CPU.

this mechanizm was the intention, but not the reality. In two important
and frequently used code paths it was possible for an active soft-IRQ to
"go unnoticed": i measured as long as 140 milliseconds (!!!) latency
between softirq activation and softirq execution in certain cases. This is
obviously bad behavior.

the two error cases are:

#1 hard-IRQ interrupts user-space code, activates softirq, and returns to
user-space code

#2 hard-IRQ interrupts the idle task, activates softirq and returns to
the idle task.

category #1 is easy to fix, in entry.S we have to check active softirqs
not only the exception and ret-from-syscall cases, but also in the
IRQ-ret-to-userspace case.

category #2 is subtle, because the idle process is kernel code, so
returning to it we do not execute active softirqs. The main two types of
idle handlers both had a window do 'miss' softirq execution:

- the HLT-based default handler could be called after schedule()'s check
for softirqs, but after enabling IRQs. In this case an interrupt handler
has a window to activate a softirq and neither the IRQ return code, nor
the idle loop would execute it immediately. The fix is to do a softirq
check right before the safe_halt call.

- the idle-poll handler does not check for softirqs either, it now does
this in every iteration.

with the attached softirq-2.4.5-A0 patch applied to vanilla 2.4.5, i see
picture-perfect lat_tcp latencies of 109 microseconds over real gigabit
network. I see very stable (and very good) TUX latencies as well. TCP
bandwidth got better as well, probably due to the caching-locality bonus
when executing softirqs right after hardirqs.

[I'd like to ask everyone who had TCP latency problems (or other
networking performance problems) to test 2.4.5 with this patch applied -

impact of the bug: all softirq-using code is affected, mostly networking.
The loopback net driver was not affected because it's not interrupt-based.
The bug went away due to strace or tcpdump because those two utilities
pumped system-calls into the system which 'fixed' the softirq handling

(other softirq-based code is the tasklet code, and the keyboard code is
using tasklets, so the keyboard code might be affected as well.)

--- linux/arch/i386/kernel/entry.S.orig Sat May 26 19:20:48 2001
+++ linux/arch/i386/kernel/entry.S Sat May 26 19:21:52 2001
@@ -214,7 +214,6 @@
jne handle_softirq

cmpl $0,need_resched(%ebx)
jne reschedule
cmpl $0,sigpending(%ebx)
@@ -275,7 +274,7 @@
movl EFLAGS(%esp),%eax # mix EFLAGS and CS
movb CS(%esp),%al
testl $(VM_MASK | 3),%eax # return to VM86 mode or non-supervisor?
- jne ret_with_reschedule
+ jne ret_from_sys_call
jmp restore_all

--- linux/arch/i386/kernel/process.c.orig Sat May 26 19:21:56 2001
+++ linux/arch/i386/kernel/process.c Sat May 26 19:28:06 2001
@@ -79,8 +79,12 @@
static void default_idle(void)
+ int this_cpu = smp_processor_id();
if (current_cpu_data.hlt_works_ok && !hlt_counter) {
+ if (softirq_active(this_cpu) & softirq_mask(this_cpu))
+ do_softirq();
if (!current->need_resched)
@@ -95,6 +99,7 @@
static void poll_idle (void)
+ int this_cpu = smp_processor_id();
int oldval;

@@ -104,14 +109,17 @@
* run here:
oldval = xchg(&current->need_resched, -1);
+ if (oldval)
+ return;

- if (!oldval)
- asm volatile(
- "2:"
- "cmpl $-1, %0;"
- "rep; nop;"
- "je 2b;"
- : :"m" (current->need_resched));
+ while (current->need_resched == -1) {
+ if (softirq_active(this_cpu) & softirq_mask(this_cpu)) {
+ __cli();
+ do_softirq();
+ __sti();
+ }
+ asm volatile( "rep; nop;" );
+ }

 \ /
  Last update: 2005-03-22 13:17    [W:0.115 / U:11.424 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site