[lkml]   [2001]   [Jul]   [13]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subject*very* strange heisenbug (kernel? libc?)
Hello all.

I have encountered a very, very strange bug, which has caused me a
good number of hours pulling my hair trying to figure out what was
going on. I don't even know if it's a kernel bug, a glibc bug, or a
misunderstanding on my part.

The basic program is simple enough - see <URL: >. All it
does is open the Linux TUN/TAP device (/dev/misc/net/tun), register a
tap device named "test0" (the actual name is unimportant), and then
create a thread which runs an idle loop before quitting immediately.
This seems sane enough. The code to open the TUN/TAP device (function
init_tap()) is practically copied from the sample code in the
documentation, and the rest is trivial. I can hardly imagine where it
might contain an error.

Yet I have gotten it to dump core. See <URL: > for the core,
which is produced by the binary at <URL: >. This is
all on i386 (detailed system characteristics follow below).

Now that's not all. There are several intriguing things about this

First of all, the place where the segfault happens: in
pthread_create(), of all places! To be precise, the function
__pthread_create_1_2() (defined in linuxthreads/pthread.c in the glibc
sources) calls thread_self() (defined in linuxthreads/internals.h),
which is inlined, and which in turn calls the macro THREAD_SELF,
defined in linuxthreads/sysdeps/i386/useldt.h (around line 57). It is
this macro which segfaults. In fact, the libc tries to determine the
thread's identity by looking up a place in memory accessed by %gs:0,
where %gs is supposed to be a selector for the LDT (normally, 0x7, in
fact); but here it happens that %gs was not initialized, and still
refers to the "flat" memory model (GDT selector 0x2b), so the program
tries to access memory page 0 and fails.

So this would look like a glibc bug (under some weird circumstances,
%gs is not properly initialized by INIT_THREAD_SELF before THREAD_SELF
is called). Except that I've heard rumors that core files were not
reliable for threaded programs. So maybe the segfault is not where I
think it is. (Except that in any case it seems that %gs is wrong at
the place where it is.)

The second funny thing is the circumstances under which I can
reproduce this bug. If I run the program directly, it doesn't
segfault. I have only been able to reproduce it by launching it with
the following ridiculous script: (<URL: >)

#! /bin/sh
if echo FOO | egrep -q FOO ; then
exec /tmp/weirdbug
exec /tmp/weirdbug

(wow!). Even then, it is unpredictable. It doesn't always work, and
I've only ever gotten anything on my dual-processor machine. Adding a
"sleep" here or there will get rid of the bug. Of course, trying to
do anything like strace or gdb will make the bug go away - therefore
it's a heisenbug.

Also, sometimes, instead of dumping core, it simply hangs and does

So what say the wise? Is this a glibc bug in the linuxthreads
implementation? A kernel bug in the TUN/TAP device with odd
consequences? A kernel bug in the scheduler leading to a very rare
race condition? A bug in my program? A karma problem?

Detailed system characteristics: RedHat-7.1 system: glibc-2.2.2 (RPM:
glibc-2.2.2-10). Kernel is stock 2.4.6 + international crypto patch
(, compiled for SMP, and with devfsd. Filesystem in /tmp is
ReiserFS. Hardware is dual Intel Pentium II (Deschutes) at 450MHz.
Compiler for kernel is stock egcs-1.1.2. Compiler for program is
RedHat's 2.96-81. Program is compiled against actual kernel headers.

Session log (I'm willing to bet nobody can reproduce it, anyway):

vega root /tmp # /sbin/modprobe tun
vega root /tmp # ls -la /dev/misc/net/tun
crw-r----- 1 root root 10, 200 Jan 1 1970 /dev/misc/net/tun
vega root /tmp # md5sum weirdbug.c
0cbeaa9de585868b3d5dfd55f6f50397 weirdbug.c
vega root /tmp # gcc -o weirdbug weirdbug.c -O6 -lpthread
vega root /tmp # gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs
gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-81)
vega root /tmp # md5sum weirdbug
7977421cf8b60a63659063d7c165c58b weirdbug
vega root /tmp # cat weirdbug-launcher
#! /bin/sh
if echo FOO | egrep -q FOO ; then
exec /tmp/weirdbug
exec /tmp/weirdbug
vega root /tmp # md5sum weirdbug-launcher
460666262466f12d89f33900e4dc74ec weirdbug-launcher
vega root /tmp # ./weirdbug
vega root /tmp # ./weirdbug-launcher
zsh: segmentation fault (core dumped) ./weirdbug-launcher

David A. Madore
(, )
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
More majordomo info at
Please read the FAQ at

 \ /
  Last update: 2005-03-22 12:57    [W:0.108 / U:0.080 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site