[lkml]   [2004]   [Jan]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subjectsemop() issues in 2.4 kernel
Hello all.

We have recently stumbled across an interesting problem with 2.4 kernel.
It seems to occur with SMP kernel and machines only - we haven't seen it
occur in non-SMP environments. The problem was first observed with
2.4.7-10smp RedHat 7.2's kernel, but I have lately been able to reproduce
it with 2.4.23 as well. I haven't tested 2.4.24 because there are no
changes in this area, but this can easily be arranged if needed. All data
reported here is from the same machine running RedHat 7.2 (but the choice
of distribution and its version should have no effect here in my
opinion - we also know it occurs on RedHat 8).

The problem is related to semop() system call which seems to "hang" the
application in certain situations. I have first looked at what glibc
does with it, but it merely issues a system call, so the problem is not
there. I will describe how our application works. The application is
multi-threaded, and most important are threads 1 and 3. Thread 1 (I will
refer to it as main thread from now on) is also a signal handler, while
all other threads block signals. Thread 3 runs in a loop, waiting for
semaphore to be released to "process" the signal. After it is notified,
it calls a function which takes care of needed things (think of this
function as a signal handler which is called indirectly). When a signal
is caught, generic signal handler in main thread updates some internal
data and uses semop() to signal it to thread 3. So main thread calls
semop() with sem_op set to 1, while thread 3 calls semop() with sem_op
set to -1 (and consequently waits until semaphore is released).

This scenario leads to hanging main thread in certain cases, which
should not happen - even the man page for semop(2) says :

If sem_op is a positive integer, the operation adds this
value to semval. Furthermore, if SEM_UNDO is asserted for
this operation, the system updates the process undo count
for this semaphore. The operation always goes through, so
no process sleeping can happen. The calling process must
have alter permissions on the semaphore set.
I have added some debugging to the kernel functions responsible for
semop() operations, namely the following :


sys_semtimedop() was introduced somewhat recently, it was all happening
in sys_semop() previously (which is now just a wrapper for
sys_semtimedop()). I have used the most naive approach to track the
problem down by simply adding prinkt()-s to the code to find out where it
happens and possibly why. Note that I have no experience in kernel
development, so this is as far as I could go. The only thing that we have
concluded from looking at this data is that the hang always occurs if
semop() issued in main thread starts executing and is then interrupted
by semop() issued in thread 3. If this occurs, main thread never wakes up
after schedule() is called. I don't know if this conclusion makes sense,
but looking at many kernel logfiles we have always noticed this pattern
before the hang. There's one very interesting thing related to this - if
one uses strace on main thread, it wakes up and continues its work
normally. The same is true if gdb is used - process wakes up and
continues to work fine when gdb detaches from it. If someone has any
ideas, suggestions etc., please let me know.

I have attached a set of files to this mail, containing the following
(sorry for a rather large post because of this) :

1. stacktrace of the hanging process
2. kernel log with my extra printk()-s
3. kernel config
4. modified sem.c file with extra printk()-s, based on 2.4.23 kernel
5. /proc/cpuinfo

Some more things for easier understanding of attached files :

messages3: main thread is 367, thread 3 is 375

There are also logs of semop() operations in many other processes -
these are from the same application since semaphores are also used for
parent/child synchronisation when forking (using a totally unrelated
semaphore obviously - which is always retrieved using semget() before
the fork and then removed). I also have a few more kernel logfiles and
process' stacktraces, but they are not attached due to size constraints
when posting to the list.


P.S. I am not subscribed to the list, please Cc: me on any
communication. I will also try to check the replies via gmane.

[unhandled content-type:application/octet-stream]
 \ /
  Last update: 2005-03-22 14:00    [from the cache]
©2003-2011 Jasper Spaans