Messages in this thread | | | Date | Tue, 27 Jan 2004 08:44:00 +0100 | From | Goran Koruga <> | Subject | semop() issues in 2.4 kernel |
| |
Hello all.
We have recently stumbled across an interesting problem with 2.4 kernel. It seems to occur with SMP kernel and machines only - we haven't seen it occur in non-SMP environments. The problem was first observed with 2.4.7-10smp RedHat 7.2's kernel, but I have lately been able to reproduce it with 2.4.23 as well. I haven't tested 2.4.24 because there are no changes in this area, but this can easily be arranged if needed. All data reported here is from the same machine running RedHat 7.2 (but the choice of distribution and its version should have no effect here in my opinion - we also know it occurs on RedHat 8).
The problem is related to semop() system call which seems to "hang" the application in certain situations. I have first looked at what glibc does with it, but it merely issues a system call, so the problem is not there. I will describe how our application works. The application is multi-threaded, and most important are threads 1 and 3. Thread 1 (I will refer to it as main thread from now on) is also a signal handler, while all other threads block signals. Thread 3 runs in a loop, waiting for semaphore to be released to "process" the signal. After it is notified, it calls a function which takes care of needed things (think of this function as a signal handler which is called indirectly). When a signal is caught, generic signal handler in main thread updates some internal data and uses semop() to signal it to thread 3. So main thread calls semop() with sem_op set to 1, while thread 3 calls semop() with sem_op set to -1 (and consequently waits until semaphore is released).
This scenario leads to hanging main thread in certain cases, which should not happen - even the man page for semop(2) says :
---------------------------------------------------------------------- If sem_op is a positive integer, the operation adds this value to semval. Furthermore, if SEM_UNDO is asserted for this operation, the system updates the process undo count for this semaphore. The operation always goes through, so no process sleeping can happen. The calling process must have alter permissions on the semaphore set. ----------------------------------------------------------------------
I have added some debugging to the kernel functions responsible for semop() operations, namely the following :
sys_semtimedop() try_atomic_semop()
sys_semtimedop() was introduced somewhat recently, it was all happening in sys_semop() previously (which is now just a wrapper for sys_semtimedop()). I have used the most naive approach to track the problem down by simply adding prinkt()-s to the code to find out where it happens and possibly why. Note that I have no experience in kernel development, so this is as far as I could go. The only thing that we have concluded from looking at this data is that the hang always occurs if semop() issued in main thread starts executing and is then interrupted by semop() issued in thread 3. If this occurs, main thread never wakes up after schedule() is called. I don't know if this conclusion makes sense, but looking at many kernel logfiles we have always noticed this pattern before the hang. There's one very interesting thing related to this - if one uses strace on main thread, it wakes up and continues its work normally. The same is true if gdb is used - process wakes up and continues to work fine when gdb detaches from it. If someone has any ideas, suggestions etc., please let me know.
I have attached a set of files to this mail, containing the following (sorry for a rather large post because of this) :
1. stacktrace of the hanging process 2. kernel log with my extra printk()-s 3. kernel config 4. modified sem.c file with extra printk()-s, based on 2.4.23 kernel 5. /proc/cpuinfo
Some more things for easier understanding of attached files :
messages3: main thread is 367, thread 3 is 375
There are also logs of semop() operations in many other processes - these are from the same application since semaphores are also used for parent/child synchronisation when forking (using a totally unrelated semaphore obviously - which is always retrieved using semget() before the fork and then removed). I also have a few more kernel logfiles and process' stacktraces, but they are not attached due to size constraints when posting to the list.
Regards, Goran
P.S. I am not subscribed to the list, please Cc: me on any communication. I will also try to check the replies via gmane.
[unhandled content-type:application/octet-stream] | |