lkml.org 
[lkml]   [2004]   [Jan]   [27]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    Subjectsemop() issues in 2.4 kernel
    Hello all.

    We have recently stumbled across an interesting problem with 2.4 kernel.
    It seems to occur with SMP kernel and machines only - we haven't seen it
    occur in non-SMP environments. The problem was first observed with
    2.4.7-10smp RedHat 7.2's kernel, but I have lately been able to reproduce
    it with 2.4.23 as well. I haven't tested 2.4.24 because there are no
    changes in this area, but this can easily be arranged if needed. All data
    reported here is from the same machine running RedHat 7.2 (but the choice
    of distribution and its version should have no effect here in my
    opinion - we also know it occurs on RedHat 8).

    The problem is related to semop() system call which seems to "hang" the
    application in certain situations. I have first looked at what glibc
    does with it, but it merely issues a system call, so the problem is not
    there. I will describe how our application works. The application is
    multi-threaded, and most important are threads 1 and 3. Thread 1 (I will
    refer to it as main thread from now on) is also a signal handler, while
    all other threads block signals. Thread 3 runs in a loop, waiting for
    semaphore to be released to "process" the signal. After it is notified,
    it calls a function which takes care of needed things (think of this
    function as a signal handler which is called indirectly). When a signal
    is caught, generic signal handler in main thread updates some internal
    data and uses semop() to signal it to thread 3. So main thread calls
    semop() with sem_op set to 1, while thread 3 calls semop() with sem_op
    set to -1 (and consequently waits until semaphore is released).

    This scenario leads to hanging main thread in certain cases, which
    should not happen - even the man page for semop(2) says :

    ----------------------------------------------------------------------
    If sem_op is a positive integer, the operation adds this
    value to semval. Furthermore, if SEM_UNDO is asserted for
    this operation, the system updates the process undo count
    for this semaphore. The operation always goes through, so
    no process sleeping can happen. The calling process must
    have alter permissions on the semaphore set.
    ----------------------------------------------------------------------

    I have added some debugging to the kernel functions responsible for
    semop() operations, namely the following :

    sys_semtimedop()
    try_atomic_semop()

    sys_semtimedop() was introduced somewhat recently, it was all happening
    in sys_semop() previously (which is now just a wrapper for
    sys_semtimedop()). I have used the most naive approach to track the
    problem down by simply adding prinkt()-s to the code to find out where it
    happens and possibly why. Note that I have no experience in kernel
    development, so this is as far as I could go. The only thing that we have
    concluded from looking at this data is that the hang always occurs if
    semop() issued in main thread starts executing and is then interrupted
    by semop() issued in thread 3. If this occurs, main thread never wakes up
    after schedule() is called. I don't know if this conclusion makes sense,
    but looking at many kernel logfiles we have always noticed this pattern
    before the hang. There's one very interesting thing related to this - if
    one uses strace on main thread, it wakes up and continues its work
    normally. The same is true if gdb is used - process wakes up and
    continues to work fine when gdb detaches from it. If someone has any
    ideas, suggestions etc., please let me know.

    I have attached a set of files to this mail, containing the following
    (sorry for a rather large post because of this) :

    1. stacktrace of the hanging process
    2. kernel log with my extra printk()-s
    3. kernel config
    4. modified sem.c file with extra printk()-s, based on 2.4.23 kernel
    5. /proc/cpuinfo

    Some more things for easier understanding of attached files :

    messages3: main thread is 367, thread 3 is 375

    There are also logs of semop() operations in many other processes -
    these are from the same application since semaphores are also used for
    parent/child synchronisation when forking (using a totally unrelated
    semaphore obviously - which is always retrieved using semget() before
    the fork and then removed). I also have a few more kernel logfiles and
    process' stacktraces, but they are not attached due to size constraints
    when posting to the list.

    Regards,
    Goran

    P.S. I am not subscribed to the list, please Cc: me on any
    communication. I will also try to check the replies via gmane.

    [unhandled content-type:application/octet-stream]
    \
     
     \ /
      Last update: 2005-03-22 14:00    [W:0.028 / U:119.988 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site