lkml.org 
[lkml]   [2012]   [Apr]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [RFC 1/2] kernel patch for dump user space stack tool
    On 04/19/2012 01:17 PM, Yanmin Zhang wrote:
    > On Thu, 2012-04-19 at 11:50 +0800, Cong Wang wrote:
    >> On 04/17/2012 10:37 PM, Tu, Xiaobing wrote:
    >>> Resend the patch because of the log is too long on a single line.
    >>>
    >>> From: xiaobing tu<xiaobing.tu@intel.com>
    >>>
    >>> Here is the kernel patch for this tool, The idea is to output user space stack call-chain from
    >>> /proc/xxx/stack, currently, /proc/xxx/stack only output kernel stack call chain. We extend
    >>> it to output user space call chain in hex format
    >>>
    >>
    >> Can you teach me why we still need this as we have pstack?
    > Cong,
    >
    > Sorry for replying so late. Xiaobing told me you sent him email and I
    > didn't receive the 1st one you sent out.


    Based on the length of your reply and the description of the patch, you
    hide lots of information in your patch description.

    >
    > I tried pstack and it does work. It means developers in the world wanted
    > the tool long long ago.
    >
    > Although not checking the source codes of pstack (sorry, I'm busy in debugging
    > many critical issues), I think pstack is based on ptrace interface, which means:
    > 1) It need traps into system for many times to collect call frames of one
    > task.
    > 2) It need send signal to the ptraced process to stop it. Such behavior
    > might have some impact if the ptraced process also processes many signals.
    > 3) The data parsing to get symbols might not be split from data collection.
    > I mean, it collects call frames of one process, then parses it; then collects the 2nd
    > task's. If there are many processes, it couldn't collect the data just at the monitor
    > time point.


    Yet another one who wants to "fix" ptrace. ;-)

    >
    > Why do we work out the tools? The original requirement is from real work.
    > We are enabling Android on Medfield. One typical error of Android is ANR.
    > When a process couldn't respond in 5 seconds, Android reports an ANR error,
    > and dumps JAVA call stack. However, it couldn't dump userspace lib (such like
    > bionic, written by C or C++). In addition, Android just dumps the stack of
    > the non-responding process. It doesn't dump stack of others. As binder is basic
    > framework in Android, processes communicate by binder in the model of client/server.
    > When one process is not responding quickly, maybe another process blocks it. We
    > need dump that process status.
    >
    > Many teams complained it's hard to debug such ANR issues, especially the ones which
    > are triggered at MTBF testing. Sometimes, an ANR happens after MTBF testing runs
    > for one week. Developers ask us to implement such tool over and over again.
    >
    > Besides ANR, sometimes, system might not respond to any user operation. Usually,
    > kernel or firmware would reset system. At that time, we also need get the call
    > chains of all the user space processes before system is reset.


    I am not familiar with Andriod at all, so a quick question is if this is
    only for Andriod, why you introduce this for all? IOW, why not provide a
    Kconfig?

    BTW, I am sure you need to put the above paragraphs into your patch
    description, to make it clear why the patch is needed.

    >
    > With our tool,
    > 1) We could collect the HEX-format call chain data and /proc/XXX/maps
    > of all the processes quickly, then parse them either after rebooting, or
    > after the issue is reported. It could catch the scene just at the time point
    > when the error happens. Our experiments shows the tool could collect the data
    > of all processes within 200ms.
    > 2) The new tool won't stop the processes and have less impact on them.
    > Considering a scenario of performance bottleneck investigation, statistics collection
    > shouldn't have big impact on running processes.
    > 3) It could support both i386 and x86-64. I tried pstack and it doesn't work
    > with x86-64.
    > 4) It follows /proc/XXX/stack interface and it's easy to use it.
    >
    > Besides this tool, we are considering to extend it to collect user space
    > call chain of current process from kernel when kernel detects some other
    > abnormal behavior.
    >

    In my previous reply, I ran 'pstrack' on my x86-64 machine, don't
    understand why you said it doesn't work with x86-64? I guess pstack
    supports more than just x86, as ptrace is available in other arch's too.

    Thanks.


    \
     
     \ /
      Last update: 2012-04-19 08:15    [W:2.568 / U:0.204 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site