Messages in this thread | | | From | Jason Schoonover <> | Subject | Re: High load average on disk I/O on 2.6.17-rc3 | Date | Sat, 6 May 2006 11:23:01 -0700 |
| |
Hi Robert,
I did started a ncftpget and managed to get 6 pdflush processes running in state D, hopefully this will give us a chance to debug it.
I've attached the entire Alt+SysReq+T output here because I have no idea how to read it.
Thanks, Jason
-------Original Message----- From: Robert Hancock Sent: Saturday 06 May 2006 10:20 To: Jason Schoonover Subject: Re: High load average on disk I/O on 2.6.17-rc3
Jason Schoonover wrote: > Hi Robert, > > There are, this is the relevant output of the process list: > > ... > 4659 pts/6 Ss 0:00 -bash > 4671 pts/5 R+ 0:12 cp -a test-dir/ new-test > 4676 ? D 0:00 [pdflush] > 4679 ? D 0:00 [pdflush] > 4687 pts/4 D+ 0:01 hdparm -t /dev/sda > 4688 ? D 0:00 [pdflush] > 4690 ? D 0:00 [pdflush] > 4692 ? D 0:00 [pdflush] > ... > > This was when I was copying a directory and then doing a performance test > with hdparm in a separate shell. The hdparm process was in [D+] state and > basically waited until the cp was finished. During the whole thing there > were up to 5 pdflush processes in [D] state. > > The 5 minute load average hit 8.90 during this test. > > Does that help?
Well, it obviously explains why the load average is high, those D state processes all count in the load average. It may be sort of a cosmetic issue, since they're not actually using any CPU, but it's still a bit unusual. For one thing, not sure why there are that many of them?
You could try enabling the SysRq triggers (if they're not already in your kernel/distro) and doing Alt-SysRq-T which will dump the kernel stack of all processes, that should show where exactly in the kernel those pdflush processes are blocked.. 015c0ca> sys_select+0x9a/0x166 <b0104a5b> do_IRQ+0x22/0x2b <b010267b> syscall_call+0x7/0xb snmpd S 3C14DC00 0 3771 1 3778 3764 (NOTLB) f7975b58 f78d8300 0000000f 3c14dc00 00000000 000186a0 d5831281 000f68dc dfbc4688 dfbc4580 b21a0540 b2014340 7b773b00 000f9663 00000001 b01273fc b21f0500 f78aef00 00200286 b011e999 f7975b68 f7975b68 00200286 00000000 Call Trace: <b01273fc> add_wait_queue+0x12/0x30 <b011e999> lock_timer_base+0x15/0x2f <b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9 <b015baca> do_select+0x3b6/0x41e <f88df929> ext3_mark_iloc_dirty+0x2a7/0x345 [ext3] <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b0113571> try_to_wake_up+0x327/0x331 <b01609e5> inode_init_once+0x4c/0x14c <b0149394> cache_alloc_refill+0x39e/0x45d <b0149062> cache_alloc_refill+0x6c/0x45d <b01117b5> __wake_up_common+0x2a/0x4f <b017605f> proc_alloc_inode+0x41/0x66 <b0160911> alloc_inode+0xf5/0x17d <b016e6c0> inotify_d_instantiate+0x3b/0x5e <b015fd63> d_instantiate+0x45/0x78 <b0160557> d_rehash+0x4f/0x5f <b0178a39> proc_lookup+0x77/0xaf <b0137669> get_page_from_freelist+0x7e/0x31a <b0137895> get_page_from_freelist+0x2aa/0x31a <b015bd1a> core_sys_select+0x1e8/0x2a2 <b01ad8db> vsnprintf+0x423/0x461 <b0137c3a> free_hot_cold_page+0x91/0xcd <b0137c90> __pagevec_free+0x1a/0x24 <b01393de> release_pages+0x147/0x150 <b0142040> page_remove_rmap+0x27/0x2a <b021303e> lock_sock+0x85/0x8d <b0213d18> sk_free+0xa8/0xe2 <b0210e8e> sock_destroy_inode+0x13/0x16 <b015f456> dput+0x1b/0x11e <b015c0ca> sys_select+0x9a/0x166 <b011b272> sys_gettimeofday+0x22/0x51 <b010267b> syscall_call+0x7/0xb smbd S 000F4240 0 3776 3764 (NOTLB) df977fb0 00000000 00000000 000f4240 00000000 00028b0a 00000012 00000002 dff18178 dff18070 dfb78580 b200c340 1685e740 000f41ff 00000000 00000000 00000001 00000007 df977fbc dff18070 00000001 df977fbc b0282b65 00000007 Call Trace: <b011fa32> sys_pause+0x14/0x1a <b010267b> syscall_call+0x7/0xb sshd S 004C4B40 0 3778 1 3875 3820 3771 (NOTLB) f7f77b60 b23c4374 00000000 004c4b40 00000000 00000000 f8545f80 000f68dd dfb81138 dfb81030 b02b0480 b200c340 ca71c780 000f95ea 00000000 00000000 dfa76ac8 00200246 b01273fc fffffffc df989a80 00200246 b023aee4 00000000 Call Trace: <b01273fc> add_wait_queue+0x12/0x30 <b023aee4> tcp_poll+0x24/0x134 <b026fd30> schedule_timeout+0x13/0x8b <b02112c0> sock_poll+0x13/0x17 <b015baca> do_select+0x3b6/0x41e <f88df929> ext3_mark_iloc_dirty+0x2a7/0x345 [ext3] <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b014f430> __getblk+0x1d/0x211 <f8874995> do_get_write_access+0x44a/0x45c [jbd] <b01114b5> __activate_task+0x47/0x56 <b0113571> try_to_wake_up+0x327/0x331 <f88e698a> __ext3_journal_stop+0x19/0x37 [ext3] <b01271f4> autoremove_wake_function+0x18/0x3a <b01117b5> __wake_up_common+0x2a/0x4f <b0149062> cache_alloc_refill+0x6c/0x45d <b01ab175> kobject_put+0x16/0x19 <b01ab69b> kobject_release+0x0/0xa <b0215457> __alloc_skb+0x4a/0xee <b0213988> sock_alloc_send_skb+0x5a/0x188 <b0217138> memcpy_fromiovec+0x29/0x4e <b0214593> sock_def_readable+0x11/0x5e <b02659e1> unix_stream_sendmsg+0x21b/0x2e6 <b015bd1a> core_sys_select+0x1e8/0x2a2 <b0211077> do_sock_write+0xa4/0xad <b0137669> get_page_from_freelist+0x7e/0x31a <b0137953> __alloc_pages+0x4e/0x261 <b0142611> __page_set_anon_rmap+0x2b/0x2f <b013d845> do_wp_page+0x244/0x286 <b013d869> do_wp_page+0x268/0x286 <b0215117> skb_dequeue+0x3b/0x41 <b0160690> destroy_inode+0x48/0x4c <b015f456> dput+0x1b/0x11e <b015c0ca> sys_select+0x9a/0x166 <b014a58a> filp_close+0x4e/0x57 <b010267b> syscall_call+0x7/0xb rpc.statd S 006ACFC0 0 3820 1 3828 3778 (NOTLB) df9a9b60 b01114b5 b0328460 006acfc0 00000000 00000000 00000100 00000000 dfb7a138 dfb7a030 b21a0540 b2014340 3b0f8a80 000f41ff 00000001 f78a4c80 00000246 00000246 b01273fc 00000304 df917b80 00000246 b023aee4 00000000 Call Trace: <b01114b5> __activate_task+0x47/0x56 <b01273fc> add_wait_queue+0x12/0x30 <b023aee4> tcp_poll+0x24/0x134 <b026fd30> schedule_timeout+0x13/0x8b <b02112c0> sock_poll+0x13/0x17 <b015baca> do_select+0x3b6/0x41e <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b021a02c> net_rx_action+0x98/0x146 <b011bdf9> __do_softirq+0x57/0xc0 <b011bfc0> local_bh_enable+0x5b/0x66 <b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6 <b02369b7> ip_push_pending_frames+0x351/0x3dd <b02170ea> memcpy_toiovec+0x29/0x4e <b0217824> skb_copy_datagram_iovec+0x49/0x1a6 <b0215932> kfree_skbmem+0x65/0x69 <b024e64d> udp_recvmsg+0x18a/0x1d3 <b021318b> sock_common_recvmsg+0x36/0x4b <b0212a1e> sock_recvmsg+0xf1/0x10d <b01271dc> autoremove_wake_function+0x0/0x3a <b015bd1a> core_sys_select+0x1e8/0x2a2 <b0210e04> move_addr_to_user+0x3a/0x52 <b0212d12> sys_recvfrom+0xfc/0x134 <b021303e> lock_sock+0x85/0x8d <b0211aa5> sys_bind+0x73/0xa0 <b0213d18> sk_free+0xa8/0xe2 <b0210e8e> sock_destroy_inode+0x13/0x16 <b015f456> dput+0x1b/0x11e <b015c0ca> sys_select+0x9a/0x166 <b014a58a> filp_close+0x4e/0x57 <b010267b> syscall_call+0x7/0xb ntpd S 39E048C0 0 3828 1 3836 3820 (NOTLB) df8d7b60 0583c2ce 00000127 39e048c0 00000000 00000000 00000000 b011bdf9 b21ddb98 b21dda90 b02b0480 b200c340 48b33bc0 000f966c 00000000 dff27540 00000246 b021773c dff27540 df98a118 df8d7bd4 df98a100 dff27540 00000000 Call Trace: <b011bdf9> __do_softirq+0x57/0xc0 <b021773c> datagram_poll+0x21/0xc0 <b026fd30> schedule_timeout+0x13/0x8b <b02112c0> sock_poll+0x13/0x17 <b015baca> do_select+0x3b6/0x41e <f88df929> ext3_mark_iloc_dirty+0x2a7/0x345 [ext3] <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6 <b0113571> try_to_wake_up+0x327/0x331 <b0214593> sock_def_readable+0x11/0x5e <b01117b5> __wake_up_common+0x2a/0x4f <b0111803> __wake_up+0x29/0x3c <b02145b5> sock_def_readable+0x33/0x5e <b0243570> tcp_rcv_established+0x40e/0x69a <f886501c> ipt_hook+0x1c/0x20 [iptable_filter] <b024963e> tcp_v4_do_rcv+0x23/0x2ba <b0232a0a> ip_local_deliver_finish+0x0/0x193 <b0232a0a> ip_local_deliver_finish+0x0/0x193 <b024a72a> tcp_v4_rcv+0x889/0x8dd <b0232bf8> ip_local_deliver+0x5b/0x1fd <b0232cf6> ip_local_deliver+0x159/0x1fd <b0213117> sk_reset_timer+0x12/0x1e <b02447b1> tcp_send_delayed_ack+0xb7/0xbd <b015bd1a> core_sys_select+0x1e8/0x2a2 <f886501c> ipt_hook+0x1c/0x20 [iptable_filter] <b011f676> __sigqueue_free+0x2a/0x2f <b010710a> convert_fxsr_to_user+0xdf/0x12e <b0107274> save_i387+0xf0/0x105 <b0101e68> setup_sigcontext+0xde/0x11c <b0102490> do_notify_resume+0x4ab/0x58b <b010729f> convert_fxsr_from_user+0x16/0xd6 <b015c0ca> sys_select+0x9a/0x166 <b0101c66> restore_sigcontext+0x102/0x14c <b0101fc0> sys_sigreturn+0x98/0xbd <b010267b> syscall_call+0x7/0xb atd S 000F4240 0 3836 1 3843 3828 (NOTLB) df8b5f1c df8b5ee8 30b8a000 000f4240 00000000 00000000 445cddcf f7e23f5c b21c8178 b21c8070 b21a0540 b2014340 a2ec1c80 000f93e0 00000001 b2014d94 b2014d8c b2014d8c 1268d3c8 00000282 b012996a 00000000 e1b033c8 00000000 Call Trace: <b012996a> hrtimer_start+0xc5/0xd0 <b0270109> do_nanosleep+0x3b/0x63 <b01299b8> hrtimer_nanosleep+0x43/0xfe <b0120244> do_sigaction+0x92/0x149 <b0129596> hrtimer_wakeup+0x0/0x1c <b0129ab0> sys_nanosleep+0x3d/0x51 <b010267b> syscall_call+0x7/0xb cron S 03A2C940 0 3843 1 3859 3836 (NOTLB) f7e23f1c f7e23ee8 f8475800 03a2c940 00000000 00000000 445ce895 f7fb5cc4 dff6e648 dff6e540 b02b0480 b200c340 23033000 000f9663 00000000 b200cd94 b200cd8c b200cd8c 264ea2e8 00000282 b012996a 00000000 2e074ae8 00000000 Call Trace: <b012996a> hrtimer_start+0xc5/0xd0 <b0270109> do_nanosleep+0x3b/0x63 <b01299b8> hrtimer_nanosleep+0x43/0xfe <b0120244> do_sigaction+0x92/0x149 <b0129596> hrtimer_wakeup+0x0/0x1c <b0129ab0> sys_nanosleep+0x3d/0x51 <b010267b> syscall_call+0x7/0xb login S 00895440 0 3859 1 5750 3860 3843 (NOTLB) df9abf68 f7f5f580 df9aa000 00895440 00000000 00000000 b18f9d4c 7ee02065 dfb81b58 dfb81a50 b02b0480 b200c340 40787740 000f95d1 00000000 f7880740 00000140 c647b080 b026f8ed df9abfb4 00000286 00000000 00895440 00000000 Call Trace: <b026f8ed> schedule+0xa0d/0xaaf <b011a7cd> do_wait+0x8ab/0x94d <b011357b> default_wake_function+0x0/0x15 <b011a896> sys_wait4+0x27/0x2a <b010267b> syscall_call+0x7/0xb getty S 000F4240 0 3860 1 3861 3859 (NOTLB) df975edc 00000246 00000000 000f4240 00000000 00000000 00000020 b01fb342 dfb7bb98 dfb7ba90 dfb81a50 b200c340 85ce1000 000f41ff 00000000 00000000 000000ff b1fdc520 b02b5ba0 b02b5ba0 b01393de df975ec8 00000001 00000202 Call Trace: <b01fb342> do_con_write+0x141f/0x1450 <b01393de> release_pages+0x147/0x150 <b01179cd> release_console_sem+0x157/0x191 <b026fd30> schedule_timeout+0x13/0x8b <b01fb3ba> con_write+0x1d/0x23 <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb getty S 000F4240 0 3861 1 3862 3860 (NOTLB) df9a7edc 00000246 00000000 000f4240 00000000 00000000 00000020 b01fb342 dfbc7138 dfbc7030 b02b0480 b200c340 861a5b40 000f41ff 00000000 00000000 000000ff b1fdc520 b02b5ba0 b02b5ba0 b01393de df9a7ec8 00000001 00000000 Call Trace: <b01fb342> do_con_write+0x141f/0x1450 <b01393de> release_pages+0x147/0x150 <b026fd30> schedule_timeout+0x13/0x8b <b01fb3ba> con_write+0x1d/0x23 <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb getty S 000F4240 0 3862 1 3863 3861 (NOTLB) df99bedc 00000246 00000000 000f4240 00000000 00000000 861a5b40 000f41ff dfbba138 dfbba030 dfbc7030 b200c340 861a5b40 000f41ff 00000000 00000000 000000ff b1fdc520 b02b5ba0 b02b5ba0 b200c380 df99bec8 00000001 00000206 Call Trace: <b01179cd> release_console_sem+0x157/0x191 <b026fd30> schedule_timeout+0x13/0x8b <b01fb3ba> con_write+0x1d/0x23 <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb getty S 002DC6C0 0 3863 1 3864 3862 (NOTLB) f7ed3edc 00000246 00000000 002dc6c0 00000000 00000000 00000020 b01fb342 dffafb98 dffafa90 b21a0540 b2014340 85fbd6c0 000f41ff 00000001 00000000 000000ff b1fb66e0 b02b5ba0 b02b5ba0 b01393de f7ed3ec8 00000001 00000000 Call Trace: <b01fb342> do_con_write+0x141f/0x1450 <b01393de> release_pages+0x147/0x150 <b026fd30> schedule_timeout+0x13/0x8b <b01fb3ba> con_write+0x1d/0x23 <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb getty S 000F4240 0 3864 1 5964 3863 (NOTLB) df9ededc 00000246 00000000 000f4240 00000000 00000000 00000020 b01fb342 dfb75138 dfb75030 b21a0540 b2014340 8638dfc0 000f41ff 00000001 00000000 000000ff b1fb66e0 b02b5ba0 b02b5ba0 b01393de df9edec8 00000001 00000000 Call Trace: <b01fb342> do_con_write+0x141f/0x1450 <b01393de> release_pages+0x147/0x150 <b026fd30> schedule_timeout+0x13/0x8b <b01fb3ba> con_write+0x1d/0x23 <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb sshd S 08B3C880 0 3875 3778 3877 3887 (NOTLB) df99fb60 008a184e 0000012b 08b3c880 00000000 00000000 97a171c0 000f4a7c b21dfb58 b21dfa50 dffbfa50 b200c340 3f0f6200 000f4a7d 00000000 00000000 df99fb54 b0111803 00000000 00000000 00000003 00000046 f797100c 00000286 Call Trace: <b0111803> __wake_up+0x29/0x3c <b026fd30> schedule_timeout+0x13/0x8b <b01ee160> tty_poll+0x4a/0x54 <b015baca> do_select+0x3b6/0x41e <b021d346> neigh_lookup+0x9c/0xa3 <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b0226a72> qdisc_restart+0x14/0x17f <b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6 <b023717e> ip_queue_xmit+0x34d/0x38f <b01114b5> __activate_task+0x47/0x56 <b011e999> lock_timer_base+0x15/0x2f <b011f2c3> __mod_timer+0x90/0x98 <b0213117> sk_reset_timer+0x12/0x1e <b0243dea> update_send_head+0x7a/0x80 <b02468c8> __tcp_push_pending_frames+0x68f/0x74b <b0215457> __alloc_skb+0x4a/0xee <b02133db> release_sock+0xf/0x97 <b023d9ae> tcp_sendmsg+0x8ab/0x985 <b015bd1a> core_sys_select+0x1e8/0x2a2 <b0211077> do_sock_write+0xa4/0xad <b0211625> sock_aio_write+0x56/0x63 <b014b99a> do_sync_write+0xc0/0xf3 <b01271dc> autoremove_wake_function+0x0/0x3a <b015c0ca> sys_select+0x9a/0x166 <b014c630> sys_write+0x3b/0x64 <b010267b> syscall_call+0x7/0xb bash S 02CD29C0 0 3877 3875 (NOTLB) df8afedc b01ecee7 00000000 02cd29c0 00000000 00000000 df8b780c df8b7800 dffbfb58 dffbfa50 b02b0480 b200c340 3f0f6200 000f4a7d 00000000 00000010 00000000 df8aff14 dffbfa50 df8affbc b01209c2 f7814ac0 f7814c18 00000000 Call Trace: <b01ecee7> tty_ldisc_deref+0x52/0x61 <b01209c2> dequeue_signal+0x36/0xa4 <b026fd30> schedule_timeout+0x13/0x8b <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb sshd S 1BA81400 0 3887 3778 3889 5760 3875 (NOTLB) df8b1b60 df8b1b58 df8b1b58 1ba81400 00000000 00000000 474e1feb 000f4a7e dffbb648 dffbb540 b02b0480 b200c340 4758da80 000f4a7e 00000000 00000001 df8b1b54 b0111803 00000000 00000000 00000003 00000046 f780480c 00000000 Call Trace: <b0111803> __wake_up+0x29/0x3c <b026fd30> schedule_timeout+0x13/0x8b <b01ee160> tty_poll+0x4a/0x54 <b015baca> do_select+0x3b6/0x41e <b024a28e> tcp_v4_rcv+0x3ed/0x8dd <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b0226a72> qdisc_restart+0x14/0x17f <b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6 <b023717e> ip_queue_xmit+0x34d/0x38f <b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6 <b023717e> ip_queue_xmit+0x34d/0x38f <b01055a9> timer_interrupt+0x72/0x7a <b011e999> lock_timer_base+0x15/0x2f <b011f2c3> __mod_timer+0x90/0x98 <b0213117> sk_reset_timer+0x12/0x1e <b0243dea> update_send_head+0x7a/0x80 <b02468c8> __tcp_push_pending_frames+0x68f/0x74b <b0104a5b> do_IRQ+0x22/0x2b <b0215457> __alloc_skb+0x4a/0xee <b02133db> release_sock+0xf/0x97 <b023d9ae> tcp_sendmsg+0x8ab/0x985 <b015bd1a> core_sys_select+0x1e8/0x2a2 <b0211077> do_sock_write+0xa4/0xad <b0211625> sock_aio_write+0x56/0x63 <b014b99a> do_sync_write+0xc0/0xf3 <b01271dc> autoremove_wake_function+0x0/0x3a <b015c0ca> sys_select+0x9a/0x166 <b014c630> sys_write+0x3b/0x64 <b010267b> syscall_call+0x7/0xb bash S 02CD29C0 0 3889 3887 (NOTLB) df9a3edc b01ecee7 00000000 02cd29c0 00000000 00000000 4758da80 000f4a7e dff18b98 dff18a90 dffbb540 b200c340 4758da80 000f4a7e 00000000 00000000 00000016 0202c205 00000002 1b02000a b200c380 f792d280 f7f8c634 f7f87280 Call Trace: <b01ecee7> tty_ldisc_deref+0x52/0x61 <b026fd30> schedule_timeout+0x13/0x8b <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb bash S 01312D00 0 5750 3859 (NOTLB) b85cfedc 00000246 00000000 01312d00 00000000 000186a0 00000020 b01fb342 dfe73138 dfe73030 b02b0480 b200c340 05584d80 000f966b 00000000 00000000 000000ff 00000297 b01ba027 000003d4 00000297 b201e000 df808800 00000000 Call Trace: <b01fb342> do_con_write+0x141f/0x1450 <b01ba027> vgacon_set_cursor_size+0x34/0xd1 <b026fd30> schedule_timeout+0x13/0x8b <b01179cd> release_console_sem+0x157/0x191 <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb sshd S 0F518240 0 5760 3778 5762 5787 3887 (NOTLB) effd1b60 058431d4 00000127 0f518240 00000000 00000000 00000000 b011bdf9 dfbba648 dfbba540 b02b0480 b200c340 61511d00 000f966c 00000000 00000001 effd1b54 b0111803 00000000 00000000 00000003 00000046 f7d6200c 00000000 Call Trace: <b011bdf9> __do_softirq+0x57/0xc0 <b0111803> __wake_up+0x29/0x3c <b026fd30> schedule_timeout+0x13/0x8b <b01ee160> tty_poll+0x4a/0x54 <b015baca> do_select+0x3b6/0x41e <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b0226a72> qdisc_restart+0x14/0x17f <b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6 <b023717e> ip_queue_xmit+0x34d/0x38f <b021b373> netif_receive_skb+0x22c/0x24a <f89008a0> tg3_poll+0x5e8/0x643 [tg3] <b011e999> lock_timer_base+0x15/0x2f <b011f2c3> __mod_timer+0x90/0x98 <b0213117> sk_reset_timer+0x12/0x1e <b0243dea> update_send_head+0x7a/0x80 <b02468c8> __tcp_push_pending_frames+0x68f/0x74b <b0104a5b> do_IRQ+0x22/0x2b <b0215457> __alloc_skb+0x4a/0xee <b02133db> release_sock+0xf/0x97 <b023d9ae> tcp_sendmsg+0x8ab/0x985 <b015bd1a> core_sys_select+0x1e8/0x2a2 <b0211077> do_sock_write+0xa4/0xad <b0211625> sock_aio_write+0x56/0x63 <b014b99a> do_sync_write+0xc0/0xf3 <b01271dc> autoremove_wake_function+0x0/0x3a <b015c0ca> sys_select+0x9a/0x166 <b014c630> sys_write+0x3b/0x64 <b010267b> syscall_call+0x7/0xb bash S 068E7780 0 5762 5760 6115 (NOTLB) d3b45f54 b013d869 b1d3cc80 068e7780 00000000 00000000 dfeb9740 b1d3cc80 dffbf138 dffbf030 dfbc7a50 b200c340 68f58540 000f9654 00000000 00000000 bfdb9080 b134470c 69e64065 f7d6f574 0000001d 000003d8 f789ee00 aff805a4 Call Trace: <b013d869> do_wp_page+0x268/0x286 <b011a7cd> do_wait+0x8ab/0x94d <b011357b> default_wake_function+0x0/0x15 <b011a896> sys_wait4+0x27/0x2a <b011a8ac> sys_waitpid+0x13/0x17 <b010267b> syscall_call+0x7/0xb sshd S 08C30AC0 0 5787 3778 5789 5760 (NOTLB) b5dbfb60 f792d2e0 000005ea 08c30ac0 00000000 00000000 a4ab0bc0 000f966b dfbc2648 dfbc2540 b02b0480 b200c340 a5ccf680 000f966b 00000000 00000001 b5dbfb54 b0111803 00000000 00000000 00000003 00000046 dcd5400c 00000000 Call Trace: <b0111803> __wake_up+0x29/0x3c <b026fd30> schedule_timeout+0x13/0x8b <b01ee160> tty_poll+0x4a/0x54 <b015baca> do_select+0x3b6/0x41e <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b011357b> default_wake_function+0x0/0x15 <b0226a72> qdisc_restart+0x14/0x17f <b021aff9> dev_queue_xmit+0x1e5/0x1eb <b02377ea> ip_output+0x1c0/0x1f6 <b023717e> ip_queue_xmit+0x34d/0x38f <b011bdf9> __do_softirq+0x57/0xc0 <b010314c> apic_timer_interrupt+0x1c/0x24 <b0244573> tcp_transmit_skb+0x5dc/0x600 <b02468c8> __tcp_push_pending_frames+0x68f/0x74b <b0215457> __alloc_skb+0x4a/0xee <b02133db> release_sock+0xf/0x97 <b023d9ae> tcp_sendmsg+0x8ab/0x985 <b015bd1a> core_sys_select+0x1e8/0x2a2 <b0211077> do_sock_write+0xa4/0xad <b0211625> sock_aio_write+0x56/0x63 <b014b99a> do_sync_write+0xc0/0xf3 <b01271dc> autoremove_wake_function+0x0/0x3a <b015c0ca> sys_select+0x9a/0x166 <b014c630> sys_write+0x3b/0x64 <b010267b> syscall_call+0x7/0xb bash S 055D4A80 0 5789 5787 (NOTLB) de097edc b01ecee7 00000000 055d4a80 00000000 0001b207 a5ccf680 000f966b dfbc2b58 dfbc2a50 dfbc2540 b200c340 a5ccf680 000f966b 00000000 00000000 00000000 de097f14 dfbc2a50 de097fbc b200c380 f7f3f580 f7f3f6d8 f7f3f6c4 Call Trace: <b01ecee7> tty_ldisc_deref+0x52/0x61 <b01216d0> get_signal_to_deliver+0x1c0/0x372 <b026fd30> schedule_timeout+0x13/0x8b <b01273fc> add_wait_queue+0x12/0x30 <b01f2383> read_chan+0x321/0x53e <b011357b> default_wake_function+0x0/0x15 <b01ee431> tty_read+0x5d/0x9f <b014bc1d> vfs_read+0xa3/0x13a <b014c5cc> sys_read+0x3b/0x64 <b010267b> syscall_call+0x7/0xb pdflush D 37E8BE80 0 5842 11 6085 3699 (L-TLB) c7803eec b1bfa900 b1bfa920 37e8be80 00000000 00000000 fc4c7fab 000f9668 dfe67688 dfe67580 b21a0540 b2014340 805ef140 000f966c 00000001 00000000 00000286 b011e999 00000286 b011e999 c7803efc c7803efc 00000286 00000000 Call Trace: <b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f <b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9 <b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5 <b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a <b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5 <b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0 <b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb syslogd S 029F6300 0 5964 1 6051 3864 (NOTLB) c8d39b60 f88df961 c2e1aed8 029f6300 00000000 00000000 f7c7c260 00001000 dffbd178 dffbd070 b02b0480 b200c340 baf35680 000f9666 00000000 00000000 b013595f dffe2340 00011200 b733848c 00000246 b01273fc df9d4680 00000000 Call Trace: <f88df961> ext3_mark_iloc_dirty+0x2df/0x345 [ext3] <b013595f> mempool_alloc+0x21/0xbf <b01273fc> add_wait_queue+0x12/0x30 <b026fd30> schedule_timeout+0x13/0x8b <b02112c0> sock_poll+0x13/0x17 <b015baca> do_select+0x3b6/0x41e <b015b65c> __pollwait+0x0/0xb8 <b011357b> default_wake_function+0x0/0x15 <f887355d> __journal_file_buffer+0x111/0x1e7 [jbd] <f8874995> do_get_write_access+0x44a/0x45c [jbd] <f887355d> __journal_file_buffer+0x111/0x1e7 [jbd] <f8874df9> journal_dirty_metadata+0x173/0x19e [jbd] <b013595f> mempool_alloc+0x21/0xbf <b013595f> mempool_alloc+0x21/0xbf <b01a543b> as_set_request+0x1c/0x6d <b01a5a45> as_update_iohist+0x38/0x2a5 <b01a63ee> as_add_request+0xc0/0xef <b019ea34> elv_insert+0x9b/0x138 <b01a27e6> __make_request+0x319/0x350 <b01342a3> generic_file_buffered_write+0x482/0x56f <b01a0268> generic_make_request+0x168/0x17a <b013595f> mempool_alloc+0x21/0xbf <b01a0ad6> submit_bio+0xa5/0xaa <b015bd1a> core_sys_select+0x1e8/0x2a2 <b0133749> wait_on_page_writeback_range+0xb3/0xf7 <b011f676> __sigqueue_free+0x2a/0x2f <b0120981> __dequeue_signal+0x14b/0x156 <b01053b7> do_gettimeofday+0x1b/0x9d <b01053b7> do_gettimeofday+0x1b/0x9d <b011b6ba> getnstimeofday+0xf/0x25 <b01acb36> rb_insert_color+0xa6/0xc8 <b01294f4> enqueue_hrtimer+0x58/0x7f <b012996a> hrtimer_start+0xc5/0xd0 <b011abdd> do_setitimer+0x16c/0x47d <b015c0ca> sys_select+0x9a/0x166 <b0101fc0> sys_sigreturn+0x98/0xbd <b010267b> syscall_call+0x7/0xb klogd R running 0 6051 1 5964 (NOTLB) pdflush D 0E205540 0 6085 11 6130 5842 (L-TLB) b4d01eec f792d280 00000204 0e205540 00000000 00000000 ad7d58c0 000f966b dffbbb58 dffbba50 b21a0540 b2014340 7e1b1bc0 000f966c 00000001 00000000 00000286 b011e999 00000286 b011e999 b4d01efc b4d01efc 00000286 00000000 Call Trace: <b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f <b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9 <b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5 <b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a <b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5 <b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0 <b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb ncftpget D 568FC100 0 6115 5762 (NOTLB) cb351a88 87280008 b013595f 568fc100 00000008 00051615 f2a64c80 b21cc340 dfbc7b58 dfbc7a50 b02b0480 b200c340 77405900 000f966c 00000000 00000282 b011e999 f2a64cd8 00000000 dfec8b84 00000000 dfec8b84 b01a216b 00000000 Call Trace: <b013595f> mempool_alloc+0x21/0xbf <b011e999> lock_timer_base+0x15/0x2f <b01a216b> get_request+0x55/0x283 <b026f9b5> io_schedule+0x26/0x30 <b01a2432> get_request_wait+0x99/0xd3 <b01271dc> autoremove_wake_function+0x0/0x3a <b01a2786> __make_request+0x2b9/0x350 <b01a0268> generic_make_request+0x168/0x17a <b013595f> mempool_alloc+0x21/0xbf <b01a0ad6> submit_bio+0xa5/0xaa <b015032c> bio_alloc+0x13/0x22 <b014dad0> submit_bh+0xe6/0x107 <b014f9d9> __block_write_full_page+0x20e/0x301 <f88e17e8> ext3_get_block+0x0/0xad [ext3] <f88e0633> ext3_ordered_writepage+0xcb/0x137 [ext3] <f88e17e8> ext3_get_block+0x0/0xad [ext3] <f88def99> bget_one+0x0/0xb [ext3] <b016a1d3> mpage_writepages+0x193/0x2e9 <f88e0568> ext3_ordered_writepage+0x0/0x137 [ext3] <b013813f> do_writepages+0x30/0x39 <b0168988> __writeback_single_inode+0x166/0x2e2 <b01aa9d3> __next_cpu+0x11/0x20 <b0136421> read_page_state_offset+0x33/0x41 <b0168f5e> sync_sb_inodes+0x185/0x23a <b01691c6> writeback_inodes+0x6e/0xbb <b0138246> balance_dirty_pages_ratelimited_nr+0xcb/0x152 <b013429e> generic_file_buffered_write+0x47d/0x56f <f88e698a> __ext3_journal_stop+0x19/0x37 [ext3] <f88dfde0> ext3_dirty_inode+0x5e/0x64 [ext3] <b0168b4c> __mark_inode_dirty+0x28/0x14c <b01354da> __generic_file_aio_write_nolock+0x3c8/0x405 <b0211c91> sock_aio_read+0x56/0x63 <b013573c> generic_file_aio_write+0x61/0xb3 <f88dde72> ext3_file_write+0x26/0x92 [ext3] <b014b99a> do_sync_write+0xc0/0xf3 <b016201a> notify_change+0x2d4/0x2e5 <b01271dc> autoremove_wake_function+0x0/0x3a <b014bdb0> vfs_write+0xa3/0x13a <b014c630> sys_write+0x3b/0x64 <b010267b> syscall_call+0x7/0xb pdflush D 00F42400 0 6130 11 6146 6085 (L-TLB) f57ddeec b1964a00 b1fdf940 00f42400 00000000 00000000 b1c75ca0 b1f29320 dfb75648 dfb75540 b21a0540 b2014340 7b01a6c0 000f966c 00000001 00000000 00000286 b011e999 00000286 b011e999 f57ddefc f57ddefc 00000286 00000000 Call Trace: <b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f <b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9 <b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5 <b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a <b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5 <b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0 <b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb pdflush D 004C4B40 0 6146 11 6156 6130 (L-TLB) dec91eec 00000001 b02f20e0 004c4b40 00000000 00000000 00000000 b02f20e0 dfb80178 dfb80070 b21a0540 b2014340 7b8afb00 000f966c 00000001 00000000 00000286 b011e999 00000286 b011e999 dec91efc dec91efc 00000286 00000000 Call Trace: <b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f <b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9 <b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5 <b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a <b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5 <b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0 <b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb pdflush D 04B571C0 0 6156 11 6168 6146 (L-TLB) d959feec b1b7c500 b1f0e020 04b571c0 00000000 00000000 14c754c0 000f9669 dff15b98 dff15a90 b21a0540 b2014340 7b10e900 000f966c 00000001 00000000 00000286 b011e999 00000286 b011e999 d959fefc d959fefc 00000286 00000000 Call Trace: <b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f <b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9 <b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5 <b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a <b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5 <b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0 <b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb pdflush D 000F4240 0 6168 11 6156 (L-TLB) caa13eec 0000000a b23ec9cc 000f4240 00000000 00000000 b01117b5 ebb31f74 b21dd178 b21dd070 b21a0540 b2014340 7ee17900 000f966c 00000001 00000000 00000286 b011e999 00000286 b011e999 caa13efc caa13efc 00000286 00000000 Call Trace: <b01117b5> __wake_up_common+0x2a/0x4f <b011e999> lock_timer_base+0x15/0x2f <b011e999> lock_timer_base+0x15/0x2f <b026fd89> schedule_timeout+0x6c/0x8b <b011ecff> process_timeout+0x0/0x9 <b026ebf5> io_schedule_timeout+0x29/0x33 <b01387eb> pdflush+0x0/0x1b5 <b01a0a1d> blk_congestion_wait+0x55/0x69 <b01271dc> autoremove_wake_function+0x0/0x3a <b0137f01> background_writeout+0x7d/0x8b <b01388ee> pdflush+0x103/0x1b5 <b0137e84> background_writeout+0x0/0x8b <b01271a1> kthread+0xa3/0xd0 <b01270fe> kthread+0x0/0xd0 <b0100bc5> kernel_thread_helper+0x5/0xb
| |