lkml.org 
[lkml]   [2007]   [Jan]   [29]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    SubjectRE: fsync occasionally very slow
    Date
    Regarding the long fsyncs, here's a trace...

    I upgraded to a more recent kernel - 2.6.18.6 - and ran it on a workstation.
    This particular box has In this case the elevator is CFQ.

    This sample came from a stall that lasted about 2.5 minutes(!) - the longest
    one I've seen yet. The box is a bit more memory constrained than the
    original system but exhibits similar behavior. It doesn't page. Also,
    there is no raid card - simply striped PATA drives.

    Thanks!
    Matt

    2 5407 5405 (NOTLB)
    e2cf9eb4 00000086 00000001 00000286 c0141c54 c1467ec0 00000000
    c18084e0
    c04ea4e0 e06c7680 00000000 00003c4f 6cd9179d 0000001b c040e3e0
    e06c7570
    7fffffff 08a4b310 c04ff5d0 00000001 c02d06ba c012e965 f7fdb3c0
    08a4b000
    Call Trace:
    [<c0141c54>] buffered_rmqueue+0x13a/0x173
    [<c02d06ba>] schedule_timeout+0x13/0x8e
    [<c012e965>] get_futex_key+0x3d/0xd3
    [<c012baa1>] add_wait_queue+0x12/0x30
    [<c012f9f4>] futex_wait+0x18c/0x1f2
    [<c014ac97>] __handle_mm_fault+0x185/0x28c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c013061d>] do_futex+0x35/0x95
    [<c0130778>] sys_futex+0xfb/0x10a
    [<c0114386>] schedule_tail+0x2e/0x81
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    PolicyScrubbe S 2C3D55DE 0 5407 5392 5408 5406 (NOTLB)
    e1ea5f2c 00000086 c012e032 2c3d55de 000001de 00000003 f6c9b570
    c18084e0
    c04ea4e0 e06c7140 00000000 00000bbe 2c3d55de 000001de f6c9b570
    e06c7030
    e1ea5f68 00000000 00000001 af637014 c02d0aec e1ea5f68 09bdcead
    000001dd
    Call Trace:
    [<c012e032>] enqueue_hrtimer+0x58/0x7f
    [<c02d0aec>] do_nanosleep+0x40/0x6d
    [<c012e3e1>] hrtimer_nanosleep+0x4c/0xe9
    [<c0126396>] sys_rt_sigaction+0x76/0x8d
    [<c012e366>] hrtimer_wakeup+0x0/0x1c
    [<c012e4c2>] sys_nanosleep+0x44/0x4f
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    PolicyScrubbe S 00000000 0 5408 5392 5407 (NOTLB)
    de739b50 00000086 e4d15030 00000000 00000213 00000000 e4d15030
    c18084e0
    c04ea4e0 e1ee3bc0 00000000 00000764 299782a2 000001c2 e4d15030
    e1ee3ab0
    de739b60 00099160 00001000 00000000 c02d0716 de739b60 00099160
    c04fdcc0
    Call Trace:
    [<c02d0716>] schedule_timeout+0x6f/0x8e
    [<c01234b2>] process_timeout+0x0/0x9
    [<c016a42f>] do_select+0x2fa/0x34b
    [<c016a020>] __pollwait+0x0/0x4c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b5e>] __wake_up_common+0x2f/0x4b
    [<c0115ba1>] __wake_up+0x27/0x3b
    [<c026c573>] sock_def_readable+0x33/0x61
    [<c029c433>] tcp_rcv_established+0x382/0x5da
    [<c02a332d>] tcp_v4_do_rcv+0x1f/0xb5
    [<c02a38ec>] tcp_v4_rcv+0x529/0x86f
    [<c011386c>] activate_task+0x55/0xa2
    [<c0113769>] __activate_task+0x1e/0x30
    [<c0113925>] resched_task+0x41/0x63
    [<c011401e>] try_to_wake_up+0x2ab/0x2b6
    [<c026d1c3>] __alloc_skb+0x49/0xf0
    [<c0115b5e>] __wake_up_common+0x2f/0x4b
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c026ec50>] skb_dequeue+0x3b/0x41
    [<c02c65f8>] unix_stream_recvmsg+0xd1/0x350
    [<c02c67f1>] unix_stream_recvmsg+0x2ca/0x350
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a705>] core_sys_select+0x285/0x365
    [<c02cfee7>] schedule+0x60b/0x625
    [<c02d06ba>] schedule_timeout+0x13/0x8e
    [<c012e965>] get_futex_key+0x3d/0xd3
    [<c012f7e9>] unqueue_me+0x84/0x8b
    [<c012fa10>] futex_wait+0x1a8/0x1f2
    [<c014cb5a>] find_extend_vma+0x17/0x58
    [<c012e965>] get_futex_key+0x3d/0xd3
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a882>] sys_select+0x9d/0x161
    [<c011e597>] sys_gettimeofday+0x22/0x55
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    gnome-termina S 1FC984CA 0 5410 1 5411 5413 5396 (NOTLB)
    df785bd0 00000082 c01e93d8 1fc984ca 000001df 00000002 e4509570
    c18084e0
    c04ea4e0 e1ee3140 00000000 0000022f 1fca0963 000001df e4509570
    e1ee3030
    df785be0 0006ab15 e45b1980 00000000 c02d0716 df785be0 0006ab15
    c04fd238
    Call Trace:
    [<c01e93d8>] tty_ldisc_try+0x33/0x38
    [<c02d0716>] schedule_timeout+0x6f/0x8e
    [<c01234b2>] process_timeout+0x0/0x9
    [<c016acff>] do_poll+0x15b/0x19a
    [<c016aeb3>] do_sys_poll+0x175/0x21b
    [<c026d1c3>] __alloc_skb+0x49/0xf0
    [<c011386c>] activate_task+0x55/0xa2
    [<c0113769>] __activate_task+0x1e/0x30
    [<c02d13a6>] _spin_lock+0xd/0x59
    [<c011356f>] task_rq_lock+0x2c/0x52
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c011401e>] try_to_wake_up+0x2ab/0x2b6
    [<c016a020>] __pollwait+0x0/0x4c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0159ab0>] do_sync_write+0xbe/0xf1
    [<c02c6a5a>] unix_ioctl+0xa4/0xb2
    [<c0269b03>] sock_ioctl+0x1a6/0x1c3
    [<c0169804>] do_ioctl+0x28/0x65
    [<c012274f>] do_gettimeofday+0x2e/0x117
    [<c01cf017>] copy_to_user+0x38/0x42
    [<c016af9c>] sys_poll+0x43/0x47
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    gnome-termina S BAC48363 0 5413 1 5443 5410 (NOTLB)
    f1491ec4 00000082 00000000 bac48363 0000001b 00000003 e6672570
    c18084e0
    c04ea4e0 dd45c680 00000000 00002a5d bac48363 0000001b e6672570
    dd45c570
    e6f3b400 b792f3e0 f1491eec e6f3b400 c016415d 00000000 dd45c570
    c012bc09
    Call Trace:
    [<c016415d>] pipe_wait+0x6d/0x8e
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c01645e0>] pipe_readv+0x209/0x275
    [<c0164671>] pipe_read+0x25/0x29
    [<c015995c>] vfs_read+0x87/0x11d
    [<c0159c3b>] sys_read+0x3b/0x63
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    gnome-pty-hel S C019B2D0 0 5411 5410 5412 (NOTLB)
    df7bfd4c 00000086 00000b80 c019b2d0 f75453c4 00000000 e1ee3030
    c18084e0
    c04ea4e0 e1ee3680 00000000 00004eb4 b9572a54 0000001b e1ee3030
    e1ee3570
    7fffffff dfaf3a80 df7bfd94 7fffffff c02d06ba 00000180 00000a00
    7fffffff
    Call Trace:
    [<c019b2d0>] __ext3_journal_stop+0x19/0x37
    [<c02d06ba>] schedule_timeout+0x13/0x8e
    [<c012bb34>] prepare_to_wait+0x12/0x4c
    [<c02c64ef>] unix_stream_data_wait+0xb8/0xf0
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c0197998>] ext3_dirty_inode+0x5d/0x63
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c02c6663>] unix_stream_recvmsg+0x13c/0x350
    [<c02695d7>] do_sock_read+0x83/0x8b
    [<c0269723>] sock_aio_read+0x56/0x64
    [<c01598a2>] do_sync_read+0xbe/0xf1
    [<c012ddd3>] ktime_get_ts+0x4b/0x51
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c0171ca7>] mntput_no_expire+0x14/0x6c
    [<c015a78c>] __fput+0x150/0x157
    [<c0159967>] vfs_read+0x92/0x11d
    [<c0159c3b>] sys_read+0x3b/0x63
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    bash S 57E802AC 0 5412 5410 6741 5411 (NOTLB)
    e08aff30 00000086 fffb8000 57e802ac 00000159 00000002 e4502ab0
    c18084e0
    c04ea4e0 dd45cbc0 00000000 000050a2 57f07a0a 00000159 e4502ab0
    dd45cab0
    fffffe00 dd45cab0 dd45cb68 00000000 c011db3a 00000001 00000000
    dd45cab0
    Call Trace:
    [<c011db3a>] do_wait+0x287/0x31e
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c01cf017>] copy_to_user+0x38/0x42
    [<c011dc66>] sys_wait4+0x28/0x2c
    [<c011dc7d>] sys_waitpid+0x13/0x1a
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    ntpd S F99DA445 0 5675 1 5464 (NOTLB)
    ddbbfb58 00000082 c040e3e0 f99da445 000001de 00000003 f7e1f570
    c18084e0
    c04ea4e0 ddff3bc0 00000000 00000c3b f99da445 000001de f7e1f570
    ddff3ab0
    7fffffff 00000000 00000100 00000000 c02d06ba ddbbfbd0 e46749c0
    00000000
    Call Trace:
    [<c02d06ba>] schedule_timeout+0x13/0x8e
    [<c0269b5a>] sock_poll+0x13/0x17
    [<c016a42f>] do_select+0x2fa/0x34b
    [<c016a020>] __pollwait+0x0/0x4c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c011eec7>] local_bh_enable+0x7b/0x91
    [<c027277d>] dev_queue_xmit+0x230/0x237
    [<c028e934>] ip_output+0x1c6/0x1fc
    [<c02905e8>] ip_push_pending_frames+0x36f/0x3fa
    [<c026ff0e>] skb_copy_and_csum_datagram+0x8b/0x31c
    [<c0270259>] skb_copy_and_csum_datagram_iovec+0x65/0xb7
    [<c026d4aa>] kfree_skbmem+0x6c/0x70
    [<c02a81b4>] udp_recvmsg+0x190/0x1d9
    [<c026c9a9>] sock_common_recvmsg+0x36/0x4b
    [<c0269456>] sock_recvmsg+0xda/0xf5
    [<c026e6ce>] skb_checksum+0x45/0x25d
    [<c026c695>] sk_reset_timer+0x12/0x1e
    [<c02a03cd>] tcp_send_delayed_ack+0xb3/0xbb
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a705>] core_sys_select+0x285/0x365
    [<c010739c>] convert_fxsr_to_user+0x100/0x157
    [<c01cee56>] __copy_to_user_ll+0x55/0x5f
    [<c01075a6>] save_i387_fxsave+0xd7/0xe9
    [<c010214b>] setup_sigcontext+0xd9/0x11e
    [<c010225c>] setup_frame+0xcc/0x1b7
    [<c0102662>] handle_signal+0x9a/0x10e
    [<c02d16c2>] _spin_unlock_irq+0xe/0x22
    [<c01026d0>] handle_signal+0x108/0x10e
    [<c0102750>] do_signal+0x7a/0xfb
    [<c010740b>] convert_fxsr_from_user+0x18/0xdc
    [<c016a882>] sys_select+0x9d/0x161
    [<c0101e29>] restore_sigcontext+0x107/0x184
    [<c0101f57>] sys_sigreturn+0xb1/0xd7
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    in.telnetd S 2397C627 0 6556 4054 6568 6829 (NOTLB)
    c9613b58 00000086 c041c9a0 2397c627 000001df 00000002 e4502570
    c18084e0
    c04ea4e0 f43aa140 00000000 00000a63 2398171b 000001df e4502570
    f43aa030
    7fffffff 00000000 00000010 00000000 c02d06ba 00000104 c02d16a0
    c01eb51f
    Call Trace:
    [<c02d06ba>] schedule_timeout+0x13/0x8e
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c01eb51f>] tty_poll+0x4a/0x54
    [<c016a42f>] do_select+0x2fa/0x34b
    [<c016a020>] __pollwait+0x0/0x4c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0217ba7>] e100_xmit_prepare+0x0/0x90
    [<c027eef7>] __qdisc_run+0x34/0x1a3
    [<c011eec7>] local_bh_enable+0x7b/0x91
    [<c027277d>] dev_queue_xmit+0x230/0x237
    [<c028e934>] ip_output+0x1c6/0x1fc
    [<c028ed28>] ip_queue_xmit+0x3be/0x40c
    [<c028b7d6>] ip_local_deliver+0x218/0x235
    [<c011386c>] activate_task+0x55/0xa2
    [<c0113769>] __activate_task+0x1e/0x30
    [<c0113fd5>] try_to_wake_up+0x262/0x2b6
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c011386c>] activate_task+0x55/0xa2
    [<c02a244a>] tcp_v4_send_check+0x83/0xc4
    [<c01221bd>] lock_timer_base+0x19/0x33
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c0122266>] __mod_timer+0x8f/0x99
    [<c026c695>] sk_reset_timer+0x12/0x1e
    [<c029ee52>] tcp_write_xmit+0x1bc/0x212
    [<c029ee7e>] tcp_write_xmit+0x1e8/0x212
    [<c029eecb>] __tcp_push_pending_frames+0x23/0x80
    [<c011ed36>] local_bh_disable+0xa/0xb
    [<c011ef1d>] local_bh_enable_ip+0x40/0x56
    [<c02d16eb>] _spin_unlock_bh+0x15/0x17
    [<c029532b>] tcp_sendmsg+0x954/0xa02
    [<c029532b>] tcp_sendmsg+0x954/0xa02
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a705>] core_sys_select+0x285/0x365
    [<c02698ef>] sock_aio_write+0x56/0x64
    [<c0159ab0>] do_sync_write+0xbe/0xf1
    [<c01ee5b1>] read_chan+0x4b8/0x4c5
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c011eb3a>] current_fs_time+0x44/0x4e
    [<c016a882>] sys_select+0x9d/0x161
    [<c0159c9e>] sys_write+0x3b/0x63
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    login S C04EA4E0 0 6568 6556 6609 (NOTLB)
    c9b47f44 00000082 00000012 c04ea4e0 c10ca50c 3fa32065 00000000
    c18084e0
    c04ea4e0 df258140 00000000 00001291 11f54f26 0000013a c040e3e0
    df258030
    fffffe00 df258030 df2580e8 00000000 c011db3a 00000001 00000000
    df258030
    Call Trace:
    [<c011db3a>] do_wait+0x287/0x31e
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c011216e>] do_page_fault+0x225/0x565
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c011dc66>] sys_wait4+0x28/0x2c
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    bash R running 0 6609 6568 (NOTLB)
    DConsole S C0113925 0 6741 5412 6749 (NOTLB)
    c7619eac 00000086 f43aaab0 c0113925 e1ee3030 f43aaab0 c7619ebc
    c18084e0
    c04ea4e0 f43aabc0 00000000 000003f8 7b69c894 0000015c c040e3e0
    f43aaab0
    7fffffff e3493180 c7619f0c e0147800 c02d06ba f7b6c900 f7af9c80
    e45b1980
    Call Trace:
    [<c0113925>] resched_task+0x41/0x63
    [<c02d06ba>] schedule_timeout+0x13/0x8e
    [<c012baa1>] add_wait_queue+0x12/0x30
    [<c01ee3a8>] read_chan+0x2af/0x4c5
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c01e94e7>] tty_ldisc_deref+0x52/0x61
    [<c01ea094>] tty_read+0x67/0x95
    [<c015995c>] vfs_read+0x87/0x11d
    [<c0159c3b>] sys_read+0x3b/0x63
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    DConsole S 006E106D 0 6749 5412 6741 (NOTLB)
    dbd11f2c 00000086 c012e032 006e106d 000001da 00000003 e6672570
    c18084e0
    c04ea4e0 f5cf0680 00000000 00001007 006e106d 000001da e6672570
    f5cf0570
    dbd11f68 00000000 00000001 00000000 c02d0aec dbd11f68 aa2b65d0
    000001e5
    Call Trace:
    [<c012e032>] enqueue_hrtimer+0x58/0x7f
    [<c02d0aec>] do_nanosleep+0x40/0x6d
    [<c012e3e1>] hrtimer_nanosleep+0x4c/0xe9
    [<c01621c1>] sys_stat64+0x20/0x27
    [<c012e366>] hrtimer_wakeup+0x0/0x1c
    [<c012e4c2>] sys_nanosleep+0x44/0x4f
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    CompressVersi D F7406880 0 6824 6823 (NOTLB)
    c9d8dea8 00000082 deed1ab0 f7406880 00000000 00000001 deee4030
    c18084e0
    c04ea4e0 c985e680 00000000 00000127 0517fc12 000001df deee4030
    c985e570
    f7406800 c9d8ded4 f7406860 00025918 c01a847b 00000000 00000000
    c985e570
    Call Trace:
    [<c01a847b>] log_wait_commit+0xc3/0xfe
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c02d071e>] schedule_timeout+0x77/0x8e
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c01234b2>] process_timeout+0x0/0x9
    [<c01a4085>] journal_stop+0x199/0x1c9
    [<c01a40d5>] journal_force_commit+0x20/0x26
    [<c019d7ed>] ext3_force_commit+0x22/0x24
    [<c019341a>] ext3_sync_file+0x82/0xb4
    [<c013d95b>] filemap_fdatawrite+0x16/0x1a
    [<c015ae2c>] do_fsync+0x41/0x65
    [<c015ae70>] __do_fsync+0x20/0x30
    [<c015ae8b>] sys_fsync+0xb/0xe
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    in.telnetd S C041C998 0 6829 4054 6830 6902 6556 (NOTLB)
    ce2dbb58 00000082 c041c9a0 c041c998 00000000 00000000 c2395030
    c18084e0
    c04ea4e0 f7e76bc0 00000000 00001a1c f158733f 00000180 c2395030
    f7e76ab0
    7fffffff 00000000 00000010 00000000 c02d06ba 00000104 c02d16a0
    c01eb51f
    Call Trace:
    [<c02d06ba>] schedule_timeout+0x13/0x8e
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c01eb51f>] tty_poll+0x4a/0x54
    [<c016a42f>] do_select+0x2fa/0x34b
    [<c016a020>] __pollwait+0x0/0x4c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0217ba7>] e100_xmit_prepare+0x0/0x90
    [<c027eef7>] __qdisc_run+0x34/0x1a3
    [<c011eec7>] local_bh_enable+0x7b/0x91
    [<c027277d>] dev_queue_xmit+0x230/0x237
    [<c028e934>] ip_output+0x1c6/0x1fc
    [<c028ed28>] ip_queue_xmit+0x3be/0x40c
    [<c02167fb>] e100_enable_irq+0x13/0x31
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c0218503>] e100_poll+0x91/0xab
    [<c0272ef2>] net_rx_action+0x97/0x17f
    [<c01edc7c>] n_tty_receive_buf+0x83d/0x899
    [<c011f059>] irq_exit+0x36/0x41
    [<c029d448>] tcp_cwnd_restart+0x1e/0xb4
    [<c02a244a>] tcp_v4_send_check+0x83/0xc4
    [<c01221bd>] lock_timer_base+0x19/0x33
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c0122266>] __mod_timer+0x8f/0x99
    [<c026c695>] sk_reset_timer+0x12/0x1e
    [<c029ee52>] tcp_write_xmit+0x1bc/0x212
    [<c029ee7e>] tcp_write_xmit+0x1e8/0x212
    [<c029eecb>] __tcp_push_pending_frames+0x23/0x80
    [<c011ed36>] local_bh_disable+0xa/0xb
    [<c011ef1d>] local_bh_enable_ip+0x40/0x56
    [<c02d16eb>] _spin_unlock_bh+0x15/0x17
    [<c029532b>] tcp_sendmsg+0x954/0xa02
    [<c029532b>] tcp_sendmsg+0x954/0xa02
    [<c01ece51>] opost+0x197/0x1a0
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a705>] core_sys_select+0x285/0x365
    [<c02698ef>] sock_aio_write+0x56/0x64
    [<c0115ba1>] __wake_up+0x27/0x3b
    [<c0159ab0>] do_sync_write+0xbe/0xf1
    [<c01ee5b1>] read_chan+0x4b8/0x4c5
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c011eb3a>] current_fs_time+0x44/0x4e
    [<c016a882>] sys_select+0x9d/0x161
    [<c0159c9e>] sys_write+0x3b/0x63
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    login S 00000004 0 6830 6829 6836 (NOTLB)
    c9eb1f44 00000082 00000004 00000004 c9eb1f54 00000000 deee4030
    c18084e0
    c04ea4e0 dfca3140 00000000 000001b1 650f3c84 00000170 deee4030
    dfca3030
    fffffe00 dfca3030 dfca30e8 00000000 c011db3a 00000001 00000000
    dfca3030
    Call Trace:
    [<c011db3a>] do_wait+0x287/0x31e
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c011216e>] do_page_fault+0x225/0x565
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c011dc66>] sys_wait4+0x28/0x2c
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    CompressVersi S 0000001D 0 6832 6831 (NOTLB)
    c4a91b50 00000086 f3140a24 0000001d c029c65c 00000000 f7fbb030
    c18084e0
    c04ea4e0 c985e140 00000000 000012b7 55fdf817 000001cc f7fbb030
    c985e030
    c4a91b60 003d4a14 00000200 00000000 c02d0716 c4a91b60 003d4a14
    c04fdda8
    Call Trace:
    [<c029c65c>] tcp_rcv_established+0x5ab/0x5da
    [<c02d0716>] schedule_timeout+0x6f/0x8e
    [<c01234b2>] process_timeout+0x0/0x9
    [<c016a42f>] do_select+0x2fa/0x34b
    [<c016a020>] __pollwait+0x0/0x4c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c02cfee7>] schedule+0x60b/0x625
    [<c02cff42>] preempt_schedule+0x41/0x53
    [<c027277d>] dev_queue_xmit+0x230/0x237
    [<c028e934>] ip_output+0x1c6/0x1fc
    [<c028ed28>] ip_queue_xmit+0x3be/0x40c
    [<c0195275>] ext3_getblk+0xdf/0x22a
    [<c015bd02>] __find_get_block+0x45/0x4c
    [<c015bd26>] __getblk+0x1d/0x35
    [<c029d448>] tcp_cwnd_restart+0x1e/0xb4
    [<c01221bd>] lock_timer_base+0x19/0x33
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c0122266>] __mod_timer+0x8f/0x99
    [<c029e4aa>] tcp_init_tso_segs+0x2f/0x40
    [<c029ecef>] tcp_write_xmit+0x59/0x212
    [<c011ed36>] local_bh_disable+0xa/0xb
    [<c011ef1d>] local_bh_enable_ip+0x40/0x56
    [<c02d16eb>] _spin_unlock_bh+0x15/0x17
    [<c0295db0>] tcp_recvmsg+0x65c/0x693
    [<c0295db0>] tcp_recvmsg+0x65c/0x693
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a705>] core_sys_select+0x285/0x365
    [<c0269723>] sock_aio_read+0x56/0x64
    [<c01cab77>] _atomic_dec_and_lock+0x2b/0x4c
    [<c01598a2>] do_sync_read+0xbe/0xf1
    [<c0162191>] cp_new_stat64+0xef/0xff
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a882>] sys_select+0x9d/0x161
    [<c0159c3b>] sys_read+0x3b/0x63
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    bash S 07757686 0 6836 6830 6895 (NOTLB)
    cd503f30 00000086 00000000 07757686 00000172 00000002 e4502ab0
    c18084e0
    c04ea4e0 c20bb680 00000000 00003a4f 077c36df 00000172 e4502ab0
    c20bb570
    fffffe00 c20bb570 c20bb628 00000000 c011db3a 00000001 00000000
    c20bb570
    Call Trace:
    [<c011db3a>] do_wait+0x287/0x31e
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c01cf017>] copy_to_user+0x38/0x42
    [<c011dc66>] sys_wait4+0x28/0x2c
    [<c011dc7d>] sys_waitpid+0x13/0x1a
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    DConsole S 00000000 0 6895 6836 6896 (NOTLB)
    cf219ec4 00000086 e4509030 00000000 00000ae5 00000001 f7b0a570
    c18084e0
    c04ea4e0 ebd84140 00000000 000001d2 1007788c 000001df f7b0a570
    ebd84030
    e6f3bc00 e6f3bc00 cf219eec e6f3bc00 c016415d 00000000 ebd84030
    c012bc09
    Call Trace:
    [<c016415d>] pipe_wait+0x6d/0x8e
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c026d1c3>] __alloc_skb+0x49/0xf0
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c01645e0>] pipe_readv+0x209/0x275
    [<c0217da3>] e100_tx_clean+0xa4/0xcb
    [<c0164671>] pipe_read+0x25/0x29
    [<c015995c>] vfs_read+0x87/0x11d
    [<c0159c3b>] sys_read+0x3b/0x63
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    DConsole S D3ECBF68 0 6896 6836 6895 (NOTLB)
    d3ecbf2c 00000086 c012e032 d3ecbf68 c180905c c1809050 c1809050
    c18084e0
    c04ea4e0 d38fabc0 00000000 000005e2 211ec3cc 000001d4 c040e3e0
    d38faab0
    d3ecbf68 00000000 00000001 00000000 c02d0aec d3ecbf68 c804672a
    000001df
    Call Trace:
    [<c012e032>] enqueue_hrtimer+0x58/0x7f
    [<c02d0aec>] do_nanosleep+0x40/0x6d
    [<c012e3e1>] hrtimer_nanosleep+0x4c/0xe9
    [<c01621c1>] sys_stat64+0x20/0x27
    [<c012e366>] hrtimer_wakeup+0x0/0x1c
    [<c012e4c2>] sys_nanosleep+0x44/0x4f
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    in.telnetd S C041C998 0 6902 4054 6903 6829 (NOTLB)
    e5a49b58 00000082 c041c9a0 c041c998 00000000 00000001 e5a49b54
    c18084e0
    c04ea4e0 ebd84bc0 00000000 00000912 7f3133ce 000001de c040e3e0
    ebd84ab0
    7fffffff 00000000 00000010 00000000 c02d06ba 00000104 c02d16a0
    c01eb51f
    Call Trace:
    [<c02d06ba>] schedule_timeout+0x13/0x8e
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c01eb51f>] tty_poll+0x4a/0x54
    [<c016a42f>] do_select+0x2fa/0x34b
    [<c016a020>] __pollwait+0x0/0x4c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c0216e82>] e100_exec_cb+0xf6/0xfe
    [<c0217c97>] e100_xmit_frame+0x60/0xc8
    [<c0217ba7>] e100_xmit_prepare+0x0/0x90
    [<c027eef7>] __qdisc_run+0x34/0x1a3
    [<c011eec7>] local_bh_enable+0x7b/0x91
    [<c027277d>] dev_queue_xmit+0x230/0x237
    [<c028e934>] ip_output+0x1c6/0x1fc
    [<c028ed28>] ip_queue_xmit+0x3be/0x40c
    [<c0141d95>] get_page_from_freelist+0x70/0x88
    [<c01468fe>] __mod_zone_page_state+0x10/0x58
    [<c014695b>] mod_zone_page_state+0x15/0x1c
    [<c0156373>] kmem_getpages+0x67/0x8e
    [<c0156e83>] cache_grow+0xe2/0x135
    [<c029d448>] tcp_cwnd_restart+0x1e/0xb4
    [<c02a244a>] tcp_v4_send_check+0x83/0xc4
    [<c029e4aa>] tcp_init_tso_segs+0x2f/0x40
    [<c029ecef>] tcp_write_xmit+0x59/0x212
    [<c029eecb>] __tcp_push_pending_frames+0x23/0x80
    [<c011ed36>] local_bh_disable+0xa/0xb
    [<c011ef1d>] local_bh_enable_ip+0x40/0x56
    [<c02d16eb>] _spin_unlock_bh+0x15/0x17
    [<c029532b>] tcp_sendmsg+0x954/0xa02
    [<c029532b>] tcp_sendmsg+0x954/0xa02
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a705>] core_sys_select+0x285/0x365
    [<c02698ef>] sock_aio_write+0x56/0x64
    [<c02cfee7>] schedule+0x60b/0x625
    [<c0159ab0>] do_sync_write+0xbe/0xf1
    [<c01ee5b1>] read_chan+0x4b8/0x4c5
    [<c012bc09>] autoremove_wake_function+0x0/0x3a
    [<c011eb3a>] current_fs_time+0x44/0x4e
    [<c016a882>] sys_select+0x9d/0x161
    [<c0159c9e>] sys_write+0x3b/0x63
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    login S C04EA4E0 0 6903 6902 6908 (NOTLB)
    da65bf44 00000082 00000012 c04ea4e0 c115db6c 353b5065 00000000
    c18084e0
    c04ea4e0 f7e76680 00000000 00000656 8ba1b5c4 00000176 c040e3e0
    f7e76570
    fffffe00 f7e76570 f7e76628 00000000 c011db3a 00000001 00000000
    f7e76570
    Call Trace:
    [<c011db3a>] do_wait+0x287/0x31e
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c011216e>] do_page_fault+0x225/0x565
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c011dc66>] sys_wait4+0x28/0x2c
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    bash S 0000AC34 0 6908 6903 6954 (NOTLB)
    d497df30 00000082 00000000 0000ac34 df8fdf7f 00000000 e6672570
    c18084e0
    c04ea4e0 f43aa680 00000000 000041bb df9146d7 00000176 e6672570
    f43aa570
    fffffe00 f43aa570 f43aa628 00000000 c011db3a 00000001 00000000
    f43aa570
    Call Trace:
    [<c011db3a>] do_wait+0x287/0x31e
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c01cf017>] copy_to_user+0x38/0x42
    [<c011dc66>] sys_wait4+0x28/0x2c
    [<c011dc7d>] sys_waitpid+0x13/0x1a
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    top S D152F000 0 6954 6908 (NOTLB)
    c2257b50 00000082 c01e93d8 d152f000 c041c9a0 00000000 ebd84ab0
    c18084e0
    c04ea4e0 c2395bc0 00000000 000001a8 7f2cf5a8 000001de ebd84ab0
    c2395ab0
    c2257b60 0006aaca 00000002 00000000 c02d0716 c2257b60 0006aaca
    c04fd7e0
    Call Trace:
    [<c01e93d8>] tty_ldisc_try+0x33/0x38
    [<c02d0716>] schedule_timeout+0x6f/0x8e
    [<c01234b2>] process_timeout+0x0/0x9
    [<c016a42f>] do_select+0x2fa/0x34b
    [<c016a020>] __pollwait+0x0/0x4c
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c0216d84>] e100_exec_cmd+0x8a/0x92
    [<c0216e37>] e100_exec_cb+0xab/0xfe
    [<c0101555>] __switch_to+0x29/0x1cb
    [<c02cfee7>] schedule+0x60b/0x625
    [<c02d171e>] _read_unlock_irq+0xe/0x22
    [<c015bc2b>] bh_lru_install+0xa3/0xac
    [<c01a349e>] do_get_write_access+0x55a/0x57d
    [<c015bd26>] __getblk+0x1d/0x35
    [<c02d13a6>] _spin_lock+0xd/0x59
    [<c011356f>] task_rq_lock+0x2c/0x52
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c011401e>] try_to_wake_up+0x2ab/0x2b6
    [<c02d13a6>] _spin_lock+0xd/0x59
    [<c011356f>] task_rq_lock+0x2c/0x52
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c011401e>] try_to_wake_up+0x2ab/0x2b6
    [<c0115b5e>] __wake_up_common+0x2f/0x4b
    [<c0115b5e>] __wake_up_common+0x2f/0x4b
    [<c01edc7c>] n_tty_receive_buf+0x83d/0x899
    [<c01edc7c>] n_tty_receive_buf+0x83d/0x899
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a705>] core_sys_select+0x285/0x365
    [<c0115b1d>] default_wake_function+0x0/0x12
    [<c022c060>] ide_execute_command+0x99/0xae
    [<c02cf87f>] __down_failed+0x7/0xc
    [<c0115ba1>] __wake_up+0x27/0x3b
    [<c0115ba1>] __wake_up+0x27/0x3b
    [<c0115ba1>] __wake_up+0x27/0x3b
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c01e94e7>] tty_ldisc_deref+0x52/0x61
    [<c02d16a0>] _spin_unlock_irqrestore+0xf/0x23
    [<c01ebfca>] tty_ioctl+0x42b/0x43b
    [<c01cf05b>] copy_from_user+0x3a/0x5f
    [<c016a882>] sys_select+0x9d/0x161
    [<c016944e>] sys_fcntl64+0x55/0x68
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    IntegrityChec D 14CB896D 0 7275 7274 (NOTLB)
    ca3f9e20 00000086 d64c1680 14cb896d 000001df 00000002 e4502ab0
    c18084e0
    c04ea4e0 d8d9abc0 00000000 00000501 14cbd373 000001df e4502ab0
    d8d9aab0
    c18084e0 00000000 ca3f9e8c ca3f9e2c c02d065a ca3f9e84 c1803400
    c013d8c9
    Call Trace:
    [<c02d065a>] io_schedule+0x26/0x30
    [<c013d8c9>] sync_page+0x3b/0x3e
    [<c02d085f>] __wait_on_bit_lock+0x31/0x58
    [<c013d88e>] sync_page+0x0/0x3e
    [<c013df25>] __lock_page+0x6a/0x71
    [<c012bc43>] wake_bit_function+0x0/0x34
    [<c012bc43>] wake_bit_function+0x0/0x34
    [<c013ee5b>] filemap_nopage+0x263/0x346
    [<c014a83c>] do_no_page+0x86/0x2a7
    [<c026d1c3>] __alloc_skb+0x49/0xf0
    [<c014acb0>] __handle_mm_fault+0x19e/0x28c
    [<c011216e>] do_page_fault+0x225/0x565
    [<c0111f49>] do_page_fault+0x0/0x565
    [<c0103419>] error_code+0x39/0x40
    xdelta D 00000000 0 7341 5384 (NOTLB)
    d5cb3bc8 00000086 00000000 00000000 00000000 c6574f40 00000000
    c18084e0
    c04ea4e0 f5cf0bc0 00000000 0000150b db597c17 000001de c040e3e0
    f5cf0ab0
    c18084e0 00000000 c1800740 d5cb3bd4 c02d065a d5cb3c2c c015a9f7
    c015aa2f
    Call Trace:
    [<c02d065a>] io_schedule+0x26/0x30
    [<c015a9f7>] sync_buffer+0x0/0x3b
    [<c015aa2f>] sync_buffer+0x38/0x3b
    [<c02d0792>] __wait_on_bit+0x2b/0x51
    [<c02d0826>] out_of_line_wait_on_bit+0x6e/0x76
    [<c015a9f7>] sync_buffer+0x0/0x3b
    [<c012bc43>] wake_bit_function+0x0/0x34
    [<c015d944>] submit_bh+0x139/0x15f
    [<c012bc43>] wake_bit_function+0x0/0x34
    [<c015bb72>] __bread_slow+0x50/0x66
    [<c015bda8>] __bread+0x2a/0x30
    [<c019478f>] ext3_get_branch+0x5f/0xbc
    [<c0194e32>] ext3_get_blocks_handle+0xac/0x362
    [<c0195180>] ext3_get_block+0x98/0xae
    [<c0178384>] do_mpage_readpage+0x23e/0x586
    [<c01cc424>] radix_tree_node_alloc+0x1c/0x54
    [<c0146974>] __inc_zone_state+0x12/0x72
    [<c017877d>] mpage_readpages+0xb1/0x123
    [<c01950e8>] ext3_get_block+0x0/0xae
    [<c011401e>] try_to_wake_up+0x2ab/0x2b6
    [<c02cfee7>] schedule+0x60b/0x625
    [<c02d1637>] _spin_unlock+0xd/0x21
    [<c01417dd>] rmqueue_bulk+0x52/0x59
    [<c0195d57>] ext3_readpages+0x16/0x1a
    [<c01950e8>] ext3_get_block+0x0/0xae
    [<c014368f>] read_pages+0x26/0xc0
    [<c0141d95>] get_page_from_freelist+0x70/0x88
    [<c0141df3>] __alloc_pages+0x46/0x26b
    [<c014383f>] __do_page_cache_readahead+0x116/0x136
    [<c01438a8>] force_page_cache_readahead+0x49/0x5f
    [<c01410df>] sys_fadvise64_64+0x157/0x1d9
    [<c014117e>] sys_fadvise64+0x1d/0x23
    [<c01028e5>] sysenter_past_esp+0x56/0x79
    pdflush S D0E8BF48 0 7363 7 7365 3798 (L-TLB)
    d0e8bf88 00000046 00000000 d0e8bf48 d0e8bf48 00000000 f7fe4ab0
    c18084e0
    c04ea4e0 e4d15680 00000000 000000c8 f1f5f0dd 000001dc f7fe4ab0
    e4d15570
    d0e8bfb4 d0e8bfa8 c014345a fffffffc c0143374 cc23bf34 00000000
    c014348d
    Call Trace:
    [<c014345a>] pdflush+0x0/0x37
    [<c0143374>] __pdflush+0x88/0x16e
    [<c014348d>] pdflush+0x33/0x37
    [<c014345a>] pdflush+0x0/0x37
    [<c012b889>] kthread+0x85/0xaa
    [<c012b804>] kthread+0x0/0xaa
    [<c0100e9d>] kernel_thread_helper+0x5/0xb
    pdflush S DE591F48 0 7365 7 7376 7363 (L-TLB)
    de591f88 00000046 00000000 de591f48 00000000 00000000 f7fe4ab0
    c18084e0
    c04ea4e0 c5fde140 00000000 000000b3 f1ec1bed 000001dc f7fe4ab0
    c5fde030
    de591fb4 de591fa8 c014345a fffffffc c0143374 cc23bf34 00000000
    c014348d
    Call Trace:
    [<c014345a>] pdflush+0x0/0x37
    [<c0143374>] __pdflush+0x88/0x16e
    [<c014348d>] pdflush+0x33/0x37
    [<c014345a>] pdflush+0x0/0x37
    [<c012b889>] kthread+0x85/0xaa
    [<c012b804>] kthread+0x0/0xaa
    [<c0100e9d>] kernel_thread_helper+0x5/0xb
    pdflush S CA7EFF48 0 7376 7 7398 7365 (L-TLB)
    ca7eff88 00000046 00000000 ca7eff48 00000000 00000000 f7fe4ab0
    c18084e0
    c04ea4e0 c2395140 00000000 0000007e f1f63258 000001dc f7fe4ab0
    c2395030
    ca7effb4 ca7effa8 c014345a fffffffc c0143374 cc23bf34 00000000
    c014348d
    Call Trace:
    [<c014345a>] pdflush+0x0/0x37
    [<c0143374>] __pdflush+0x88/0x16e
    [<c014348d>] pdflush+0x33/0x37
    [<c014345a>] pdflush+0x0/0x37
    [<c012b889>] kthread+0x85/0xaa
    [<c012b804>] kthread+0x0/0xaa
    [<c0100e9d>] kernel_thread_helper+0x5/0xb
    pdflush S F1EC3A74 0 7398 7 7399 7376 (L-TLB)
    d1601f88 00000046 00000000 f1ec3a74 000001dc 00000003 e6672570
    c18084e0
    c04ea4e0 c657b680 00000000 00000160 f1ec3a74 000001dc e6672570
    c657b570
    d1601fb4 d1601fa8 c014345a fffffffc c0143374 de591f34 00000000
    c014348d
    Call Trace:
    [<c014345a>] pdflush+0x0/0x37
    [<c0143374>] __pdflush+0x88/0x16e
    [<c014348d>] pdflush+0x33/0x37
    [<c014345a>] pdflush+0x0/0x37
    [<c012b889>] kthread+0x85/0xaa
    [<c012b804>] kthread+0x0/0xaa
    [<c0100e9d>] kernel_thread_helper+0x5/0xb
    pdflush S CA7EDF98 0 7399 7 7398 (L-TLB)
    ca7edf88 00000046 00000000 ca7edf98 ca7edf98 00000000 d38fa570
    c18084e0
    c04ea4e0 c2395680 00000000 000002c3 f1f63d2d 000001dc d38fa570
    c2395570
    ca7edfb4 ca7edfa8 c014345a fffffffc c0143374 d0e8bf34 00000000
    c014348d
    Call Trace:
    [<c014345a>] pdflush+0x0/0x37
    [<c0143374>] __pdflush+0x88/0x16e
    [<c014348d>] pdflush+0x33/0x37
    [<c014345a>] pdflush+0x0/0x37
    [<c012b889>] kthread+0x85/0xaa
    [<c012b804>] kthread+0x0/0xaa
    [<c0100e9d>] kernel_thread_helper+0x5/0xb


    -----Original Message-----
    From: Andrew Morton [mailto:akpm@osdl.org]
    Sent: Friday, January 26, 2007 6:17 AM
    To: Matthew Kirk
    Cc: linux-kernel@vger.kernel.org
    Subject: Re: fsync occasionally very slow

    On Fri, 26 Jan 2007 05:37:38 -0500
    "Matthew Kirk" <mkirk01@rcn.com> wrote:

    > I am working on an application that fsync(2) files and directories as
    > needed. I’m seeing intermittent stalls on fsync that can last many
    > seconds.

    Please generate an all-task backtrace during the stall via:

    dmesg -n 8
    <wait for a stall>
    echo t > /prov/sysrq-trigger
    dmesg -s 1000000 > foo

    and send foo.

    --
    No virus found in this incoming message.
    Checked by AVG Free Edition.
    Version: 7.1.410 / Virus Database: 268.17.11/652 - Release Date: 1/25/2007


    --
    No virus found in this outgoing message.
    Checked by AVG Free Edition.
    Version: 7.1.410 / Virus Database: 268.17.14/658 - Release Date: 1/29/2007

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2007-01-29 23:05    [W:0.080 / U:94.196 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site