lkml.org 
[lkml]   [2007]   [Aug]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: high system cpu load during intense disk i/o
    Andrew Morton wrote:
    > On Mon, 06 Aug 2007 16:20:30 +0200 Dimitrios Apostolou <jimis@gmx.net> wrote:
    >
    >> Andrew Morton wrote:
    >>> On Sun, 5 Aug 2007 19:03:12 +0300 Dimitrios Apostolou <jimis@gmx.net> wrote:
    >>>
    >>>> was my report so complicated?
    >>> We're bad.
    >>>
    >>> Seems that your context switch rate when running two instances of
    >>> badblocks against two different disks went batshit insane. It doesn't
    >>> happen here.
    >>>
    >>> Please capture the `vmstat 1' output while running the problematic
    >>> workload.
    >>>
    >>> The oom-killing could have been unrelated to the CPU load problem. iirc
    >>> badblocks uses a lot of memory, so it might have been genuine. Keep an eye
    >>> on the /proc/meminfo output and send the kernel dmesg output from the
    >>> oom-killing event.
    >> Please see the attached files. Unfortunately I don't see any useful info
    >> in them:
    >> *_before: before running any badblocks process
    >> *_while: while running badblocks process, but without any cron job
    >> having kicked in
    >> *_bad: 5 minutes later that some cron jobs kicked in
    >>
    >> About the OOM killer, indeed I believe that it is unrelated. It started
    >> killing after about 2 days, that hundreds of processes were stuck as
    >> running and taking up memory, so I suppose the 256 MB RAM were truly
    >> filled. I just mentioned it because its behaviour is completely
    >> non-helpful. It doesn't touch the badblocks process, it rarely touches
    >> the stuck as running cron jobs, but it kills other irrelevant processes.
    >> If you still want the killing logs, tell me and I'll search for them.
    >
    > ah. Your context-switch rate during the dual-badblocks run is not high at
    > all.
    >
    > I suspect I was fooled by the oprofile output, which showed tremendous
    > amounts of load in schedule() and switch_to(). The percentages which
    > opreport shows are the percentage of non-halted CPU time. So if you have a
    > function in the kernel which is using 1% of the total CPU, and the CPU is
    > halted for 95% of the time, it appears that the function is taking 20% of
    > CPU!
    >
    > The fix for that is to boot with the "idle=poll" boot parameter, to make
    > the CPU spin when it has nothing else to do.

    I'm attaching the new oprofile output. I can't see any difference
    however. :-s


    Dimitris


    Mon Aug 6 21:46:59 EEST 2007
    + opcontrol --vmlinux=/usr/src/linux-2.6.22-ARCH/vmlinux
    + opcontrol --start
    Using default event: CPU_CLK_UNHALTED:100000:0:1:1
    Daemon started.
    Profiler running.
    + sleep 5
    + opcontrol --shutdown
    Stopping profiling.
    Killing daemon.
    + echo

    + echo

    + echo

    + opreport
    CPU: PIII, speed 798.025 MHz (estimated)
    Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
    CPU_CLK_UNHALT...|
    samples| %|
    ------------------
    7917 57.2658 vmlinux
    2204 15.9421 libc-2.6.so
    958 6.9295 libpython2.5.so.1.0
    815 5.8951 ide_core
    773 5.5913 perl
    242 1.7505 ld-2.6.so
    216 1.5624 mpop
    204 1.4756 bash
    121 0.8752 libgnutls.so.13.3.0
    64 0.4629 badblocks
    CPU_CLK_UNHALT...|
    samples| %|
    ------------------
    48 75.0000 badblocks
    10 15.6250 anon (tgid:4338 range:0xb7f6c000-0xb7f6d000)
    6 9.3750 anon (tgid:4339 range:0xb7f64000-0xb7f65000)
    46 0.3327 ISO8859-1.so
    37 0.2676 ext3
    34 0.2459 imap-login
    CPU_CLK_UNHALT...|
    samples| %|
    ------------------
    31 91.1765 imap-login
    1 2.9412 anon (tgid:3989 range:0xb7f80000-0xb7f81000)
    1 2.9412 anon (tgid:3990 range:0xb7f72000-0xb7f73000)
    1 2.9412 anon (tgid:4332 range:0xb7fd6000-0xb7fd7000)
    33 0.2387 libext2fs.so.2.4
    24 0.1736 jbd
    24 0.1736 libpthread-2.6.so
    21 0.1519 gawk
    15 0.1085 oprofile
    13 0.0940 libcrypto.so.0.9.8
    12 0.0868 ide_disk
    11 0.0796 skge
    7 0.0506 sshd
    6 0.0434 dovecot-auth
    CPU_CLK_UNHALT...|
    samples| %|
    ------------------
    5 83.3333 dovecot-auth
    1 16.6667 anon (tgid:3971 range:0xb7f8c000-0xb7f8d000)
    4 0.0289 libnetsnmp.so.15.0.0
    4 0.0289 libnetsnmpmibs.so.15.0.0
    4 0.0289 imap
    4 0.0289 dovecot
    3 0.0217 grep
    3 0.0217 reiserfs
    3 0.0217 locale-archive
    1 0.0072 libdl-2.6.so
    1 0.0072 init
    1 0.0072 screen-4.0.3
    + echo

    + echo

    + echo

    + opreport -l /usr/src/linux-2.6.22-ARCH/vmlinux
    CPU: PIII, speed 798.025 MHz (estimated)
    Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
    samples % symbol name
    2477 31.2871 __switch_to
    1625 20.5255 schedule
    246 3.1072 mask_and_ack_8259A
    222 2.8041 __blockdev_direct_IO
    147 1.8568 follow_page
    107 1.3515 put_page
    106 1.3389 do_wp_page
    91 1.1494 native_load_tls
    88 1.1115 __bio_add_page
    80 1.0105 delay_tsc
    63 0.7958 __handle_mm_fault
    50 0.6316 submit_page_section
    49 0.6189 get_page_from_freelist
    48 0.6063 dio_bio_add_page
    45 0.5684 dequeue_task
    41 0.5179 get_user_pages
    40 0.5052 bio_alloc_bioset
    39 0.4926 enable_8259A_irq
    38 0.4800 dio_send_cur_page
    38 0.4800 page_address
    38 0.4800 sysenter_past_esp
    37 0.4673 dio_get_page
    37 0.4673 find_get_page
    36 0.4547 bio_add_page
    36 0.4547 kmem_cache_alloc
    36 0.4547 unmap_vmas
    35 0.4421 kmem_cache_free
    34 0.4295 blk_rq_map_sg
    33 0.4168 blk_recount_segments
    32 0.4042 do_page_fault
    31 0.3916 generic_file_direct_write
    30 0.3789 __generic_file_aio_write_nolock
    30 0.3789 vm_normal_page
    28 0.3537 __link_path_walk
    28 0.3537 __mutex_lock_slowpath
    26 0.3284 irq_entries_start
    25 0.3158 filemap_nopage
    25 0.3158 vfs_write
    23 0.2905 find_vma
    22 0.2779 __d_lookup
    22 0.2779 dio_bio_complete
    22 0.2779 mark_page_accessed
    21 0.2653 add_timer_randomness
    21 0.2653 max_block
    21 0.2653 page_fault
    19 0.2400 restore_all
    18 0.2274 __add_entropy_words
    18 0.2274 preempt_schedule
    17 0.2147 __copy_to_user_ll
    17 0.2147 __generic_unplug_device
    17 0.2147 do_sys_poll
    17 0.2147 fget_light
    17 0.2147 math_state_restore
    16 0.2021 __mod_timer
    16 0.2021 generic_file_direct_IO
    16 0.2021 preempt_schedule_irq
    15 0.1895 block_llseek
    15 0.1895 do_sync_write
    14 0.1768 current_fs_time
    14 0.1768 do_generic_mapping_read
    14 0.1768 mempool_free
    14 0.1768 native_flush_tlb_single
    14 0.1768 radix_tree_lookup
    14 0.1768 rw_verify_area
    13 0.1642 __alloc_pages
    13 0.1642 __make_request
    13 0.1642 _spin_lock_irqsave
    13 0.1642 cond_resched
    12 0.1516 cfq_completed_request
    12 0.1516 strnlen_user
    11 0.1389 __mutex_unlock_slowpath
    11 0.1389 do_lookup
    11 0.1389 generic_make_request
    11 0.1389 get_request
    11 0.1389 restore_nocheck
    10 0.1263 blkdev_direct_IO
    10 0.1263 load_elf_binary
    10 0.1263 unix_poll
    9 0.1137 cfq_insert_request
    9 0.1137 cfq_set_request
    9 0.1137 copy_page_range
    9 0.1137 do_mmap_pgoff
    9 0.1137 elv_insert
    9 0.1137 generic_file_aio_write_nolock
    9 0.1137 generic_permission
    9 0.1137 recalc_task_prio
    8 0.1010 blkdev_get_blocks
    8 0.1010 cache_reap
    8 0.1010 dio_new_bio
    8 0.1010 file_update_time
    8 0.1010 sys_llseek
    8 0.1010 vsnprintf
    7 0.0884 bio_put
    7 0.0884 blk_backing_dev_unplug
    7 0.0884 cfq_dispatch_requests
    7 0.0884 copy_process
    7 0.0884 device_not_available
    7 0.0884 disk_round_stats
    7 0.0884 elv_queue_empty
    7 0.0884 free_block
    7 0.0884 generic_unplug_device
    7 0.0884 hrtimer_interrupt
    7 0.0884 kmap_atomic_prot
    7 0.0884 mutex_remove_waiter
    7 0.0884 need_resched
    7 0.0884 page_remove_rmap
    7 0.0884 permission
    6 0.0758 __blk_put_request
    6 0.0758 __copy_from_user_ll
    6 0.0758 __copy_user_intel
    6 0.0758 __end_that_request_first
    6 0.0758 __find_get_block_slow
    6 0.0758 bio_init
    6 0.0758 blk_remove_plug
    6 0.0758 cfq_may_queue
    6 0.0758 dio_cleanup
    6 0.0758 dio_complete
    6 0.0758 dput
    6 0.0758 getname
    6 0.0758 inotify_inode_queue_event
    6 0.0758 kfree
    6 0.0758 memcpy
    6 0.0758 mempool_alloc
    6 0.0758 path_walk
    6 0.0758 rb_erase
    6 0.0758 read_tsc
    6 0.0758 task_rq_lock
    6 0.0758 try_to_wake_up
    6 0.0758 vfs_llseek
    5 0.0632 __const_udelay
    5 0.0632 __fput
    5 0.0632 blk_plug_device
    5 0.0632 cfq_remove_request
    5 0.0632 copy_strings
    5 0.0632 current_io_context
    5 0.0632 debug_mutex_add_waiter
    5 0.0632 dnotify_parent
    5 0.0632 do_IRQ
    5 0.0632 do_filp_open
    5 0.0632 do_path_lookup
    5 0.0632 drain_array
    5 0.0632 elv_rqhash_add
    5 0.0632 free_poll_entry
    5 0.0632 generic_segment_checks
    5 0.0632 handle_level_irq
    5 0.0632 idle_cpu
    5 0.0632 init_request_from_bio
    5 0.0632 io_schedule
    5 0.0632 ip_append_data
    5 0.0632 note_interrupt
    5 0.0632 proc_sys_lookup_table_one
    5 0.0632 sched_clock
    5 0.0632 sys_write
    5 0.0632 up_read
    4 0.0505 __dentry_open
    4 0.0505 __freed_request
    4 0.0505 __pagevec_lru_add_active
    4 0.0505 _atomic_dec_and_lock
    4 0.0505 bdev_read_only
    4 0.0505 cfq_queue_empty
    4 0.0505 copy_to_user
    4 0.0505 debug_mutex_free_waiter
    4 0.0505 do_munmap
    4 0.0505 do_sync_read
    4 0.0505 do_wait
    4 0.0505 dummy_inode_permission
    4 0.0505 elv_completed_request
    4 0.0505 end_that_request_last
    4 0.0505 filemap_write_and_wait
    4 0.0505 find_extend_vma
    4 0.0505 find_vma_prev
    4 0.0505 generic_file_aio_read
    4 0.0505 get_empty_filp
    4 0.0505 get_request_wait
    4 0.0505 hweight32
    4 0.0505 internal_add_timer
    4 0.0505 irq_exit
    4 0.0505 native_read_tsc
    4 0.0505 number
    4 0.0505 poll_freewait
    4 0.0505 proc_lookup
    4 0.0505 setup_arg_pages
    4 0.0505 should_remove_suid
    4 0.0505 sock_poll
    4 0.0505 submit_bio
    4 0.0505 tick_sched_timer
    4 0.0505 unmap_region
    4 0.0505 vma_adjust
    3 0.0379 __atomic_notifier_call_chain
    3 0.0379 __do_page_cache_readahead
    3 0.0379 __elv_add_request
    3 0.0379 __find_get_block
    3 0.0379 __mark_inode_dirty
    3 0.0379 __pte_alloc
    3 0.0379 anon_vma_prepare
    3 0.0379 anon_vma_unlink
    3 0.0379 bio_endio
    3 0.0379 bit_waitqueue
    3 0.0379 call_rcu
    3 0.0379 cfq_init_prio_data
    3 0.0379 cfq_put_queue
    3 0.0379 cfq_service_tree_add
    3 0.0379 clear_bdi_congested
    3 0.0379 common_interrupt
    3 0.0379 credit_entropy_store
    3 0.0379 deactivate_task
    3 0.0379 debug_mutex_lock_common
    3 0.0379 debug_mutex_unlock
    3 0.0379 del_timer
    3 0.0379 down_read_trylock
    3 0.0379 dummy_inode_getattr
    3 0.0379 elv_dispatch_sort
    3 0.0379 elv_may_queue
    3 0.0379 elv_put_request
    3 0.0379 elv_set_request
    3 0.0379 enqueue_task
    3 0.0379 error_code
    3 0.0379 flush_old_exec
    3 0.0379 free_hot_cold_page
    3 0.0379 generic_fillattr
    3 0.0379 kmem_cache_zalloc
    3 0.0379 link_path_walk
    3 0.0379 lock_timer_base
    3 0.0379 locks_remove_flock
    3 0.0379 may_expand_vm
    3 0.0379 native_load_esp0
    3 0.0379 open_namei
    3 0.0379 path_lookup_open
    3 0.0379 pipe_poll
    3 0.0379 proc_flush_task
    3 0.0379 release_vm86_irqs
    3 0.0379 remove_vma
    3 0.0379 rq_init
    3 0.0379 sock_alloc_send_skb
    3 0.0379 strncpy_from_user
    3 0.0379 sys_mmap2
    3 0.0379 sys_mprotect
    3 0.0379 touch_atime
    3 0.0379 vfs_read
    3 0.0379 vm_acct_memory
    3 0.0379 vm_stat_account
    3 0.0379 vma_link
    2 0.0253 I_BDEV
    2 0.0253 __dec_zone_state
    2 0.0253 __group_complete_signal
    2 0.0253 __inc_zone_state
    2 0.0253 __mutex_lock_interruptible_slowpath
    2 0.0253 __path_lookup_intent_open
    2 0.0253 __pollwait
    2 0.0253 __rb_rotate_right
    2 0.0253 __vm_enough_memory
    2 0.0253 __vma_link_rb
    2 0.0253 __wake_up
    2 0.0253 __wake_up_bit
    2 0.0253 alloc_inode
    2 0.0253 anon_vma_ctor
    2 0.0253 bio_alloc
    2 0.0253 bio_free
    2 0.0253 bio_fs_destructor
    2 0.0253 bio_get_nr_vecs
    2 0.0253 bio_phys_segments
    2 0.0253 blk_queue_bounce
    2 0.0253 cache_alloc_refill
    2 0.0253 check_pgt_cache
    2 0.0253 debug_mutex_set_owner
    2 0.0253 dio_bio_end_io
    2 0.0253 dio_zero_block
    2 0.0253 do_notify_parent
    2 0.0253 do_notify_resume
    2 0.0253 do_sys_open
    2 0.0253 down_read
    2 0.0253 dst_alloc
    2 0.0253 effective_prio
    2 0.0253 elv_dequeue_request
    2 0.0253 elv_next_request
    2 0.0253 file_free_rcu
    2 0.0253 file_read_actor
    2 0.0253 filp_close
    2 0.0253 find_next_bit
    2 0.0253 find_next_zero_bit
    2 0.0253 find_vma_prepare
    2 0.0253 flush_tlb_mm
    2 0.0253 flush_tlb_page
    2 0.0253 generic_file_mmap
    2 0.0253 get_index
    2 0.0253 get_unused_fd
    2 0.0253 getnstimeofday
    2 0.0253 handle_IRQ_event
    2 0.0253 icmp_push_reply
    2 0.0253 icmp_send
    2 0.0253 kill_fasync
    2 0.0253 kunmap_atomic
    2 0.0253 lru_cache_add_active
    2 0.0253 mntput_no_expire
    2 0.0253 native_read_cr0
    2 0.0253 notifier_call_chain
    2 0.0253 ordered_bio_endio
    2 0.0253 page_cache_readahead
    2 0.0253 percpu_counter_mod
    2 0.0253 poll_initwait
    2 0.0253 profile_munmap
    2 0.0253 put_filp
    2 0.0253 put_io_context
    2 0.0253 quicklist_trim
    2 0.0253 radix_tree_gang_lookup_tag
    2 0.0253 rb_insert_color
    2 0.0253 remove_suid
    2 0.0253 resume_userspace
    2 0.0253 sched_balance_self
    2 0.0253 seq_printf
    2 0.0253 smp_apic_timer_interrupt
    2 0.0253 sys_brk
    2 0.0253 sys_fstat64
    2 0.0253 sys_read
    2 0.0253 sys_rt_sigaction
    2 0.0253 sys_stat64
    2 0.0253 system_call
    2 0.0253 vma_merge
    2 0.0253 work_notifysig
    2 0.0253 zone_watermark_ok
    1 0.0126 __activate_task
    1 0.0126 __alloc_skb
    1 0.0126 __blocking_notifier_call_chain
    1 0.0126 __bread
    1 0.0126 __delay
    1 0.0126 __do_softirq
    1 0.0126 __follow_mount
    1 0.0126 __iget
    1 0.0126 __ip_select_ident
    1 0.0126 __put_user_4
    1 0.0126 __rb_rotate_left
    1 0.0126 __rcu_process_callbacks
    1 0.0126 __rmqueue
    1 0.0126 __sigqueue_alloc
    1 0.0126 __sigqueue_free
    1 0.0126 __sk_dst_check
    1 0.0126 __tasklet_schedule
    1 0.0126 __tcp_ack_snd_check
    1 0.0126 __tcp_push_pending_frames
    1 0.0126 __user_walk_fd
    1 0.0126 __vma_link
    1 0.0126 _read_lock_irqsave
    1 0.0126 account_system_time
    1 0.0126 account_user_time
    1 0.0126 add_disk_randomness
    1 0.0126 add_wait_queue
    1 0.0126 all_vm_events
    1 0.0126 alloc_buffer_head
    1 0.0126 alloc_pid
    1 0.0126 anon_pipe_buf_release
    1 0.0126 anon_vma_link
    1 0.0126 arch_setup_additional_pages
    1 0.0126 bio_hw_segments
    1 0.0126 blk_do_ordered
    1 0.0126 blk_start_queueing
    1 0.0126 blockable_page_cache_readahead
    1 0.0126 blocking_notifier_call_chain
    1 0.0126 bmap
    1 0.0126 can_vma_merge_after
    1 0.0126 cfq_add_rq_rb
    1 0.0126 cfq_choose_req
    1 0.0126 cfq_cic_rb_lookup
    1 0.0126 cfq_rb_erase
    1 0.0126 check_userspace
    1 0.0126 copy_semundo
    1 0.0126 core_sys_select
    1 0.0126 cp_new_stat64
    1 0.0126 csum_partial_copy_generic
    1 0.0126 d_alloc
    1 0.0126 dec_zone_page_state
    1 0.0126 dequeue_signal
    1 0.0126 devinet_ioctl
    1 0.0126 do_fork
    1 0.0126 do_group_exit
    1 0.0126 do_sigaction
    1 0.0126 drive_stat_acct
    1 0.0126 dummy_capable
    1 0.0126 dummy_file_alloc_security
    1 0.0126 dummy_task_kill
    1 0.0126 dummy_vm_enough_memory
    1 0.0126 dup_fd
    1 0.0126 elv_rb_add
    1 0.0126 end_that_request_first
    1 0.0126 exit_itimers
    1 0.0126 fasync_helper
    1 0.0126 fget
    1 0.0126 file_kill
    1 0.0126 file_move
    1 0.0126 file_ra_state_init
    1 0.0126 flush_sigqueue
    1 0.0126 fn_hash_lookup
    1 0.0126 fput
    1 0.0126 free_page_and_swap_cache
    1 0.0126 free_pgtables
    1 0.0126 get_futex_key
    1 0.0126 get_io_context
    1 0.0126 get_task_mm
    1 0.0126 hrtimer_try_to_cancel
    1 0.0126 icmp_glue_bits
    1 0.0126 ifind_fast
    1 0.0126 inotify_dentry_parent_queue_event
    1 0.0126 invalidate_inode_buffers
    1 0.0126 ip_route_input
    1 0.0126 irq_enter
    1 0.0126 local_bh_enable_ip
    1 0.0126 may_open
    1 0.0126 mempool_alloc_slab
    1 0.0126 mempool_free_slab
    1 0.0126 n_tty_receive_buf
    1 0.0126 native_apic_write
    1 0.0126 native_set_pte_at
    1 0.0126 open_exec
    1 0.0126 opost
    1 0.0126 page_add_file_rmap
    1 0.0126 pipe_release
    1 0.0126 pipe_write
    1 0.0126 prio_tree_insert
    1 0.0126 prio_tree_remove
    1 0.0126 proc_sys_lookup_table
    1 0.0126 profile_tick
    1 0.0126 pty_chars_in_buffer
    1 0.0126 put_tty_queue
    1 0.0126 rb_first
    1 0.0126 recalc_sigpending_tsk
    1 0.0126 release_open_intent
    1 0.0126 release_task
    1 0.0126 restore_sigcontext
    1 0.0126 ret_from_exception
    1 0.0126 ret_from_intr
    1 0.0126 scheduler_tick
    1 0.0126 search_binary_handler
    1 0.0126 secure_ip_id
    1 0.0126 show_stat
    1 0.0126 sigprocmask
    1 0.0126 snprintf
    1 0.0126 sock_fasync
    1 0.0126 softlockup_tick
    1 0.0126 special_mapping_nopage
    1 0.0126 split_vma
    1 0.0126 sys_access
    1 0.0126 sys_clone
    1 0.0126 sys_execve
    1 0.0126 sys_futex
    1 0.0126 sys_open
    1 0.0126 sys_rt_sigprocmask
    1 0.0126 syscall_exit
    1 0.0126 sysctl_head_next
    1 0.0126 task_running_tick
    1 0.0126 tcp_poll
    1 0.0126 tcp_rcv_space_adjust
    1 0.0126 tcp_recvmsg
    1 0.0126 tcp_transmit_skb
    1 0.0126 test_set_page_writeback
    1 0.0126 tick_do_update_jiffies64
    1 0.0126 tick_program_event
    1 0.0126 timespec_trunc
    1 0.0126 try_to_del_timer_sync
    1 0.0126 tty_ioctl
    1 0.0126 tty_ldisc_ref_wait
    1 0.0126 udp_flush_pending_frames
    1 0.0126 unuse_table
    1 0.0126 update_process_times
    1 0.0126 update_wall_time
    1 0.0126 vfs_stat_fd
    1 0.0126 vma_prio_tree_remove
    1 0.0126 wake_up_inode
    1 0.0126 wake_up_new_task
    1 0.0126 wake_up_process
    1 0.0126 write_cache_pages
    Mon Aug 6 21:38:04 EEST 2007
    Using default event: CPU_CLK_UNHALTED:100000:0:1:1
    Daemon started.
    Profiler running.
    Stopping profiling.
    Killing daemon.



    CPU: PIII, speed 798.025 MHz (estimated)
    Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
    CPU_CLK_UNHALT...|
    samples| %|
    ------------------
    40131 97.6495 vmlinux
    318 0.7738 libc-2.6.so
    168 0.4088 oprofiled
    CPU_CLK_UNHALT...|
    samples| %|
    ------------------
    167 99.4048 oprofiled
    1 0.5952 anon (tgid:4159 range:0xb7f55000-0xb7f56000)
    162 0.3942 bash
    133 0.3236 ld-2.6.so
    51 0.1241 oprofile
    49 0.1192 ISO8859-1.so
    33 0.0803 ext3
    20 0.0487 jbd
    5 0.0122 locale-archive
    4 0.0097 gawk
    4 0.0097 imap-login
    3 0.0073 grep
    3 0.0073 reiserfs
    2 0.0049 cat
    2 0.0049 libcrypto.so.0.9.8
    2 0.0049 dovecot-auth
    1 0.0024 ide_core
    1 0.0024 ide_disk
    1 0.0024 libncurses.so.5.6
    1 0.0024 which
    1 0.0024 libnetsnmp.so.15.0.0
    1 0.0024 libnetsnmpmibs.so.15.0.0
    1 0.0024 dovecot
    CPU_CLK_UNHALT...|
    samples| %|
    ------------------
    1 100.000 anon (tgid:3968 range:0xb7fa7000-0xb7fa8000)



    CPU: PIII, speed 798.025 MHz (estimated)
    Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
    samples % symbol name
    39019 97.2291 cpu_idle
    263 0.6554 poll_idle
    131 0.3264 quicklist_trim
    92 0.2292 do_wp_page
    50 0.1246 check_pgt_cache
    44 0.1096 __handle_mm_fault
    37 0.0922 get_page_from_freelist
    31 0.0772 page_fault
    21 0.0523 unmap_vmas
    15 0.0374 __copy_to_user_ll
    13 0.0324 filemap_nopage
    12 0.0299 radix_tree_lookup
    11 0.0274 copy_process
    10 0.0249 __link_path_walk
    10 0.0249 sysenter_past_esp
    9 0.0224 do_page_fault
    9 0.0224 find_get_page
    8 0.0199 kmem_cache_free
    6 0.0150 __d_lookup
    6 0.0150 __find_get_block
    6 0.0150 page_address
    5 0.0125 _atomic_dec_and_lock
    5 0.0125 do_mmap_pgoff
    5 0.0125 free_hot_cold_page
    5 0.0125 kmem_cache_alloc
    5 0.0125 load_elf_binary
    5 0.0125 strnlen_user
    4 0.0100 __wake_up_bit
    4 0.0100 copy_page_range
    4 0.0100 d_alloc
    4 0.0100 error_code
    4 0.0100 find_vma
    4 0.0100 flush_tlb_mm
    4 0.0100 inode_init_once
    4 0.0100 kmap_atomic_prot
    4 0.0100 put_page
    4 0.0100 restore_nocheck
    4 0.0100 schedule
    4 0.0100 vm_normal_page
    3 0.0075 __follow_mount
    3 0.0075 __mutex_lock_slowpath
    3 0.0075 __rmqueue
    3 0.0075 anon_vma_unlink
    3 0.0075 ktime_get_ts
    3 0.0075 mark_page_accessed
    3 0.0075 memcpy
    3 0.0075 prio_tree_insert
    3 0.0075 strncpy_from_user
    3 0.0075 sys_close
    2 0.0050 __atomic_notifier_call_chain
    2 0.0050 __find_get_block_slow
    2 0.0050 __lookup_mnt
    2 0.0050 __mutex_unlock_slowpath
    2 0.0050 __rb_rotate_left
    2 0.0050 __rcu_process_callbacks
    2 0.0050 __switch_to
    2 0.0050 anon_vma_prepare
    2 0.0050 cache_alloc_refill
    2 0.0050 debug_mutex_add_waiter
    2 0.0050 do_path_lookup
    2 0.0050 dup_fd
    2 0.0050 generic_fillattr
    2 0.0050 generic_make_request
    2 0.0050 get_task_mm
    2 0.0050 get_unused_fd
    2 0.0050 getnstimeofday
    2 0.0050 hrtimer_interrupt
    2 0.0050 hweight32
    2 0.0050 inotify_dentry_parent_queue_event
    2 0.0050 link_path_walk
    2 0.0050 mask_and_ack_8259A
    2 0.0050 may_open
    2 0.0050 path_walk
    2 0.0050 pgd_alloc
    2 0.0050 preempt_schedule
    2 0.0050 proc_lookup
    2 0.0050 tick_nohz_restart_sched_tick
    2 0.0050 unlink_file_vma
    2 0.0050 vm_stat_account
    2 0.0050 vma_merge
    2 0.0050 zone_watermark_ok
    1 0.0025 __alloc_pages
    1 0.0025 __alloc_skb
    1 0.0025 __copy_from_user_ll
    1 0.0025 __copy_user_intel
    1 0.0025 __dentry_open
    1 0.0025 __do_softirq
    1 0.0025 __end_that_request_first
    1 0.0025 __fput
    1 0.0025 __free_pages
    1 0.0025 __getblk
    1 0.0025 __inc_zone_state
    1 0.0025 __kmalloc
    1 0.0025 __pagevec_lru_add
    1 0.0025 __pte_alloc
    1 0.0025 __put_task_struct
    1 0.0025 __tasklet_schedule
    1 0.0025 __vm_enough_memory
    1 0.0025 __vma_link_rb
    1 0.0025 _read_lock_irqsave
    1 0.0025 _spin_lock_irqsave
    1 0.0025 account_user_time
    1 0.0025 alloc_buffer_head
    1 0.0025 alloc_inode
    1 0.0025 apic_timer_interrupt
    1 0.0025 arch_get_unmapped_area_topdown
    1 0.0025 arch_pick_mmap_layout
    1 0.0025 attach_pid
    1 0.0025 bio_init
    1 0.0025 block_read_full_page
    1 0.0025 cache_reap
    1 0.0025 call_rcu
    1 0.0025 can_share_swap_page
    1 0.0025 cfq_set_request
    1 0.0025 check_userspace
    1 0.0025 cleanup_timers
    1 0.0025 clockevents_program_event
    1 0.0025 clocksource_get_next
    1 0.0025 compute_creds
    1 0.0025 copy_from_user
    1 0.0025 copy_strings
    1 0.0025 create_read_pipe
    1 0.0025 current_fs_time
    1 0.0025 current_io_context
    1 0.0025 d_instantiate
    1 0.0025 d_splice_alias
    1 0.0025 debug_mutex_unlock
    1 0.0025 delayed_put_task_struct
    1 0.0025 dnotify_flush
    1 0.0025 dnotify_parent
    1 0.0025 do_exit
    1 0.0025 do_fcntl
    1 0.0025 do_lookup
    1 0.0025 do_munmap
    1 0.0025 do_notify_resume
    1 0.0025 do_softirq
    1 0.0025 do_sys_open
    1 0.0025 do_wait
    1 0.0025 down_read_trylock
    1 0.0025 down_write
    1 0.0025 dput
    1 0.0025 dummy_inode_getattr
    1 0.0025 eligible_child
    1 0.0025 elv_dispatch_sort
    1 0.0025 end_buffer_write_sync
    1 0.0025 enqueue_hrtimer
    1 0.0025 exit_mmap
    1 0.0025 fget
    1 0.0025 fget_light
    1 0.0025 file_read_actor
    1 0.0025 filp_close
    1 0.0025 find_busiest_group
    1 0.0025 find_lock_page
    1 0.0025 find_next_zero_bit
    1 0.0025 find_or_create_page
    1 0.0025 find_vma_prev
    1 0.0025 flush_old_exec
    1 0.0025 flush_sigqueue
    1 0.0025 flush_tlb_page
    1 0.0025 fput
    1 0.0025 free_block
    1 0.0025 free_page_and_swap_cache
    1 0.0025 free_pgtables
    1 0.0025 free_poll_entry
    1 0.0025 generic_file_mmap
    1 0.0025 generic_segment_checks
    1 0.0025 get_empty_filp
    1 0.0025 get_next_timer_interrupt
    1 0.0025 getname
    1 0.0025 inode_change_ok
    1 0.0025 internal_add_timer
    1 0.0025 invalidate_inode_buffers
    1 0.0025 kill_fasync
    1 0.0025 kmap
    1 0.0025 kmem_cache_zalloc
    1 0.0025 ktime_get
    1 0.0025 kunmap_atomic
    1 0.0025 ll_back_merge_fn
    1 0.0025 lookup_hash
    1 0.0025 lookup_mnt
    1 0.0025 lru_cache_add_active
    1 0.0025 mempool_free_slab
    1 0.0025 mm_alloc
    1 0.0025 mm_release
    1 0.0025 mmput
    1 0.0025 mntput_no_expire
    1 0.0025 native_set_pte_at
    1 0.0025 number
    1 0.0025 page_add_file_rmap
    1 0.0025 page_add_new_anon_rmap
    1 0.0025 page_remove_rmap
    1 0.0025 page_waitqueue
    1 0.0025 path_release
    1 0.0025 peer_avl_rebalance
    1 0.0025 percpu_counter_mod
    1 0.0025 permission
    1 0.0025 pipe_write
    1 0.0025 prio_tree_replace
    1 0.0025 proc_flush_task
    1 0.0025 proc_pid_lookup
    1 0.0025 radix_tree_tag_set
    1 0.0025 rb_insert_color
    1 0.0025 remove_vma
    1 0.0025 restore_all
    1 0.0025 rm_from_queue_full
    1 0.0025 rw_verify_area
    1 0.0025 sched_balance_self
    1 0.0025 set_bh_page
    1 0.0025 set_normalized_timespec
    1 0.0025 set_task_comm
    1 0.0025 setup_sigcontext
    1 0.0025 show_stat
    1 0.0025 sync_page
    1 0.0025 sys_dup2
    1 0.0025 sys_faccessat
    1 0.0025 sys_ftruncate
    1 0.0025 sys_read
    1 0.0025 sys_rt_sigprocmask
    1 0.0025 sys_select
    1 0.0025 task_rq_lock
    1 0.0025 task_running_tick
    1 0.0025 tcp_poll
    1 0.0025 tick_do_update_jiffies64
    1 0.0025 tick_nohz_stop_sched_tick
    1 0.0025 try_to_wake_up
    1 0.0025 unix_poll
    1 0.0025 unlock_buffer
    1 0.0025 up_read
    1 0.0025 update_wall_time
    1 0.0025 vfs_getattr
    1 0.0025 vfs_permission
    1 0.0025 vfs_read
    1 0.0025 vm_acct_memory
    1 0.0025 vsnprintf
    1 0.0025 wake_up_inode
    1 0.0025 wake_up_new_task
    1 0.0025 work_notifysig
    procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
    r b swpd free buff cache si so bi bo in cs us sy id wa
    4 2 0 161676 9716 55700 0 0 248 2020 50 131 5 4 84 7
    4 2 0 161668 9716 55704 0 0 0 16000 253 504 80 20 0 0
    3 2 0 161668 9716 55704 0 0 0 16000 249 522 66 34 0 0
    3 2 0 161608 9716 55704 0 0 0 15936 251 508 80 20 0 0
    3 2 0 161548 9716 55704 0 0 0 15936 249 523 73 28 0 0
    4 2 0 161488 9716 55704 0 0 0 15936 250 517 84 16 0 0
    3 2 0 161488 9724 55704 0 0 0 15876 252 469 57 43 0 0
    4 2 0 161428 9724 55704 0 0 0 15872 253 504 60 40 0 0
    3 2 0 161428 9724 55704 0 0 0 15936 248 500 24 76 0 0
    4 2 0 161428 9740 55704 0 0 0 15840 258 475 52 48 0 0
    \
     
     \ /
      Last update: 2007-08-06 21:07    [W:0.091 / U:0.512 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site