lkml.org 
[lkml]   [2006]   [Apr]   [4]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
Subjectwhat's my kernel doing?

We have a 4CPU AMD (v40z) machine running RedHat 4.0 (2.6.9-11.ELsmp), and
I noticed high CPU utilization especially on the system side:

[root@mars ~]# mpstat 5
Linux 2.6.9-11.ELsmp (mars) 04/04/2006

01:37:32 PM CPU %user %nice %system %iowait %irq %soft %idle
intr/s
01:37:37 PM all 38.50 0.00 50.95 0.00 0.30 3.20 7.05
6750.80
01:37:42 PM all 41.91 0.00 53.10 0.00 0.30 3.10 1.60
6884.60
01:37:47 PM all 39.97 0.00 53.08 0.00 0.25 3.05 3.65
6672.65


Initially I thought it was a huge file system cache that system was
scanning a lot.

[VCRS@mars admin]$free
total used free shared buffers cached
Mem: 16032636 12539720 3492916 0 244440 11640608
-/+ buffers/cache: 654672 15377964
Swap: 8385920 28 8385892

However, I unmounted/umounted the oracle file system in directio mode the
cache didn't go away, so it must be some root disk cache.. (there also
used to be a parameter to control pagecache up to kernel 2.6
(/proc/sys/vm/pagecache), but it seems to be gone now, couldn't find an
equivalent nor the reason it is gone)

Then I installed the 'profiler', which I can barely understand, but
there's a difference between sun & mars (identical machines, both running
oracle databases). It seems weird that some usb process is taking up so
much cpu, however on sun (which doesn't have a similar system utilization)
it's near the 'top' too, just at a much lower percentage.

If anyone here could give me better advice on how to go about this problem
(high system cpu utilization), or point me to some resources, I'd really
appreciate it.


[root@sun ~]# opreport|head
CPU: AMD64 processors, speed 2592.7 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
1567396 54.8034 oracle
904729 31.6335 vmlinux
136014 4.7557 libc-2.3.4.so
33860 1.1839 e1000
26277 0.9188 vxfs

[root@sun ~]# opreport -l
/usr/lib/debug/lib/modules/2.6.9-5.EL/vmlinux|head
CPU: AMD64 processors, speed 2592.7 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
samples % symbol name
113702 11.3522 usbfs_add_bus
80084 7.9957 get_user_pages
53515 5.3430 init_8259A
40519 4.0455 sys_access
28709 2.8663 load_elf32_binary
26368 2.6326 setscheduler
23848 2.3810 __ide_dma_write


Oh yeah, 2.6.9-5EL was the only 'kernel-debug' rpm I could find.


[root@mars tmp]# opreport|head
CPU: AMD64 processors, speed 2592.66 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
23361362 66.3184 vmlinux
9122930 25.8982 oracle
1102906 3.1309 no-vmlinux
539234 1.5308 libc-2.3.4.so
183650 0.5213 e1000

[root@mars tmp]# opreport -l
/usr/lib/debug/lib/modules/2.6.9-5.EL/vmlinux|head
CPU: AMD64 processors, speed 2592.66 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
samples % symbol name
8155421 34.4932 usbfs_add_bus
4776837 20.2035 get_user_pages
2954158 12.4945 sys_access
1702339 7.2000 setscheduler
779719 3.2978 sys_fchdir
367546 1.5545 sys_open
260518 1.1019 init_8259A





































Please let me know which list would be appropriate for a question like
this, if this list is for kernel development only.

Thanks,
andrey

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2006-04-04 19:44    [W:1.051 / U:0.012 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site