Messages in this thread | | | Date | Fri, 29 Feb 2008 08:18:18 -0500 | From | Mark Seger <> | Subject | Collectl now support monitoring Interrupts by CPU |
| |
In keeping with the spirit of adding new useful features when they help solve real world problems, I've added this ability to collectl as a direct result of a problem we were recently having when doing some network performance testing. It turned out all the interrupts were being processed by cpu0 (this was on an 8-core system) but all collectl told us was the aggregate number! Once we moved to a NIC/driver that supported multiple queues that could distribute interrupts to multiple CPUs it only made sense to enhance collectl to let us verify that this was indeed happening - I personally find examining /proc/interrupts for changes more trouble than it's worth.
In any event, the following is an example of how collectl can present this data, first summarized by CPU:
#Time Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7 12:49:55 4828 16K 1000 16K 18 16K 16K 0 12:49:56 4804 16K 1000 16K 0 16K 16K 0 12:49:57 4811 16K 1000 16K 18 16K 16K 0 12:49:58 4789 16K 1000 16K 0 17K 16K 44
and here by the type of the interrupt itself:
# INTERRUPT DETAILS # Int Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7 Type Device(s) 12:48:50 082 0 0 0 7731 0 0 0 0 PCI-MSI-X eth2 (queue 0) 12:48:50 098 0 0 0 0 2037 0 0 0 PCI-MSI-X eth2 (queue 2) 12:48:50 122 0 0 2240 0 0 0 0 0 PCI-MSI-X eth2 (queue 5) 12:48:50 138 0 7084 0 0 0 0 0 0 PCI-MSI-X eth2 (queue 7) 12:48:50 154 0 0 0 0 0 7723 0 0 PCI-MSI-X eth3 (queue 0) 12:48:50 162 9082 0 0 0 0 0 0 0 PCI-MSI-X eth3 (queue 1) 12:48:50 178 0 0 0 0 0 0 8253 0 PCI-MSI-X eth3 (queue 3) 12:48:50 210 0 0 0 0 0 0 0 6417 PCI-MSI-X eth3 (queue 7) 12:48:50 218 1 0 0 0 0 0 0 0 PCI-MSI eth0
You can also look at all CPU loads and interrupts together like this:
# SINGLE CPU STATISTICS # CPU USER NICE SYS WAIT IRQ SOFT STEAL IDLE INTRPT 07:09:28 0 0 0 0 0 0 0 0 100 16 07:09:28 1 0 0 0 0 0 0 0 100 0 07:09:28 2 0 0 0 0 0 0 0 100 999 07:09:28 3 0 0 0 0 0 0 0 100 16
For those not familiar with collectl, you can collect virtually everything all the existing linux 'stat' utilitie provide plus a lot more such as processes (including I/O stats), Infininband, Quadrics, Slab, Lustre and more. Plus a lot more feature too numerous to list but there's a pretty good summary here - http://collectl.sourceforge.net/Features.html
But don't take my word for it, check out the website at http://collectl.sourceforge.net/ where you can see a pretty good set of examples in the documentation and even follow the tutorial.
-mark
| |