Messages in this thread | | | From | Venkatesh Pallipadi <> | Subject | [PATCH 0/6] Proper kernel irq time accounting | Date | Thu, 16 Sep 2010 18:56:30 -0700 |
| |
Here is some background information about interrupt time accounting in kernel and related problems
Interrupts always run in the context of currently running task. Softirqs mostly run in the context of currently running task, unless softirqd gets involved. /proc/interrupts and /proc/softirqs are the interfaces that report the number of interrupts and softirqs per CPU since boot. /proc/stat has fields that report per CPU and system-wide interrupt and softirq processing time in clock_t units.
There are two problems with the way interrupts are accounted by the kernel.
(1) Coarse grained interrupt time reporting On most archs (except s390, powerpc, ia64 with CONFIG_VIRT_CPU_ACCOUNTING), the interrupt and softirq time reported in /proc/stat is tick sample based. Kernel looks at what it is doing at each CPU tick and accounts the entire tick to that particular activity. This means the data in /proc/stat is pretty coarse grained.
One related problem (atleast on x86), with recent "Run irq handlers with interrupts disabled" change, timer interrupt cannot fire when there is an interrupt being serviced [1]. As a result sampling based hardirq time in /proc/stat cannot capture any hardirq time at all.
(2) Accounting irq processing time to current task/taskgroup Whenever irq processing happens, kernel accounts that time to currently running task. The exec_runtime reported in /proc/<pid>/schedstat and <cgroup>/cpuacct.usage* includes any irq processing that happened while the task was running. The scheduler vruntime calculations also account any irq processing to the current task. This means exec time accounting during heavy irq processing is kind of random, depending on when and which CPU processing happens and what task happened to be running on that CPU at that time.
Solution to (1) involves adding extra timing on irq entry/exit to get the fine granularity info and then exporting it to user. The following patchset addresses this problem in a way similar to [2][3]. Keeps most of the code that does the timing generic (CONFIG_IRQ_TIME_ACCOUNTING), based off of sched_clock(). And adds support for this in x86. The new fine granularity time information is exported in /proc/interrupts and /proc/softirqs as a reference implementation. Whether it actually belongs there or somewhere else is open for discussion.
One partial solution proposed in [2][3] for (2) above, was to capture this interrupt time at task/taskgroup level and let user know how much irq processing time kernel charged to each task/taskgroup. But, that solution did not solve task timeslice including irq processing. Peter Zijlstra and Martin Schwidefsky disagreed with that approach and wanted to see more complete solution in not accounting irq processing time to tasks at all.
The patchset below tries this more complete solution, with two scheduler related changes. First, to take out irq processing time from the time scheduler accounts to the task. Second, make adjustments to the CPU power, to account for irq processing activity on the CPU. That in turn results in irq busy CPU pulling tasks corresponding to its non-irq processing bandwidth that it has got.
The changes here is only enabled for CONFIG_IRQ_TIME_ACCOUNTING as of now. I haven't gotten enough testing cycles on this patchset yet. But, here they are for review/feedback.
Thanks, Venki
References:
[1] http://lkml.indiana.edu/hypermail//linux/kernel/1005.3/00864.html lkml subject - "genirq: Run irq handlers with interrupts disabled"
[2] http://lkml.indiana.edu/hypermail//linux/kernel/1005.3/00411.html lkml subject - "Finer granularity and task/cgroup irq time accounting"
[3] http://lkml.indiana.edu/hypermail//linux/kernel/1007.2/00987.html lkml subject - "Finer granularity and task/cgroup irq time accounting"
| |