Messages in this thread |  | | From | Changwoo Min <> | | Subject | [PATCH v2 0/5] sched_ext: Support high-performance monotonically non-decreasing clock | | Date | Mon, 2 Dec 2024 13:38:44 +0900 |
| |
Many BPF schedulers (such as scx_central, scx_lavd, scx_rusty, scx_bpfland, scx_flash) frequently call bpf_ktime_get_ns() for tracking tasks' runtime properties. If supported, bpf_ktime_get_ns() eventually reads a hardware timestamp counter (TSC). However, reading a hardware TSC is not performant in some hardware platforms, degrading IPC.
This patchset addresses the performance problem of reading hardware TSC by leveraging the rq clock in the scheduler core, introducing a scx_bpf_clock_get_ns() function for BPF schedulers. Whenever the rq clock is fresh and valid, scx_bpf_clock_get_ns() provides the rq clock, which is already updated by the scheduler core (update_rq_clock), so it can reduce reading the hardware TSC.
When the rq lock is released (rq_unpin_lock) or a long-running operations are done by the BPF scheduler (ops.running, ops.update_idle), the rq clock is invalidated, so a subsequent scx_bpf_clock_get_ns() call gets the fresh sched_clock for the caller.
In addition, scx_bpf_clock_get_ns() guarantees the clock is monotonically non-decreasing for the same CPU, so the clock cannot go backward in the same CPU.
Using scx_bpf_clock_get_ns() reduces the number of reading hardware TSC by 40-70% (65% for scx_lavd, 58% for scx_bpfland, and 43% for scx_rusty) for the following benchmark.
perf bench -f simple sched messaging -t -g 20 -l 6000
The patchset begins by managing the status of rq clock in the scheduler core, then implementing scx_bpf_clock_get_ns(), and finally applying it to the BPF schedulers.
ChangeLog v1 -> v2: - Rename SCX_RQ_CLK_UPDATED to SCX_RQ_CLK_VALID to denote the validity of an rq clock clearly. - Rearrange the clock and flags fields in struct scx_rq to make sure they are in the same cacheline to minimize the cache misses - Add an additional explanation to the commit message in the 2/5 patch describing when the rq clock will be reused with an example. - Fix typos - Rebase the code to the tip of Tejun's sched_ext repo
Changwoo Min (5): sched_ext: Implement scx_rq_clock_update/stale() sched_ext: Manage the validity of scx_rq_clock sched_ext: Implement scx_bpf_clock_get_ns() sched_ext: Add scx_bpf_clock_get_ns() for BPF scheduler sched_ext: Replace bpf_ktime_get_ns() to scx_bpf_clock_get_ns()
kernel/sched/core.c | 6 +- kernel/sched/ext.c | 74 ++++++++++++++++++++++++ kernel/sched/sched.h | 24 +++++++- tools/sched_ext/include/scx/common.bpf.h | 1 + tools/sched_ext/include/scx/compat.bpf.h | 5 ++ tools/sched_ext/scx_central.bpf.c | 4 +- tools/sched_ext/scx_flatcg.bpf.c | 2 +- 7 files changed, 110 insertions(+), 6 deletions(-)
-- 2.47.1
|  |