Messages in this thread | | | From | Song Liu <> | Subject | Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF | Date | Thu, 18 Mar 2021 03:52:51 +0000 |
| |
> On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > > Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu: >> Hi Song, >> >> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote: >>> >>> perf uses performance monitoring counters (PMCs) to monitor system >>> performance. The PMCs are limited hardware resources. For example, >>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu. >>> >>> Modern data center systems use these PMCs in many different ways: >>> system level monitoring, (maybe nested) container level monitoring, per >>> process monitoring, profiling (in sample mode), etc. In some cases, >>> there are more active perf_events than available hardware PMCs. To allow >>> all perf_events to have a chance to run, it is necessary to do expensive >>> time multiplexing of events. >>> >>> On the other hand, many monitoring tools count the common metrics (cycles, >>> instructions). It is a waste to have multiple tools create multiple >>> perf_events of "cycles" and occupy multiple PMCs. >> >> Right, it'd be really helpful when the PMCs are frequently or mostly shared. >> But it'd also increase the overhead for uncontended cases as BPF programs >> need to run on every context switch. Depending on the workload, it may >> cause a non-negligible performance impact. So users should be aware of it. > > Would be interesting to, humm, measure both cases to have a firm number > of the impact, how many instructions are added when sharing using > --bpf-counters? > > I.e. compare the "expensive time multiplexing of events" with its > avoidance by using --bpf-counters. > > Song, have you perfmormed such measurements?
I have got some measurements with perf-bench-sched-messaging:
The system: x86_64 with 23 cores (46 HT)
The perf-stat command: perf stat -e cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target, etc.>
The benchmark command and output: ./perf bench sched messaging -g 40 -l 50000 -t # Running 'sched/messaging' benchmark: # 20 sender and receiver threads per group # 40 groups == 1600 threads run Total time: 10X.XXX [sec]
I use the "Total time" as measurement, so smaller number is better.
For each condition, I run the command 5 times, and took the median of "Total time".
Baseline (no perf-stat) 104.873 [sec] # global perf stat -a 107.887 [sec] perf stat -a --bpf-counters 106.071 [sec] # per task perf stat 106.314 [sec] perf stat --bpf-counters 105.965 [sec] # per cpu perf stat -C 1,3,5 107.063 [sec] perf stat -C 1,3,5 --bpf-counters 106.406 [sec]
From the data, --bpf-counters is slightly better than the regular event for all targets. I noticed that the results are not very stable. There are a couple 108.xx runs in some of the conditions (w/ and w/o --bpf-counters).
I also measured the average runtime of the BPF programs, with
sysctl kernel.bpf_stats_enabled=1
For each event, if we have one leader and two followers, the total run time is about 340ns. IOW, 340ns for two perf-stat reading instructions, 340ns for two perf-stat reading cycles, etc.
Thanks, Song
| |