Messages in this thread | | | Date | Fri, 5 Aug 2016 12:55:04 -0300 | From | Arnaldo Carvalho de Melo <> | Subject | Re: perf test BPF failing on f24: fix |
| |
Em Fri, Aug 05, 2016 at 11:35:32AM -0300, Arnaldo Carvalho de Melo escreveu: > Em Fri, Aug 05, 2016 at 06:45:50PM +0900, Masami Hiramatsu escreveu: > > See the symbol address calcurated from symbol map, in successful case > > the address exactly same address of SyS_epoll_wait. This indicates > > something might wrong in the symbol map. (maybe KASLR?) > > Could you check what happen if nokaslr is passed to your kernel?
> [acme@jouet linux]$ grep RANDOMIZE_BASE ../build/v4.7.0+/.config > CONFIG_RANDOMIZE_BASE=y
> Yes, it is in place, I'll check rebooting with nokaslr.
> Can you think about how to properly warn the user about this > possibility, if it is confirmed to cause this issue?
> I wonder why this doesn't seem to affect annotation, that also uses vmlinux, > for instance:
> [root@jouet ~]# perf record --all-kernel -F 10000 ping -f -c 100000 localhost > PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
> --- localhost.localdomain ping statistics --- > 100000 packets transmitted, 100000 received, 0% packet loss, time 732ms > rtt min/avg/max/mdev = 0.003/0.003/0.048/0.002 ms, ipg/ewma 0.007/0.003 ms > [ perf record: Woken up 2 times to write data ] > [ perf record: Captured and wrote 0.298 MB perf.data (7340 samples) ] > [root@jouet ~]# perf report -v | grep '\[' | head -10 > Using /lib/modules/4.7.0+/build/vmlinux for symbols > 20.42% ping /lib/modules/4.7.0+/build/vmlinux 0x960374 v [k] ipt_do_table > 2.55% ping /lib/modules/4.7.0+/build/vmlinux 0x900b8a v [k] nf_iterate > 1.94% ping /lib/modules/4.7.0+/build/vmlinux 0x95190b v [k] fib_table_lookup > 1.68% ping /lib/modules/4.7.0+/build/vmlinux 0x2accfa v [k] __local_bh_enable_ip > 1.68% ping /lib/modules/4.7.0+/build/vmlinux 0x9e41ab v [k] _raw_spin_lock_irqsave > 1.66% ping /lib/modules/4.7.0+/build/vmlinux 0x55fd94 v [k] avc_has_perm > 1.65% ping /lib/modules/4.7.0+/build/vmlinux 0x90e3e2 v [k] __ip_append_data.isra.44 > 1.31% ping /lib/modules/4.7.0+/build/vmlinux 0x5ead59 v [k] copy_user_enhanced_fast_string > 1.21% ping /lib/modules/4.7.0+/build/vmlinux 0x938820 v [k] raw_local_deliver > 1.14% ping /lib/modules/4.7.0+/build/vmlinux 0x6c6cea v [k] n_tty_write > [root@jouet ~]#
> [root@jouet ~]# perf annotate -v --stdio ipt_do_table > Looking at the vmlinux_path (8 entries long) > Using /lib/modules/4.7.0+/build/vmlinux for symbols > symbol__disassemble: filename=/lib/modules/4.7.0+/build/vmlinux, sym=ipt_do_table, start=0xffffffffbd7601c0, end=0xffffffffbd7607b0 > annotating [0x3cef9b0] /lib/modules/4.7.0+/build/vmlinux : [0x4159750] ipt_do_table > Executing: objdump --start-address=0xffffffff817601c0 --stop-address=0xffffffff817607b0 -l -d --no-show-raw -S -C /lib/modules/4.7.0+/build/vmlinux 2>/dev/nu > Percent | Source code & Disassembly of vmlinux for cycles:ppp (1498 samples) > ---------------------------------------------------------------------------------- > <SNIP>
> [root@jouet ~]# grep ffffffffbd7601c0 /proc/kallsyms > ffffffffbd7601c0 T ipt_do_table > [root@jouet ~]#
> Anyway, will reboot this machine with nokaslr to check that possibility.
So, more info, but executive summary: distro kernels don't exhibit this problem, but they have CONFIG_RANDOMIZE_BASE=y, the kernel where I noticed this problem (4.7.0+) doesn't exhibit this problem when used with 'nokaslr', but I think more investigation is needed to get to the bottom of this, i.e. to emit meaningful messages when kaslr is in place (or is suspected to be):
Using a distro kernel:
[root@jouet ~]# uname -r 4.6.4-301.fc24.x86_64 [root@jouet ~]# perf probe sys_epoll_wait Added new event: probe:sys_epoll_wait (on sys_epoll_wait)
You can now use it in all perf tools, such as:
perf record -e probe:sys_epoll_wait -aR sleep 1
[root@jouet ~]# rpm -q kernel-debuginfo kernel-debuginfo-4.6.3-300.fc24.x86_64 [root@jouet ~]# uname -r 4.6.4-301.fc24.x86_64 [root@jouet ~]#
So it must be using just /proc/kallsyms, since there is no matching vmlinux:
[root@jouet ~]# perf probe -d *:* Removed event: probe:sys_epoll_wait [root@jouet ~]# perf probe -v sys_epoll_wait probe-definition(0): sys_epoll_wait symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (8 entries long) symsrc__init: cannot get elf header. Could not open debuginfo. Try to use symbols. Looking at the vmlinux_path (8 entries long) symsrc__init: cannot get elf header. Using /proc/kcore for kernel object code Using /proc/kallsyms for symbols Opening /sys/kernel/debug/tracing//kprobe_events write=1 Writing event: p:probe/sys_epoll_wait _text+2698592 Added new event: probe:sys_epoll_wait (on sys_epoll_wait)
You can now use it in all perf tools, such as:
perf record -e probe:sys_epoll_wait -aR sleep 1
[root@jouet ~]#
Ok, installing a matching kernel vmlinux:
[root@jouet ~]# perf probe -d *:* Removed event: probe:sys_epoll_wait [root@jouet ~]# dnf debuginfo-install kernel
Trying again:
[root@jouet ~]# perf probe -v sys_epoll_wait probe-definition(0): sys_epoll_wait symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (8 entries long) Using /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux for symbols Open Debuginfo file: /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux Try to find probe point from debuginfo. Symbol sys_epoll_wait address found : ffffffff81292d60 Matched function: SyS_epoll_wait found inline addr: 0xffffffff812930f3 Probe point found: compat_SyS_epoll_pwait+147 found inline addr: 0xffffffff81292ed6 Probe point found: SyS_epoll_pwait+134 found inline addr: 0xffffffff81292d60 Probe point found: SyS_epoll_wait+0 Found 3 probe_trace_events. Opening /sys/kernel/debug/tracing//kprobe_events write=1 Writing event: p:probe/sys_epoll_wait _text+2699507 Writing event: p:probe/sys_epoll_wait_1 _text+2698966 Writing event: p:probe/sys_epoll_wait_2 _text+2698592 Added new events: probe:sys_epoll_wait (on sys_epoll_wait) probe:sys_epoll_wait_1 (on sys_epoll_wait) probe:sys_epoll_wait_2 (on sys_epoll_wait)
You can now use it in all perf tools, such as:
perf record -e probe:sys_epoll_wait_2 -aR sleep 1
[root@jouet ~]#
So, a distro kernel works with or without vmlinux, but then...
[root@jouet ~]# grep RANDOMIZE_BASE /boot/config-4.6.4-301.fc24.x86_64 CONFIG_RANDOMIZE_BASE=y CONFIG_RANDOMIZE_BASE_MAX_OFFSET=0x40000000 [root@jouet ~]#
It also has kaslr :-\ Rebooting with the kernel where the problem happened, this time with nokaslr:
[root@jouet ~]# uname -r 4.7.0+ [root@jouet ~]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.7.0+ root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet LANG=en_US.UTF-8 nokaslr [root@jouet ~]# perf probe -v sys_epoll_wait probe-definition(0): sys_epoll_wait symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (8 entries long) Using /lib/modules/4.7.0+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux Try to find probe point from debuginfo. Symbol sys_epoll_wait address found : ffffffff81295b50 Matched function: SyS_epoll_wait found inline addr: 0xffffffff81295ee7 Probe point found: compat_SyS_epoll_pwait+151 found inline addr: 0xffffffff81295cca Probe point found: SyS_epoll_pwait+138 found inline addr: 0xffffffff81295b50 Probe point found: SyS_epoll_wait+0 Found 3 probe_trace_events. Opening /sys/kernel/debug/tracing//kprobe_events write=1 Writing event: p:probe/sys_epoll_wait _text+2711271 Writing event: p:probe/sys_epoll_wait_1 _text+2710730 Writing event: p:probe/sys_epoll_wait_2 _text+2710352 Added new events: probe:sys_epoll_wait (on sys_epoll_wait) probe:sys_epoll_wait_1 (on sys_epoll_wait) probe:sys_epoll_wait_2 (on sys_epoll_wait) You can now use it in all perf tools, such as: perf record -e probe:sys_epoll_wait_2 -aR sleep 1
[root@jouet ~]#
It works, but why was annotation worked when "nokaslr" wasn't used?
- Arnaldo
| |