Messages in this thread Patches in this message | | | Subject | Re: ERROR: INT DW_ATE_unsigned_1 Error emitting BTF type | From | Yonghong Song <> | Date | Sat, 6 Feb 2021 00:26:44 -0800 |
| |
On 2/5/21 10:52 PM, Sedat Dilek wrote: > On Sat, Feb 6, 2021 at 7:26 AM Sedat Dilek <sedat.dilek@gmail.com> wrote: >> >> On Sat, Feb 6, 2021 at 6:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote: >>> >>> On Sat, Feb 6, 2021 at 6:44 AM Sedat Dilek <sedat.dilek@gmail.com> wrote: >>>> >>>> On Sat, Feb 6, 2021 at 4:34 AM Sedat Dilek <sedat.dilek@gmail.com> wrote: >>>>> >>>>> On Fri, Feb 5, 2021 at 10:54 PM Yonghong Song <yhs@fb.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 2/5/21 12:31 PM, Sedat Dilek wrote: >>>>>>> On Fri, Feb 5, 2021 at 9:03 PM Yonghong Song <yhs@fb.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 2/5/21 11:24 AM, Arnaldo Carvalho de Melo wrote: >>>>>>>>> Em Fri, Feb 05, 2021 at 11:10:08AM -0800, Yonghong Song escreveu: >>>>>>>>>> On 2/5/21 11:06 AM, Sedat Dilek wrote: >>>>>>>>>>> On Fri, Feb 5, 2021 at 7:53 PM Sedat Dilek <sedat.dilek@gmail.com> wrote: >>>>>>>>>>> Grepping through linux.git/tools I guess some BTF tools/libs need to >>>>>>>>>>> know what BTF_INT_UNSIGNED is? >>>>>>>>> >>>>>>>>>> BTF_INT_UNSIGNED needs kernel support. Maybe to teach pahole to >>>>>>>>>> ignore this for now until kernel infrastructure is ready. >>>>>>>>> >>>>>>>>> Yeah, I thought about doing that. >>>>>>>>> >>>>>>>>>> Not sure whether this information will be useful or not >>>>>>>>>> for BTF. This needs to be discussed separately. >>>>>>>>> >>>>>>>>> Maybe search for the rationale for its introduction in DWARF. >>>>>>>> >>>>>>>> In LLVM, we have: >>>>>>>> uint8_t BTFEncoding; >>>>>>>> switch (Encoding) { >>>>>>>> case dwarf::DW_ATE_boolean: >>>>>>>> BTFEncoding = BTF::INT_BOOL; >>>>>>>> break; >>>>>>>> case dwarf::DW_ATE_signed: >>>>>>>> case dwarf::DW_ATE_signed_char: >>>>>>>> BTFEncoding = BTF::INT_SIGNED; >>>>>>>> break; >>>>>>>> case dwarf::DW_ATE_unsigned: >>>>>>>> case dwarf::DW_ATE_unsigned_char: >>>>>>>> BTFEncoding = 0; >>>>>>>> break; >>>>>>>> >>>>>>>> I think DW_ATE_unsigned can be ignored in pahole since >>>>>>>> the default encoding = 0. A simple comment is enough. >>>>>>>> >>>>>>> >>>>>>> Yonghong Son, do you have a patch/diff for me? >>>>>> >>>>>> Looking at error message from log: >>>>>> >>>>>> LLVM_OBJCOPY=/opt/binutils/bin/objcopy /opt/pahole/bin/pahole -J >>>>>> .tmp_vmlinux.btf >>>>>> [115] INT DW_ATE_unsigned_1 Error emitting BTF type >>>>>> Encountered error while encoding BTF. >>>>>> >>>>>> Not exactly what is the root cause. Maybe bt->bit_size is not >>>>>> encoded correctly. Could you put vmlinux (in the above it is >>>>>> .tmp_vmlinux.btf) somewhere, I or somebody else can investigate >>>>>> and provide a proper fix. >>>>>> >>>>> >>>>> [ TO: Masahiro ] >>>>> >>>>> Thanks for taking care Yonghong - hope this is your first name, if not >>>>> I am sorry. >>>>> In case of mixing my first and last name you will make me female - >>>>> Dilek is a Turkish female first name :-). >>>>> So, in some cultures you need to be careful. >>>>> >>>>> Anyway... back to business and facts. >>>>> >>>>> Out of frustration I killed my last build via `make distclean`. >>>>> The whole day I tested diverse combination of GCC-10 and LLVM-12 >>>>> together with BTF Kconfigs, selfmade pahole, etc. >>>>> >>>>> I will do ne run with some little changes: >>>>> >>>>> #1: Pass LLVM_IAS=1 to make (means use Clang's Integrated ASsembler - >>>>> as per Nick this leads to the same error - should be unrelated) >>>>> #2: I did: DEBUG_INFO_COMPRESSED y -> n >>>>> >>>>> #2 I did in case you need vmlinux and I have to upload - I will >>>>> compress the resulting vmlinux with ZSTD. >>>>> You need vmlinux or .tmp_vmlinux.btf file? >>>>> Nick was not allowed from his company to download from a Dropbox link. >>>>> So, as an alternative I can offer GoogleDrive... >>>>> ...or bomb into your INBOX :-). >>>>> >>>>> Now, why I CCed Masahiro: >>>>> >>>>> In case of ERRORs when running `scripts/link-vmlinux.sh` above files >>>>> will be removed. >>>>> >>>>> Last, I found a hack to bypass this - means to keep these files (I >>>>> need to check old emails). >>>>> >>>>> Masahiro, you see a possibility to have a way to keep these files in >>>>> case of ERRORs without doing hackery? >>>>> >>>>> From a previous post in this thread: >>>>> >>>>> + info BTF .btf.vmlinux.bin.o >>>>> + [ != silent_ ] >>>>> + printf %-7s %s\n BTF .btf.vmlinux.bin.o >>>>> BTF .btf.vmlinux.bin.o >>>>> + LLVM_OBJCOPY=llvm-objcopy /opt/pahole/bin/pahole -J .tmp_vmlinux.btf >>>>> [2] INT long unsigned int Error emitting BTF type >>>>> Encountered error while encoding BTF. >>>>> + llvm-objcopy --only-section=.BTF --set-section-flags >>>>> .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o >>>>> ... >>>>> + info BTFIDS vmlinux >>>>> + [ != silent_ ] >>>>> + printf %-7s %s\n BTFIDS vmlinux >>>>> BTFIDS vmlinux >>>>> + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux >>>>> FAILED: load BTF from vmlinux: Invalid argument >>>>> + on_exit >>>>> + [ 255 -ne 0 ] >>>>> + cleanup >>>>> + rm -f .btf.vmlinux.bin.o >>>>> + rm -f .tmp_System.map >>>>> + rm -f .tmp_vmlinux.btf .tmp_vmlinux.kallsyms1 >>>>> .tmp_vmlinux.kallsyms1.S .tmp_vmlinux.kallsyms1.o >>>>> .tmp_vmlinux.kallsyms2 .tmp_vmlinux.kallsyms2.S .tmp_vmlinux.kallsyms >>>>> 2.o >>>>> + rm -f System.map >>>>> + rm -f vmlinux >>>>> + rm -f vmlinux.o >>>>> make[3]: *** [Makefile:1166: vmlinux] Error 255 >>>>> >>>>> ^^^ Look here. >>>>> >>>> >>>> With this diff: >>>> >>>> $ git diff scripts/link-vmlinux.sh >>>> diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh >>>> index eef40fa9485d..40f1b6aae553 100755 >>>> --- a/scripts/link-vmlinux.sh >>>> +++ b/scripts/link-vmlinux.sh >>>> @@ -330,7 +330,7 @@ vmlinux_link vmlinux "${kallsymso}" ${btf_vmlinux_bin_o} >>>> # fill in BTF IDs >>>> if [ -n "${CONFIG_DEBUG_INFO_BTF}" -a -n "${CONFIG_BPF}" ]; then >>>> info BTFIDS vmlinux >>>> - ${RESOLVE_BTFIDS} vmlinux >>>> + ##${RESOLVE_BTFIDS} vmlinux >>>> fi >>>> >>>> if [ -n "${CONFIG_BUILDTIME_TABLE_SORT}" ]; then >>>> >>>> This files are kept - not removed: >>>> >>>> $ LC_ALL=C ll .*btf* vmlinux vmlinux.o >>>> -rwxr-xr-x 1 dileks dileks 31M Feb 6 06:37 .btf.vmlinux.bin.o >>>> -rwxr-xr-x 1 dileks dileks 348M Feb 6 06:37 .tmp_vmlinux.btf >>>> -rwxr-xr-x 1 dileks dileks 348M Feb 6 06:37 vmlinux >>>> -rw-r--r-- 1 dileks dileks 344M Feb 6 06:37 vmlinux.o >>>> >>>> Pleas let me know where to upload - Dropbox or GoogleDrive or >>>> elsewhere and give me a link. >>>> >>> >>> >>> WOW, ZSTD is great :-). >>> >>> $ zstd -19 -T0 -v vmlinux >>> *** zstd command line interface 64-bits v1.4.8, by Yann Collet *** >>> Note: 2 physical core(s) detected >>> vmlinux : 22.71% (364466016 => 82784801 bytes, vmlinux.zst) >>> >>> $ du -m vmlinux* >>> 348 vmlinux >>> 79 vmlinux.zst >>> >> >> Dropbox link: >> https://www.dropbox.com/sh/kvyh8ps7na0r1h5/AABfyNfDZ2bESse_bo4h05fFa?dl=0 >> >> I hope this is public available. >> > > Inspecting vmlinux with llvm-dwarf: > > $ /opt/llvm-toolchain/bin/llvm-dwarfdump vmlinux | grep DW_AT_name | > grep DW_ATE_ | sort | uniq > DW_AT_name ("DW_ATE_signed_1") > DW_AT_name ("DW_ATE_signed_16") > DW_AT_name ("DW_ATE_signed_32") > DW_AT_name ("DW_ATE_signed_64") > DW_AT_name ("DW_ATE_signed_8") > DW_AT_name ("DW_ATE_unsigned_1") > DW_AT_name ("DW_ATE_unsigned_128") > DW_AT_name ("DW_ATE_unsigned_16") > DW_AT_name ("DW_ATE_unsigned_24") > DW_AT_name ("DW_ATE_unsigned_32") > DW_AT_name ("DW_ATE_unsigned_40") > DW_AT_name ("DW_ATE_unsigned_64") > DW_AT_name ("DW_ATE_unsigned_8") > > - Sedat -
Thanks for the above dropbot link, I am able to reproduce the issue.
I tried to use latest llvm + Nick's patch + latest pahole + dwarf5 config option to compile kernel with LLVM=1 LLVM_IAS=1, somehow I did not hit the issue. It complained like
MODPOST vmlinux.symvers WARNING: modpost: vmlinux.o(.text+0x6ce73): Section mismatch in reference from the function __nodes _weight() to the variable .init.data:numa_nodes_parsed The function __nodes_weight() references the variable __initdata numa_nodes_parsed. This is often because __nodes_weight lacks a __initdata annotation or the annotation of numa_nodes_parsed is wrong.
but otherwise compilation is fine.
With the above vmlinux, the issue appears to be handling DW_ATE_signed_1, DW_ATE_unsigned_{1,24,40}.
The following patch should fix the issue:
-bash-4.4$ git diff
diff --git a/dwarf_loader.c b/dwarf_loader.c
index b73d786..0341b5e 100644
--- a/dwarf_loader.c
+++ b/dwarf_loader.c
@@ -467,8 +467,16 @@ static struct base_type *base_type__new(Dwarf_Die *die, struct cu *cu)
if (bt != NULL) {
tag__init(&bt->tag, cu, die);
- bt->name = strings__add(strings, attr_string(die, DW_AT_name)); - bt->bit_size = attr_numeric(die, DW_AT_byte_size) * 8; + const char *name = attr_string(die, DW_AT_name); + bt->name = strings__add(strings, name); + /* DW_ATE_unsigned_1 has DW_AT_byte_size == 0. + * specially process it.
+ */
+ if (strcmp(name, "DW_ATE_unsigned_1") == 0) + bt->bit_size = 1; + else
+ bt->bit_size = attr_numeric(die, DW_AT_byte_size) * 8; +
uint64_t encoding = attr_numeric(die, DW_AT_encoding); bt->is_bool = encoding == DW_ATE_boolean; bt->is_signed = encoding == DW_ATE_signed; diff --git a/libbtf.c b/libbtf.c index 9f76283..b5aa077 100644 --- a/libbtf.c +++ b/libbtf.c @@ -367,13 +367,32 @@ static void btf_log_func_param(const struct btf_elf *btfe, } }
+/* btf requires power-of-2 bytes, yet dwarf may have something like + * DW_ATE_unsigned_24 which encodes as 24 bits (3 bytes). + */ +static int bits_to_int_bytes(uint16_t bit_size) +{ + if (bit_size <= 8) + return 1; + if (bit_size <= 16) + return 2; + if (bit_size <= 32) + return 4; + if (bit_size <= 64)
+ return 8;
+ if (bit_size <= 128) + return 16; + /* BTF supports upto 16byte int (__int128). */ + return -1; +} + int32_t btf_elf__add_base_type(struct btf_elf *btfe, const struct base_type *bt, const char *name) { struct btf *btf = btfe->btf; const struct btf_type *t; uint8_t encoding = 0; - int32_t id; + int32_t id, nbytes;
if (bt->is_signed) { encoding = BTF_INT_SIGNED; @@ -384,7 +403,13 @@ int32_t btf_elf__add_base_type(struct btf_elf *btfe, const struct base_type *b t, return -1; } - id = btf__add_int(btf, name, BITS_ROUNDUP_BYTES(bt->bit_size), encoding); + nbytes = bits_to_int_bytes(bt->bit_size); + if (nbytes < 0) { + fprintf(stderr, "not supported bit_size %hu\n", bt->bit_size); + return -1; + } + + id = btf__add_int(btf, name, nbytes, encoding); if (id < 0) { btf_elf__log_err(btfe, BTF_KIND_INT, name, true, "Error emitting BTF type"); } else { -bash-4.4$ Please help do a test. I can submit a formal patch tomorrow.
| |