lkml.org 
[lkml]   [2021]   [Feb]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: ERROR: INT DW_ATE_unsigned_1 Error emitting BTF type
    On Sat, Feb 6, 2021 at 9:27 AM Yonghong Song <yhs@fb.com> wrote:
    >
    >
    >
    > On 2/5/21 10:52 PM, Sedat Dilek wrote:
    > > On Sat, Feb 6, 2021 at 7:26 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
    > >>
    > >> On Sat, Feb 6, 2021 at 6:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
    > >>>
    > >>> On Sat, Feb 6, 2021 at 6:44 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
    > >>>>
    > >>>> On Sat, Feb 6, 2021 at 4:34 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
    > >>>>>
    > >>>>> On Fri, Feb 5, 2021 at 10:54 PM Yonghong Song <yhs@fb.com> wrote:
    > >>>>>>
    > >>>>>>
    > >>>>>>
    > >>>>>> On 2/5/21 12:31 PM, Sedat Dilek wrote:
    > >>>>>>> On Fri, Feb 5, 2021 at 9:03 PM Yonghong Song <yhs@fb.com> wrote:
    > >>>>>>>>
    > >>>>>>>>
    > >>>>>>>>
    > >>>>>>>> On 2/5/21 11:24 AM, Arnaldo Carvalho de Melo wrote:
    > >>>>>>>>> Em Fri, Feb 05, 2021 at 11:10:08AM -0800, Yonghong Song escreveu:
    > >>>>>>>>>> On 2/5/21 11:06 AM, Sedat Dilek wrote:
    > >>>>>>>>>>> On Fri, Feb 5, 2021 at 7:53 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
    > >>>>>>>>>>> Grepping through linux.git/tools I guess some BTF tools/libs need to
    > >>>>>>>>>>> know what BTF_INT_UNSIGNED is?
    > >>>>>>>>>
    > >>>>>>>>>> BTF_INT_UNSIGNED needs kernel support. Maybe to teach pahole to
    > >>>>>>>>>> ignore this for now until kernel infrastructure is ready.
    > >>>>>>>>>
    > >>>>>>>>> Yeah, I thought about doing that.
    > >>>>>>>>>
    > >>>>>>>>>> Not sure whether this information will be useful or not
    > >>>>>>>>>> for BTF. This needs to be discussed separately.
    > >>>>>>>>>
    > >>>>>>>>> Maybe search for the rationale for its introduction in DWARF.
    > >>>>>>>>
    > >>>>>>>> In LLVM, we have:
    > >>>>>>>> uint8_t BTFEncoding;
    > >>>>>>>> switch (Encoding) {
    > >>>>>>>> case dwarf::DW_ATE_boolean:
    > >>>>>>>> BTFEncoding = BTF::INT_BOOL;
    > >>>>>>>> break;
    > >>>>>>>> case dwarf::DW_ATE_signed:
    > >>>>>>>> case dwarf::DW_ATE_signed_char:
    > >>>>>>>> BTFEncoding = BTF::INT_SIGNED;
    > >>>>>>>> break;
    > >>>>>>>> case dwarf::DW_ATE_unsigned:
    > >>>>>>>> case dwarf::DW_ATE_unsigned_char:
    > >>>>>>>> BTFEncoding = 0;
    > >>>>>>>> break;
    > >>>>>>>>
    > >>>>>>>> I think DW_ATE_unsigned can be ignored in pahole since
    > >>>>>>>> the default encoding = 0. A simple comment is enough.
    > >>>>>>>>
    > >>>>>>>
    > >>>>>>> Yonghong Son, do you have a patch/diff for me?
    > >>>>>>
    > >>>>>> Looking at error message from log:
    > >>>>>>
    > >>>>>> LLVM_OBJCOPY=/opt/binutils/bin/objcopy /opt/pahole/bin/pahole -J
    > >>>>>> .tmp_vmlinux.btf
    > >>>>>> [115] INT DW_ATE_unsigned_1 Error emitting BTF type
    > >>>>>> Encountered error while encoding BTF.
    > >>>>>>
    > >>>>>> Not exactly what is the root cause. Maybe bt->bit_size is not
    > >>>>>> encoded correctly. Could you put vmlinux (in the above it is
    > >>>>>> .tmp_vmlinux.btf) somewhere, I or somebody else can investigate
    > >>>>>> and provide a proper fix.
    > >>>>>>
    > >>>>>
    > >>>>> [ TO: Masahiro ]
    > >>>>>
    > >>>>> Thanks for taking care Yonghong - hope this is your first name, if not
    > >>>>> I am sorry.
    > >>>>> In case of mixing my first and last name you will make me female -
    > >>>>> Dilek is a Turkish female first name :-).
    > >>>>> So, in some cultures you need to be careful.
    > >>>>>
    > >>>>> Anyway... back to business and facts.
    > >>>>>
    > >>>>> Out of frustration I killed my last build via `make distclean`.
    > >>>>> The whole day I tested diverse combination of GCC-10 and LLVM-12
    > >>>>> together with BTF Kconfigs, selfmade pahole, etc.
    > >>>>>
    > >>>>> I will do ne run with some little changes:
    > >>>>>
    > >>>>> #1: Pass LLVM_IAS=1 to make (means use Clang's Integrated ASsembler -
    > >>>>> as per Nick this leads to the same error - should be unrelated)
    > >>>>> #2: I did: DEBUG_INFO_COMPRESSED y -> n
    > >>>>>
    > >>>>> #2 I did in case you need vmlinux and I have to upload - I will
    > >>>>> compress the resulting vmlinux with ZSTD.
    > >>>>> You need vmlinux or .tmp_vmlinux.btf file?
    > >>>>> Nick was not allowed from his company to download from a Dropbox link.
    > >>>>> So, as an alternative I can offer GoogleDrive...
    > >>>>> ...or bomb into your INBOX :-).
    > >>>>>
    > >>>>> Now, why I CCed Masahiro:
    > >>>>>
    > >>>>> In case of ERRORs when running `scripts/link-vmlinux.sh` above files
    > >>>>> will be removed.
    > >>>>>
    > >>>>> Last, I found a hack to bypass this - means to keep these files (I
    > >>>>> need to check old emails).
    > >>>>>
    > >>>>> Masahiro, you see a possibility to have a way to keep these files in
    > >>>>> case of ERRORs without doing hackery?
    > >>>>>
    > >>>>> From a previous post in this thread:
    > >>>>>
    > >>>>> + info BTF .btf.vmlinux.bin.o
    > >>>>> + [ != silent_ ]
    > >>>>> + printf %-7s %s\n BTF .btf.vmlinux.bin.o
    > >>>>> BTF .btf.vmlinux.bin.o
    > >>>>> + LLVM_OBJCOPY=llvm-objcopy /opt/pahole/bin/pahole -J .tmp_vmlinux.btf
    > >>>>> [2] INT long unsigned int Error emitting BTF type
    > >>>>> Encountered error while encoding BTF.
    > >>>>> + llvm-objcopy --only-section=.BTF --set-section-flags
    > >>>>> .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
    > >>>>> ...
    > >>>>> + info BTFIDS vmlinux
    > >>>>> + [ != silent_ ]
    > >>>>> + printf %-7s %s\n BTFIDS vmlinux
    > >>>>> BTFIDS vmlinux
    > >>>>> + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
    > >>>>> FAILED: load BTF from vmlinux: Invalid argument
    > >>>>> + on_exit
    > >>>>> + [ 255 -ne 0 ]
    > >>>>> + cleanup
    > >>>>> + rm -f .btf.vmlinux.bin.o
    > >>>>> + rm -f .tmp_System.map
    > >>>>> + rm -f .tmp_vmlinux.btf .tmp_vmlinux.kallsyms1
    > >>>>> .tmp_vmlinux.kallsyms1.S .tmp_vmlinux.kallsyms1.o
    > >>>>> .tmp_vmlinux.kallsyms2 .tmp_vmlinux.kallsyms2.S .tmp_vmlinux.kallsyms
    > >>>>> 2.o
    > >>>>> + rm -f System.map
    > >>>>> + rm -f vmlinux
    > >>>>> + rm -f vmlinux.o
    > >>>>> make[3]: *** [Makefile:1166: vmlinux] Error 255
    > >>>>>
    > >>>>> ^^^ Look here.
    > >>>>>
    > >>>>
    > >>>> With this diff:
    > >>>>
    > >>>> $ git diff scripts/link-vmlinux.sh
    > >>>> diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
    > >>>> index eef40fa9485d..40f1b6aae553 100755
    > >>>> --- a/scripts/link-vmlinux.sh
    > >>>> +++ b/scripts/link-vmlinux.sh
    > >>>> @@ -330,7 +330,7 @@ vmlinux_link vmlinux "${kallsymso}" ${btf_vmlinux_bin_o}
    > >>>> # fill in BTF IDs
    > >>>> if [ -n "${CONFIG_DEBUG_INFO_BTF}" -a -n "${CONFIG_BPF}" ]; then
    > >>>> info BTFIDS vmlinux
    > >>>> - ${RESOLVE_BTFIDS} vmlinux
    > >>>> + ##${RESOLVE_BTFIDS} vmlinux
    > >>>> fi
    > >>>>
    > >>>> if [ -n "${CONFIG_BUILDTIME_TABLE_SORT}" ]; then
    > >>>>
    > >>>> This files are kept - not removed:
    > >>>>
    > >>>> $ LC_ALL=C ll .*btf* vmlinux vmlinux.o
    > >>>> -rwxr-xr-x 1 dileks dileks 31M Feb 6 06:37 .btf.vmlinux.bin.o
    > >>>> -rwxr-xr-x 1 dileks dileks 348M Feb 6 06:37 .tmp_vmlinux.btf
    > >>>> -rwxr-xr-x 1 dileks dileks 348M Feb 6 06:37 vmlinux
    > >>>> -rw-r--r-- 1 dileks dileks 344M Feb 6 06:37 vmlinux.o
    > >>>>
    > >>>> Pleas let me know where to upload - Dropbox or GoogleDrive or
    > >>>> elsewhere and give me a link.
    > >>>>
    > >>>
    > >>>
    > >>> WOW, ZSTD is great :-).
    > >>>
    > >>> $ zstd -19 -T0 -v vmlinux
    > >>> *** zstd command line interface 64-bits v1.4.8, by Yann Collet ***
    > >>> Note: 2 physical core(s) detected
    > >>> vmlinux : 22.71% (364466016 => 82784801 bytes, vmlinux.zst)
    > >>>
    > >>> $ du -m vmlinux*
    > >>> 348 vmlinux
    > >>> 79 vmlinux.zst
    > >>>
    > >>
    > >> Dropbox link:
    > >> https://www.dropbox.com/sh/kvyh8ps7na0r1h5/AABfyNfDZ2bESse_bo4h05fFa?dl=0
    > >>
    > >> I hope this is public available.
    > >>
    > >
    > > Inspecting vmlinux with llvm-dwarf:
    > >
    > > $ /opt/llvm-toolchain/bin/llvm-dwarfdump vmlinux | grep DW_AT_name |
    > > grep DW_ATE_ | sort | uniq
    > > DW_AT_name ("DW_ATE_signed_1")
    > > DW_AT_name ("DW_ATE_signed_16")
    > > DW_AT_name ("DW_ATE_signed_32")
    > > DW_AT_name ("DW_ATE_signed_64")
    > > DW_AT_name ("DW_ATE_signed_8")
    > > DW_AT_name ("DW_ATE_unsigned_1")
    > > DW_AT_name ("DW_ATE_unsigned_128")
    > > DW_AT_name ("DW_ATE_unsigned_16")
    > > DW_AT_name ("DW_ATE_unsigned_24")
    > > DW_AT_name ("DW_ATE_unsigned_32")
    > > DW_AT_name ("DW_ATE_unsigned_40")
    > > DW_AT_name ("DW_ATE_unsigned_64")
    > > DW_AT_name ("DW_ATE_unsigned_8")
    > >
    > > - Sedat -
    >
    > Thanks for the above dropbot link, I am able to reproduce the issue.
    >
    > I tried to use latest llvm + Nick's patch + latest pahole + dwarf5
    > config option to compile kernel with LLVM=1 LLVM_IAS=1, somehow
    > I did not hit the issue. It complained like
    >
    > MODPOST vmlinux.symvers
    > WARNING: modpost: vmlinux.o(.text+0x6ce73): Section mismatch in
    > reference from the function __nodes
    > _weight() to the variable .init.data:numa_nodes_parsed
    > The function __nodes_weight() references
    > the variable __initdata numa_nodes_parsed.
    > This is often because __nodes_weight lacks a __initdata
    > annotation or the annotation of numa_nodes_parsed is wrong.
    >
    > but otherwise compilation is fine.
    >
    > With the above vmlinux, the issue appears to be handling
    > DW_ATE_signed_1, DW_ATE_unsigned_{1,24,40}.
    >
    > The following patch should fix the issue:
    >
    > -bash-4.4$ git diff
    >
    > diff --git a/dwarf_loader.c b/dwarf_loader.c
    >
    > index b73d786..0341b5e 100644
    >
    > --- a/dwarf_loader.c
    >
    > +++ b/dwarf_loader.c
    >
    > @@ -467,8 +467,16 @@ static struct base_type *base_type__new(Dwarf_Die
    > *die, struct cu *cu)
    >
    >
    > if (bt != NULL) {
    >
    > tag__init(&bt->tag, cu, die);
    >
    > - bt->name = strings__add(strings, attr_string(die,
    > DW_AT_name));
    > - bt->bit_size = attr_numeric(die, DW_AT_byte_size) * 8;
    >
    > + const char *name = attr_string(die, DW_AT_name);
    >
    > + bt->name = strings__add(strings, name);
    >
    > + /* DW_ATE_unsigned_1 has DW_AT_byte_size == 0.
    >
    > + * specially process it.
    >
    > + */
    >
    > + if (strcmp(name, "DW_ATE_unsigned_1") == 0)
    >
    > + bt->bit_size = 1;
    >
    > + else
    >
    > + bt->bit_size = attr_numeric(die,
    > DW_AT_byte_size) * 8;
    > +
    >
    > uint64_t encoding = attr_numeric(die, DW_AT_encoding);
    > bt->is_bool = encoding == DW_ATE_boolean;
    > bt->is_signed = encoding == DW_ATE_signed;
    > diff --git a/libbtf.c b/libbtf.c
    > index 9f76283..b5aa077 100644
    > --- a/libbtf.c
    > +++ b/libbtf.c
    > @@ -367,13 +367,32 @@ static void btf_log_func_param(const struct
    > btf_elf *btfe,
    > }
    > }
    >
    > +/* btf requires power-of-2 bytes, yet dwarf may have something like
    > + * DW_ATE_unsigned_24 which encodes as 24 bits (3 bytes).
    > + */
    > +static int bits_to_int_bytes(uint16_t bit_size)
    > +{
    > + if (bit_size <= 8)
    > + return 1;
    > + if (bit_size <= 16)
    > + return 2;
    > + if (bit_size <= 32)
    > + return 4;
    > + if (bit_size <= 64)
    >
    > + return 8;
    >
    > + if (bit_size <= 128)
    > + return 16;
    > + /* BTF supports upto 16byte int (__int128). */
    > + return -1;
    > +}
    > +
    > int32_t btf_elf__add_base_type(struct btf_elf *btfe, const struct
    > base_type *bt,
    > const char *name)
    > {
    > struct btf *btf = btfe->btf;
    > const struct btf_type *t;
    > uint8_t encoding = 0;
    > - int32_t id;
    > + int32_t id, nbytes;
    >
    > if (bt->is_signed) {
    > encoding = BTF_INT_SIGNED;
    > @@ -384,7 +403,13 @@ int32_t btf_elf__add_base_type(struct btf_elf
    > *btfe, const struct base_type *b
    > t,
    > return -1;
    > }
    > - id = btf__add_int(btf, name, BITS_ROUNDUP_BYTES(bt->bit_size),
    > encoding);
    > + nbytes = bits_to_int_bytes(bt->bit_size);
    > + if (nbytes < 0) {
    > + fprintf(stderr, "not supported bit_size %hu\n",
    > bt->bit_size);
    > + return -1;
    > + }
    > +
    > + id = btf__add_int(btf, name, nbytes, encoding);
    > if (id < 0) {
    > btf_elf__log_err(btfe, BTF_KIND_INT, name, true, "Error
    > emitting BTF type");
    > } else {
    > -bash-4.4$
    >
    > Please help do a test. I can submit a formal patch tomorrow.

    Thanks for the patch.

    Can you attach the diff as Gmail has totally truncated/malformed it?

    For the Linux breakage - you will need some additional Clang specific patches.
    Is this Linux 5.11-rcX?
    The "Blocking bugs" are listed in the first post of "Linux 5.11 release cycle".
    Hope this helps.

    This is cool co-working :-).

    - Sedat -

    [1] https://github.com/ClangBuiltLinux/linux/issues/1228

    \
     
     \ /
      Last update: 2021-02-06 09:34    [W:4.447 / U:0.176 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site