lkml.org 
[lkml]   [2021]   [Feb]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patches in this message
/
SubjectRe: ERROR: INT DW_ATE_unsigned_1 Error emitting BTF type
From
Date


On 2/5/21 10:52 PM, Sedat Dilek wrote:
> On Sat, Feb 6, 2021 at 7:26 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> On Sat, Feb 6, 2021 at 6:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>
>>> On Sat, Feb 6, 2021 at 6:44 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>
>>>> On Sat, Feb 6, 2021 at 4:34 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>>
>>>>> On Fri, Feb 5, 2021 at 10:54 PM Yonghong Song <yhs@fb.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2/5/21 12:31 PM, Sedat Dilek wrote:
>>>>>>> On Fri, Feb 5, 2021 at 9:03 PM Yonghong Song <yhs@fb.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2/5/21 11:24 AM, Arnaldo Carvalho de Melo wrote:
>>>>>>>>> Em Fri, Feb 05, 2021 at 11:10:08AM -0800, Yonghong Song escreveu:
>>>>>>>>>> On 2/5/21 11:06 AM, Sedat Dilek wrote:
>>>>>>>>>>> On Fri, Feb 5, 2021 at 7:53 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>>>>>>>> Grepping through linux.git/tools I guess some BTF tools/libs need to
>>>>>>>>>>> know what BTF_INT_UNSIGNED is?
>>>>>>>>>
>>>>>>>>>> BTF_INT_UNSIGNED needs kernel support. Maybe to teach pahole to
>>>>>>>>>> ignore this for now until kernel infrastructure is ready.
>>>>>>>>>
>>>>>>>>> Yeah, I thought about doing that.
>>>>>>>>>
>>>>>>>>>> Not sure whether this information will be useful or not
>>>>>>>>>> for BTF. This needs to be discussed separately.
>>>>>>>>>
>>>>>>>>> Maybe search for the rationale for its introduction in DWARF.
>>>>>>>>
>>>>>>>> In LLVM, we have:
>>>>>>>> uint8_t BTFEncoding;
>>>>>>>> switch (Encoding) {
>>>>>>>> case dwarf::DW_ATE_boolean:
>>>>>>>> BTFEncoding = BTF::INT_BOOL;
>>>>>>>> break;
>>>>>>>> case dwarf::DW_ATE_signed:
>>>>>>>> case dwarf::DW_ATE_signed_char:
>>>>>>>> BTFEncoding = BTF::INT_SIGNED;
>>>>>>>> break;
>>>>>>>> case dwarf::DW_ATE_unsigned:
>>>>>>>> case dwarf::DW_ATE_unsigned_char:
>>>>>>>> BTFEncoding = 0;
>>>>>>>> break;
>>>>>>>>
>>>>>>>> I think DW_ATE_unsigned can be ignored in pahole since
>>>>>>>> the default encoding = 0. A simple comment is enough.
>>>>>>>>
>>>>>>>
>>>>>>> Yonghong Son, do you have a patch/diff for me?
>>>>>>
>>>>>> Looking at error message from log:
>>>>>>
>>>>>> LLVM_OBJCOPY=/opt/binutils/bin/objcopy /opt/pahole/bin/pahole -J
>>>>>> .tmp_vmlinux.btf
>>>>>> [115] INT DW_ATE_unsigned_1 Error emitting BTF type
>>>>>> Encountered error while encoding BTF.
>>>>>>
>>>>>> Not exactly what is the root cause. Maybe bt->bit_size is not
>>>>>> encoded correctly. Could you put vmlinux (in the above it is
>>>>>> .tmp_vmlinux.btf) somewhere, I or somebody else can investigate
>>>>>> and provide a proper fix.
>>>>>>
>>>>>
>>>>> [ TO: Masahiro ]
>>>>>
>>>>> Thanks for taking care Yonghong - hope this is your first name, if not
>>>>> I am sorry.
>>>>> In case of mixing my first and last name you will make me female -
>>>>> Dilek is a Turkish female first name :-).
>>>>> So, in some cultures you need to be careful.
>>>>>
>>>>> Anyway... back to business and facts.
>>>>>
>>>>> Out of frustration I killed my last build via `make distclean`.
>>>>> The whole day I tested diverse combination of GCC-10 and LLVM-12
>>>>> together with BTF Kconfigs, selfmade pahole, etc.
>>>>>
>>>>> I will do ne run with some little changes:
>>>>>
>>>>> #1: Pass LLVM_IAS=1 to make (means use Clang's Integrated ASsembler -
>>>>> as per Nick this leads to the same error - should be unrelated)
>>>>> #2: I did: DEBUG_INFO_COMPRESSED y -> n
>>>>>
>>>>> #2 I did in case you need vmlinux and I have to upload - I will
>>>>> compress the resulting vmlinux with ZSTD.
>>>>> You need vmlinux or .tmp_vmlinux.btf file?
>>>>> Nick was not allowed from his company to download from a Dropbox link.
>>>>> So, as an alternative I can offer GoogleDrive...
>>>>> ...or bomb into your INBOX :-).
>>>>>
>>>>> Now, why I CCed Masahiro:
>>>>>
>>>>> In case of ERRORs when running `scripts/link-vmlinux.sh` above files
>>>>> will be removed.
>>>>>
>>>>> Last, I found a hack to bypass this - means to keep these files (I
>>>>> need to check old emails).
>>>>>
>>>>> Masahiro, you see a possibility to have a way to keep these files in
>>>>> case of ERRORs without doing hackery?
>>>>>
>>>>> From a previous post in this thread:
>>>>>
>>>>> + info BTF .btf.vmlinux.bin.o
>>>>> + [ != silent_ ]
>>>>> + printf %-7s %s\n BTF .btf.vmlinux.bin.o
>>>>> BTF .btf.vmlinux.bin.o
>>>>> + LLVM_OBJCOPY=llvm-objcopy /opt/pahole/bin/pahole -J .tmp_vmlinux.btf
>>>>> [2] INT long unsigned int Error emitting BTF type
>>>>> Encountered error while encoding BTF.
>>>>> + llvm-objcopy --only-section=.BTF --set-section-flags
>>>>> .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
>>>>> ...
>>>>> + info BTFIDS vmlinux
>>>>> + [ != silent_ ]
>>>>> + printf %-7s %s\n BTFIDS vmlinux
>>>>> BTFIDS vmlinux
>>>>> + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
>>>>> FAILED: load BTF from vmlinux: Invalid argument
>>>>> + on_exit
>>>>> + [ 255 -ne 0 ]
>>>>> + cleanup
>>>>> + rm -f .btf.vmlinux.bin.o
>>>>> + rm -f .tmp_System.map
>>>>> + rm -f .tmp_vmlinux.btf .tmp_vmlinux.kallsyms1
>>>>> .tmp_vmlinux.kallsyms1.S .tmp_vmlinux.kallsyms1.o
>>>>> .tmp_vmlinux.kallsyms2 .tmp_vmlinux.kallsyms2.S .tmp_vmlinux.kallsyms
>>>>> 2.o
>>>>> + rm -f System.map
>>>>> + rm -f vmlinux
>>>>> + rm -f vmlinux.o
>>>>> make[3]: *** [Makefile:1166: vmlinux] Error 255
>>>>>
>>>>> ^^^ Look here.
>>>>>
>>>>
>>>> With this diff:
>>>>
>>>> $ git diff scripts/link-vmlinux.sh
>>>> diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
>>>> index eef40fa9485d..40f1b6aae553 100755
>>>> --- a/scripts/link-vmlinux.sh
>>>> +++ b/scripts/link-vmlinux.sh
>>>> @@ -330,7 +330,7 @@ vmlinux_link vmlinux "${kallsymso}" ${btf_vmlinux_bin_o}
>>>> # fill in BTF IDs
>>>> if [ -n "${CONFIG_DEBUG_INFO_BTF}" -a -n "${CONFIG_BPF}" ]; then
>>>> info BTFIDS vmlinux
>>>> - ${RESOLVE_BTFIDS} vmlinux
>>>> + ##${RESOLVE_BTFIDS} vmlinux
>>>> fi
>>>>
>>>> if [ -n "${CONFIG_BUILDTIME_TABLE_SORT}" ]; then
>>>>
>>>> This files are kept - not removed:
>>>>
>>>> $ LC_ALL=C ll .*btf* vmlinux vmlinux.o
>>>> -rwxr-xr-x 1 dileks dileks 31M Feb 6 06:37 .btf.vmlinux.bin.o
>>>> -rwxr-xr-x 1 dileks dileks 348M Feb 6 06:37 .tmp_vmlinux.btf
>>>> -rwxr-xr-x 1 dileks dileks 348M Feb 6 06:37 vmlinux
>>>> -rw-r--r-- 1 dileks dileks 344M Feb 6 06:37 vmlinux.o
>>>>
>>>> Pleas let me know where to upload - Dropbox or GoogleDrive or
>>>> elsewhere and give me a link.
>>>>
>>>
>>>
>>> WOW, ZSTD is great :-).
>>>
>>> $ zstd -19 -T0 -v vmlinux
>>> *** zstd command line interface 64-bits v1.4.8, by Yann Collet ***
>>> Note: 2 physical core(s) detected
>>> vmlinux : 22.71% (364466016 => 82784801 bytes, vmlinux.zst)
>>>
>>> $ du -m vmlinux*
>>> 348 vmlinux
>>> 79 vmlinux.zst
>>>
>>
>> Dropbox link:
>> https://www.dropbox.com/sh/kvyh8ps7na0r1h5/AABfyNfDZ2bESse_bo4h05fFa?dl=0
>>
>> I hope this is public available.
>>
>
> Inspecting vmlinux with llvm-dwarf:
>
> $ /opt/llvm-toolchain/bin/llvm-dwarfdump vmlinux | grep DW_AT_name |
> grep DW_ATE_ | sort | uniq
> DW_AT_name ("DW_ATE_signed_1")
> DW_AT_name ("DW_ATE_signed_16")
> DW_AT_name ("DW_ATE_signed_32")
> DW_AT_name ("DW_ATE_signed_64")
> DW_AT_name ("DW_ATE_signed_8")
> DW_AT_name ("DW_ATE_unsigned_1")
> DW_AT_name ("DW_ATE_unsigned_128")
> DW_AT_name ("DW_ATE_unsigned_16")
> DW_AT_name ("DW_ATE_unsigned_24")
> DW_AT_name ("DW_ATE_unsigned_32")
> DW_AT_name ("DW_ATE_unsigned_40")
> DW_AT_name ("DW_ATE_unsigned_64")
> DW_AT_name ("DW_ATE_unsigned_8")
>
> - Sedat -

Thanks for the above dropbot link, I am able to reproduce the issue.

I tried to use latest llvm + Nick's patch + latest pahole + dwarf5
config option to compile kernel with LLVM=1 LLVM_IAS=1, somehow
I did not hit the issue. It complained like

MODPOST vmlinux.symvers
WARNING: modpost: vmlinux.o(.text+0x6ce73): Section mismatch in
reference from the function __nodes
_weight() to the variable .init.data:numa_nodes_parsed
The function __nodes_weight() references
the variable __initdata numa_nodes_parsed.
This is often because __nodes_weight lacks a __initdata
annotation or the annotation of numa_nodes_parsed is wrong.

but otherwise compilation is fine.

With the above vmlinux, the issue appears to be handling
DW_ATE_signed_1, DW_ATE_unsigned_{1,24,40}.

The following patch should fix the issue:

-bash-4.4$ git diff

diff --git a/dwarf_loader.c b/dwarf_loader.c

index b73d786..0341b5e 100644

--- a/dwarf_loader.c

+++ b/dwarf_loader.c

@@ -467,8 +467,16 @@ static struct base_type *base_type__new(Dwarf_Die
*die, struct cu *cu)


if (bt != NULL) {

tag__init(&bt->tag, cu, die);

- bt->name = strings__add(strings, attr_string(die,
DW_AT_name));
- bt->bit_size = attr_numeric(die, DW_AT_byte_size) * 8;
+ const char *name = attr_string(die, DW_AT_name);
+ bt->name = strings__add(strings, name);
+ /* DW_ATE_unsigned_1 has DW_AT_byte_size == 0.
+ * specially process it.

+ */

+ if (strcmp(name, "DW_ATE_unsigned_1") == 0)
+ bt->bit_size = 1;
+ else

+ bt->bit_size = attr_numeric(die,
DW_AT_byte_size) * 8;
+

uint64_t encoding = attr_numeric(die, DW_AT_encoding);
bt->is_bool = encoding == DW_ATE_boolean;
bt->is_signed = encoding == DW_ATE_signed;
diff --git a/libbtf.c b/libbtf.c
index 9f76283..b5aa077 100644
--- a/libbtf.c
+++ b/libbtf.c
@@ -367,13 +367,32 @@ static void btf_log_func_param(const struct
btf_elf *btfe,
}
}

+/* btf requires power-of-2 bytes, yet dwarf may have something like
+ * DW_ATE_unsigned_24 which encodes as 24 bits (3 bytes).
+ */
+static int bits_to_int_bytes(uint16_t bit_size)
+{
+ if (bit_size <= 8)
+ return 1;
+ if (bit_size <= 16)
+ return 2;
+ if (bit_size <= 32)
+ return 4;
+ if (bit_size <= 64)

+ return 8;

+ if (bit_size <= 128)
+ return 16;
+ /* BTF supports upto 16byte int (__int128). */
+ return -1;
+}
+
int32_t btf_elf__add_base_type(struct btf_elf *btfe, const struct
base_type *bt,
const char *name)
{
struct btf *btf = btfe->btf;
const struct btf_type *t;
uint8_t encoding = 0;
- int32_t id;
+ int32_t id, nbytes;

if (bt->is_signed) {
encoding = BTF_INT_SIGNED;
@@ -384,7 +403,13 @@ int32_t btf_elf__add_base_type(struct btf_elf
*btfe, const struct base_type *b
t,
return -1;
}
- id = btf__add_int(btf, name, BITS_ROUNDUP_BYTES(bt->bit_size),
encoding);
+ nbytes = bits_to_int_bytes(bt->bit_size);
+ if (nbytes < 0) {
+ fprintf(stderr, "not supported bit_size %hu\n",
bt->bit_size);
+ return -1;
+ }
+
+ id = btf__add_int(btf, name, nbytes, encoding);
if (id < 0) {
btf_elf__log_err(btfe, BTF_KIND_INT, name, true, "Error
emitting BTF type");
} else {
-bash-4.4$
Please help do a test. I can submit a formal patch tomorrow.

\
 
 \ /
  Last update: 2021-02-06 09:29    [W:0.612 / U:0.500 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site