Messages in this thread Patch in this message | | | Subject | Re: slub freelist issue / BUG: unable to handle page fault for address: 000000003ffe0018 | From | Vegard Nossum <> | Date | Tue, 23 Mar 2021 19:58:17 +0100 |
| |
(trimmed CCs)
On 2021-03-23 19:32, Kirill A. Shutemov wrote: > On Fri, Jun 12, 2020 at 02:26:58PM +0200, Rafael J. Wysocki wrote: >> On 6/11/2020 3:40 AM, Kaneda, Erik wrote: >>> >>>> -----Original Message----- >>>> From: Vegard Nossum <vegard.nossum@oracle.com> >>>> Sent: Friday, June 5, 2020 7:45 AM >>>> To: Vlastimil Babka <vbabka@suse.cz>; Rafael J. Wysocki >>>> <rafael@kernel.org>; Moore, Robert <robert.moore@intel.com>; Kaneda, >>>> Erik <erik.kaneda@intel.com> >>>> Cc: Kees Cook <keescook@chromium.org>; Wysocki, Rafael J >>>> <rafael.j.wysocki@intel.com>; Christoph Lameter <cl@linux.com>; Andrew >>>> Morton <akpm@linux-foundation.org>; Marco Elver <elver@google.com>; >>>> Waiman Long <longman@redhat.com>; LKML <linux- >>>> kernel@vger.kernel.org>; Linux MM <linux-mm@kvack.org>; ACPI Devel >>>> Maling List <linux-acpi@vger.kernel.org>; Len Brown <lenb@kernel.org>; >>>> Steven Rostedt <rostedt@goodmis.org> >>>> Subject: Re: slub freelist issue / BUG: unable to handle page fault for >>>> address: 000000003ffe0018 >>>> >>>> On 2020-06-05 16:08, Vlastimil Babka wrote: >>>>> On 6/5/20 3:12 PM, Rafael J. Wysocki wrote: >>>>>> On Fri, Jun 5, 2020 at 2:48 PM Vegard Nossum >>>> <vegard.nossum@oracle.com> wrote: >>>>>>> On 2020-06-05 11:36, Vegard Nossum wrote: >>>>>>>> On 2020-06-05 11:11, Vlastimil Babka wrote: >>>>>>>>> On 6/4/20 8:46 PM, Vlastimil Babka wrote: >>>>>>>>>> On 6/4/20 7:57 PM, Kees Cook wrote: >>>>>>>>>>> On Thu, Jun 04, 2020 at 07:20:18PM +0200, Vegard Nossum wrote: >>>>>>>>>>>> On 2020-06-04 19:18, Vlastimil Babka wrote: >>>>>>>>>>>>> On 6/4/20 7:14 PM, Vegard Nossum wrote: >>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I ran into a boot problem with latest linus/master >>>>>>>>>>>>>> (6929f71e46bdddbf1c4d67c2728648176c67c555) that manifests >>>> like this: >>>>>>>>>>>>> Hi, what's the .config you use? >>>>>>>>>>>> Pretty much x86_64 defconfig minus a few options (PCI, USB, >>>>>>>>>>>> ...) >>>>>>>>>>> Oh yes indeed. I immediately crash in the same way with this config. >>>>>>>>>>> I'll >>>>>>>>>>> start digging... >>>>>>>>>>> >>>>>>>>>>> (defconfig finishes boot) >>>>>>>>>> This is funny, booting with slub_debug=F results in: >>>>>>>>>> I'm not sure if it's ACPI or ftrace wrong here, but looks like >>>>>>>>>> the changed free pointer offset merely exposes a bug in something >>>>>>>>>> else. >>>>>>>>> So, with Kees' patch reverted, booting with slub_debug=F (or even >>>>>>>>> more specific slub_debug=F,ftrace_event_field) also hits this bug >>>>>>>>> below. I wanted to bisect it, but v5.7 was also bad, and also >>>>>>>>> v5.6. Didn't try further in history. So it's not new at all, and >>>>>>>>> likely very specific to your config+QEMU? (and related to the ACPI >>>>>>>>> error messages that precede it?). >>>>>>>> I see it too, but not on v5.0. I can bisect it. >>>>>>> commit 67a72420a326b45514deb3f212085fb2cd1595b5 >>>>>>> Author: Bob Moore <robert.moore@intel.com> >>>>>>> Date: Fri Aug 16 14:43:21 2019 -0700 >>>>>>> >>>>>>> ACPICA: Increase total number of possible Owner IDs >>>>>>> >>>>>>> ACPICA commit 1f1652dad88b9d767767bc1f7eb4f7d99e6b5324 >>>>>>> >>>>>>> From 255 to 4095 possible IDs. >>>>>>> >>>>>>> Link: https://github.com/acpica/acpica/commit/1f1652da >>>>>>> Reported-by: Hedi Berriche <hedi.berriche @hpe.com> >>>>>>> Signed-off-by: Bob Moore <robert.moore@intel.com> >>>>>>> Signed-off-by: Erik Schmauss <erik.schmauss@intel.com> >>>>>>> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >>>>>> Bob, Erik, did we miss something in that patch? >>>>> Maybe the patch just changes layout in a way that exposes the bug. >>>>> >>>>> Anyway the "ftrace_event_field" cache is not really involved, this is >>>>> just because of slab merging. After adding "slub_nomerge" to >>>>> "slub_debug=F", it starts making more sense, as the cache becomes >>>>> Acpi-Namespace >>>>> >>>>> [ 0.140408] ------------[ cut here ]------------ >>>>> [ 0.140837] cache_from_obj: Wrong slab cache. Acpi-Namespace but >>>> object is from kmalloc-64 >>>>> [ 0.141406] WARNING: CPU: 0 PID: 1 at mm/slab.h:524 >>>> kmem_cache_free+0x1d3/0x250 >>>>> [ 0.142105] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.7.0+ #45 >>>>> [ 0.142393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >>>> BIOS rel-1.13.0-0-gf21b5a4-rebuilt.opensuse.org 04/01/2014 >>>>> [ 0.142393] RIP: 0010:kmem_cache_free+0x1d3/0x250 >>>>> [ 0.142393] Code: 18 4d 85 ed 0f 84 10 ff ff ff 4c 39 ed 74 2f 49 8b 4d 58 48 >>>> 8b 55 58 48 c7 c6 10 47 a1 ac 48 c7 c7 00 c2 b0 ac e8 b1 cc eb ff <0f> 0b 48 89 de >>>> 4c 89 ef e8 10 d7 ff ff 48 8b 15 59 36 9b 00 4c 89 >>>>> [ 0.142393] RSP: 0018:ffffb39cc0013dc0 EFLAGS: 00010282 >>>>> [ 0.142393] RAX: 0000000000000000 RBX: ffff937287409e00 RCX: >>>> 0000000000000000 >>>>> [ 0.142393] RDX: 0000000000000001 RSI: 0000000000000092 RDI: >>>> ffffffffacfdd32c >>>>> [ 0.142393] RBP: ffff93728742ef00 R08: ffffb39cc0013c7d R09: >>>> 00000000000000fc >>>>> [ 0.142393] R10: ffffb39cc0013c78 R11: ffffb39cc0013c7d R12: >>>> ffff937307409e00 >>>>> [ 0.142393] R13: ffff937287401d00 R14: 0000000000000000 R15: >>>> 0000000000000000 >>>>> [ 0.142393] FS: 0000000000000000(0000) GS:ffff937287a00000(0000) >>>> knlGS:0000000000000000 >>>>> [ 0.142393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>> [ 0.142393] CR2: 0000000000000000 CR3: 0000000003a0a000 CR4: >>>> 00000000003406f0 >>>>> [ 0.142393] Call Trace: >>>>> [ 0.142393] acpi_os_release_object+0x5/0x10 >>>>> [ 0.142393] acpi_ns_delete_children+0x46/0x59 >>>>> [ 0.142393] acpi_ns_delete_namespace_subtree+0x5c/0x79 >>>>> [ 0.142393] ? acpi_sleep_proc_init+0x1f/0x1f >>>>> [ 0.142393] acpi_ns_terminate+0xc/0x31 >>>>> [ 0.142393] acpi_ut_subsystem_shutdown+0x45/0xa3 >>>>> [ 0.142393] ? acpi_sleep_proc_init+0x1f/0x1f >>>>> [ 0.142393] acpi_terminate+0x5/0xf >>>>> [ 0.142393] acpi_init+0x27b/0x308 >>>>> [ 0.142393] ? video_setup+0x79/0x79 >>>>> [ 0.142393] do_one_initcall+0x7b/0x160 >>>>> [ 0.142393] kernel_init_freeable+0x190/0x1f2 >>>>> [ 0.142393] ? rest_init+0x9a/0x9a >>>>> [ 0.142393] kernel_init+0x5/0xf6 >>>>> [ 0.142393] ret_from_fork+0x22/0x30 >>>>> [ 0.142393] ---[ end trace 3539f236ef812ba1 ]--- >>>>> [ 0.142396] ------------[ cut here ]------------ >>>>> >>>>> I've also changed the warning so it's not printed just once, and also >>>>> prints tracking info (see the hunk at the end of my mail, I'll turn this to a >>>> proper patch later). >>>>> With "slub_debug=FU slub_nomerge" there are now multiple warnings, >>>> but they all look the same: >>>>> [ 0.143815] ------------[ cut here ]------------ >>>>> [ 0.144131] cache_from_obj: Wrong slab cache. Acpi-Namespace but >>>> object is from kmalloc-64 >>>>> [ 0.144929] WARNING: CPU: 0 PID: 1 at mm/slab.h:524 >>>> kmem_cache_free+0x1d3/0x250 >>>>> [ 0.145129] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.7.0+ #45 >>>>> [ 0.145129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >>>> BIOS rel-1.13.0-0-gf21b5a4-rebuilt.opensuse.org 04/01/2014 >>>>> [ 0.145129] RIP: 0010:kmem_cache_free+0x1d3/0x250 >>>>> [ 0.145129] Code: 18 4d 85 ed 0f 84 10 ff ff ff 4c 39 ed 74 2f 49 8b 4d 58 48 >>>> 8b 55 58 48 c7 c6 10 47 c1 8d 48 c7 c7 00 c2 d0 8d e8 b1 cc eb ff <0f> 0b 48 89 de >>>> 4c 89 ef e8 10 d7 ff ff 48 8b 15 59 36 9b 00 4c 89 >>>>> [ 0.145129] RSP: 0018:ffff990b80013dc0 EFLAGS: 00010282 >>>>> [ 0.145129] RAX: 0000000000000000 RBX: ffff972d474ada80 RCX: >>>> 0000000000000000 >>>>> [ 0.145129] RDX: 0000000000000001 RSI: 0000000000000092 RDI: >>>> ffffffff8e1dd32c >>>>> [ 0.145129] RBP: ffff972d47425680 R08: ffff990b80013c7d R09: >>>> 00000000000000fc >>>>> [ 0.145129] R10: ffff990b80013c78 R11: ffff990b80013c7d R12: >>>> ffff972dc74ada80 >>>>> [ 0.145129] R13: ffff972d474038c0 R14: 0000000000000000 R15: >>>> 0000000000000000 >>>>> [ 0.145129] FS: 0000000000000000(0000) GS:ffff972d47a00000(0000) >>>> knlGS:0000000000000000 >>>>> [ 0.145129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>> [ 0.145129] CR2: 0000000000000000 CR3: 000000000660a000 CR4: >>>> 00000000003406f0 >>>>> [ 0.145129] Call Trace: >>>>> [ 0.145129] acpi_os_release_object+0x5/0x10 >>>>> [ 0.145129] acpi_ns_delete_children+0x46/0x59 >>>>> [ 0.145129] acpi_ns_delete_namespace_subtree+0x5c/0x79 >>>>> [ 0.145129] ? acpi_sleep_proc_init+0x1f/0x1f >>>>> [ 0.145129] acpi_ns_terminate+0xc/0x31 >>>>> [ 0.145129] acpi_ut_subsystem_shutdown+0x45/0xa3 >>>>> [ 0.145129] ? acpi_sleep_proc_init+0x1f/0x1f >>>>> [ 0.145129] acpi_terminate+0x5/0xf >>>>> [ 0.145129] acpi_init+0x27b/0x308 >>>>> [ 0.145129] ? video_setup+0x79/0x79 >>>>> [ 0.145129] do_one_initcall+0x7b/0x160 >>>>> [ 0.145129] kernel_init_freeable+0x190/0x1f2 >>>>> [ 0.145129] ? rest_init+0x9a/0x9a >>>>> [ 0.145129] kernel_init+0x5/0xf6 >>>>> [ 0.145129] ret_from_fork+0x22/0x30 >>>>> [ 0.145129] ---[ end trace 574554fca7bd06bb ]--- >>>>> [ 0.145133] INFO: Allocated in acpi_ns_root_initialize+0xb6/0x2d1 age=58 >>>> cpu=0 pid=0 >>>>> [ 0.145881] kmem_cache_alloc_trace+0x1a9/0x1c0 >>>>> [ 0.146132] acpi_ns_root_initialize+0xb6/0x2d1 >>>>> [ 0.146578] acpi_initialize_subsystem+0x65/0xa8 >>>>> [ 0.147024] acpi_early_init+0x5d/0xd1 >>>>> [ 0.147132] start_kernel+0x45b/0x518 >>>>> [ 0.147491] secondary_startup_64+0xb6/0xc0 >>>>> [ 0.147897] ------------[ cut here ]------------ >>>>> >>>>> And it seems ACPI is allocating an object via kmalloc() and then >>>>> freeing it via kmem_cache_free(<"Acpi-Namespace" kmem_cache>) which >>>> is wrong. >>>>>> ./scripts/faddr2line vmlinux 'acpi_ns_root_initialize+0xb6' >>>>> acpi_ns_root_initialize+0xb6/0x2d1: >>>>> kmalloc at include/linux/slab.h:555 >>>>> (inlined by) kzalloc at include/linux/slab.h:669 (inlined by) >>>>> acpi_os_allocate_zeroed at include/acpi/platform/aclinuxex.h:57 >>>>> (inlined by) acpi_ns_root_initialize at >>>>> drivers/acpi/acpica/nsaccess.c:102 >>>>> >>> Hi Vegard, >>> >>>> That's it :-) This fixes it for me: >>> We'll take this patch for ACPICA and it will be in the next release. >>> >>> Rafael, do you want to take this as a part of the next rc? >> >> Yes, I do. > > Folks, what happened to the patch? I don't see it in current upstream. > > Looks like it got reported again: > > https://lore.kernel.org/r/a1461e21-c744-767d-6dfc-6641fd3e3ce2@siemens.com >
I've attached a properly formatted patch.
Thanks,
Vegard From a35302316c1c36e22b60d1644db0340558de695f Mon Sep 17 00:00:00 2001 From: Vegard Nossum <vegard.nossum@oracle.com> Date: Mon, 8 Jun 2020 09:57:51 +0200 Subject: [PATCH] ACPICA: Always create namespace nodes using acpi_ns_create_node()
ACPI is allocating an object using kmalloc(), but then frees it using kmem_cache_free(<"Acpi-Namespace" kmem_cache>).
This is wrong and can lead to boot failures manifesting like this:
hpet0: 3 comparators, 64-bit 100.000000 MHz counter clocksource: Switched to clocksource tsc-early BUG: unable to handle page fault for address: 000000003ffe0018 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.6.0+ #211 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 RIP: 0010:kmem_cache_alloc+0x70/0x1d0 Code: 00 00 4c 8b 45 00 65 49 8b 50 08 65 4c 03 05 6f cc e7 7e 4d 8b 20 4d 85 e4 0f 84 3d 01 00 00 8b 45 20 48 8b 7d 00 48 8d 4a 01 <49> 8b 1c 04 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 c5 8b 45 20 RSP: 0000:ffffc90000013df8 EFLAGS: 00010206 RAX: 0000000000000018 RBX: ffffffff81c49200 RCX: 0000000000000002 RDX: 0000000000000001 RSI: 0000000000000dc0 RDI: 000000000002b300 RBP: ffff88803e403d00 R08: ffff88803ec2b300 R09: 0000000000000001 R10: 0000000000000dc0 R11: 0000000000000006 R12: 000000003ffe0000 R13: ffffffff8110a583 R14: 0000000000000dc0 R15: ffffffff81c49a80 FS: 0000000000000000(0000) GS:ffff88803ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000003ffe0018 CR3: 0000000001c0a001 CR4: 00000000003606f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __trace_define_field+0x33/0xa0 event_trace_init+0xeb/0x2b4 tracer_init_tracefs+0x60/0x195 ? register_tracer+0x1e7/0x1e7 do_one_initcall+0x74/0x160 kernel_init_freeable+0x190/0x1f0 ? rest_init+0x9a/0x9a kernel_init+0x5/0xf6 ret_from_fork+0x35/0x40 CR2: 000000003ffe0018 ---[ end trace 707efa023f2ee960 ]--- RIP: 0010:kmem_cache_alloc+0x70/0x1d0
Bisection leads to unrelated changes in slab; Vlastimil Babka suggests an unrelated layout or slab merge change merely exposed the underlying bug.
Link: https://lore.kernel.org/lkml/4dc93ff8-f86e-f4c9-ebeb-6d3153a78d03@oracle.com/ Link: https://lore.kernel.org/r/a1461e21-c744-767d-6dfc-6641fd3e3ce2@siemens.com Debugged-by: Vlastimil Babka <vbabka@suse.cz> Debugged-by: Kees Cook <keescook@chromium.org> Reported-by: Jan Kiszka <jan.kiszka@siemens.com> Fixes: f79c8e4136eac37255ead8875593ae33a2c16d20 ("ACPICA: Namespace: simplify creation of the initial/default namespace") Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com> --- drivers/acpi/acpica/nsaccess.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/acpi/acpica/nsaccess.c b/drivers/acpi/acpica/nsaccess.c index 3f045b5953b2e..1be9b34732ab8 100644 --- a/drivers/acpi/acpica/nsaccess.c +++ b/drivers/acpi/acpica/nsaccess.c @@ -98,14 +98,12 @@ acpi_status acpi_ns_root_initialize(void) * predefined names are at the root level. It is much easier to * just create and link the new node(s) here. */ - new_node = - ACPI_ALLOCATE_ZEROED(sizeof(struct acpi_namespace_node)); + new_node = acpi_ns_create_node(*ACPI_CAST_PTR (u32, init_val->name)); if (!new_node) { status = AE_NO_MEMORY; goto unlock_and_exit; } - ACPI_COPY_NAMESEG(new_node->name.ascii, init_val->name); new_node->descriptor_type = ACPI_DESC_TYPE_NAMED; new_node->type = init_val->type; -- 2.16.1.72.g5be1f00a9.dirty
| |