This message generated a parse failure. Raw output follows here. Please use 'back' to navigate. From devnull@lkml.org Tue May 21 03:31:44 2024 >From mailfetcher Fri Jan 27 13:20:33 2023 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pi4 X-Spam-Level: X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DMARC_PASS,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,RCVD_IN_ZEN_BLOCKED_OPENDNS, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,URIBL_ZEN_BLOCKED_OPENDNS autolearn=ham au Authentication-Results: pi4.bmw12.nl; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: pi4.bmw12.nl; dkim=pass (Good 1024 bit rsa-sha256 signature) header.d=amd.com header.a=rsa-sha256 Authentication-Results: pi4.bmw12.nl; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=linux-kernel-owner@vger.kernel.org; receiver=) Received: from secure.jasper.es [188.166.10.231] by 1dc7d414a5be with IMAP (fetchmail-6.3.26) for (single-drop); Fri, 27 Jan 2023 13:20:33 +0100 (CET) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by pi4.bmw12.nl (Postfix) with ESMTP id E76823F227 for ; Fri, 27 Jan 2023 13:20:30 +0100 (CET) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233996AbjA0MSi (ORCPT ); Fri, 27 Jan 2023 07:18:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231743AbjA0MSS (ORCPT ); Fri, 27 Jan 2023 07 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2060f.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe59::60f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD3B355B1; Fri, 27 Jan 2023 04:14:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZoGhIw1DCA5dexHKrKZ4mSXpSvUrOwZ1lrFBnOwgA85yNz1+E5k4Zc6H0CXcefoDZN+2WF/7q7z3vu1rsCodGPs6Iu13Tkf0c23CbYyAOx8O8xqQ7CqTjmHIvR9MLW85E23cMGhmSP2NYpGxpjxf4dreK9eA9CpvpqKl2M5kDiO20JVzY1IcuU8RMEwU ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mfyTKzE9H+ZVGFH8WyuS4e59qdz5f8vx9ga+sokDxaw=; b=hDfcpPBQUlkDps+HswOEt6696LpBV8L0sVAvWyFq6bH8cBwqtbesq8m Received: from DM6PR12MB2843.namprd12.prod.outlook.com (2603:10b6:5:48::24) by LV2PR12MB6015.namprd12.prod.outlook.com (2603:10b6:408:14f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.23; Fri, 27 Jan 2 Received: from DM6PR12MB2843.namprd12.prod.outlook.com ([fe80::2867:7b21:95a4:aaf]) by DM6PR12MB2843.namprd12.prod.outlook.com ([fe80::2867:7b21:95a4:aaf%7]) with mapi id 15.20.6043.023; Fri, 27 Jan 2023 12:13:48 +0000 Message-Id: Date: Fri, 27 Jan 2023 23:13:38 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 Subject: Re: [Question PATCH kernel] x86/amd/sev/nmi+vc: Fix stack handling (why is this happening?) Content-Language: en-US To: Peter Zijlstra Cc: kvm@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Sean Christopherson , Jiri Kosina , Ingo Molnar , Dave Hansen < References: <20230127035616.508966-1-aik@amd.com> From: Alexey Kardashevskiy In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: SYBPR01CA0116.ausprd01.prod.outlook.com (2603:10c6:10:1::32) To DM6PR12MB2843.namprd12.prod.outlook.com (2603:10b6:5:48::24) Mime-Version: 1.0 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR12MB2843:EE_|LV2PR12MB6015:EE_ x-ms-office365-filtering-correlation-id: 8c2bf0c3-c85a-48c9-ff50-08db005ff1c8 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xqvxlUPgNEd2Dfvewfr3u7hXSEUZWQk7+Y5XHHpSQ0wASlZhbb7c/za20WRt8X6IPxZFjOptQCaGExtPhYOLWvLmR7cJbi+Cy7SWToXGAnswOQPk6F5hwe8iL7R6p9fHBbcWppvncUlcYDMrRHvbJxPqGx1rf19Th7ED1kAA2R7HTa8aig2kB1aiXqozPXywg2t4qB3eeykjYHB2951M3GQ5xTr3bJQ65LPSVop8ctPzzjONEmRRwOJK5qOYrt X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB2843.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(366004)(346002)(39860400002)(396003)(136003)(451199018)(66899018)(83380400001)(31686004)(36756003)(31600 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?dUZaMHNGTlR5RFo3NHErcUVGclZBenRTbjNGa0ZYT3E5bFFZMGF6bkI2WEJ0?= =?utf-8?B?Nk81QzZCczBHYWFaOUFncjI2YTZqbkhueCtETU4xL3JOVUFyYkE3RDNkWHdK?= =?utf-8?B?eWh6UUVVNUdpWkNUZWhnOXA1VjNleUFVVWdKUlYwZUFGVVFQRFhIclFPZkNQ?= =?utf-8?B?RVMrZVRUeVZiT2JHc1JSN01Pb X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8c2bf0c3-c85a-48c9-ff50-08db005ff1c8 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB2843.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2023 12:13:48.2681 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 6Xe0aYs2UcMwwnSkyM+SMDDyrZOTxDY0xeiUwW7GHdkn3zQHufwu/8nnFRu8UNlicF6Q+mKpqFWzsrZJSMxQxg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6015 Precedence: bulk List-Id: X-Mailing-List: linux-kernel@vger.kernel.org On 27/1/23 20:08, Peter Zijlstra wrote: > On Fri, Jan 27, 2023 at 02:56:16PM +1100, Alexey Kardashevskiy wrote: >> AMD SEV-ES guest kernels compiled without CONFIG_PARAVIRT crash when >> "perf" executes. "./perf record sleep 20" is an example. >> >> Some debugging revealed this happens when CONFIG_PARAVIRT_XXL is not >> defined. The problem seems to be that between DEFINE_IDTENTRY_RAW(exc_nmi) >> and actual reading of DB7 (which in turn causes #VC) every function is >> inlined > > Very much on purpose. > >> and no stack frame is created (?). > > Silly compilers ;-) I think you can force it to by using inline asm with > a rsp dependency or somesuch. > >> Replacing __always_inline with >> noinline in local_db_save() or native_get_debugreg() fixes the problem. > > It will create others. > >> Why is this crash happening and how to fix that? I am still reading >> the assembly but was hoping for a shortcut here :) Thanks, > > Welcome to the wonderful shit show that is x86 exceptions :/ > > I thought sev_es_*() is supposed to fix this. Joerg, any clue? > >> aik-Standard-PC-i440FX-PIIX-1996 login: [ 15.775303] BUG: NMI stack guard page was hit at 0000000003983d50 (stack is 000000007feb1fa4..00000000574369c2) >> [ 15.775314] stack guard page: 0000 [#1] PREEMPT SMP NOPTI >> [ 15.775316] CPU: 0 PID: 790 Comm: sleep Not tainted 6.2.0-rc4_aik-debugswap_ruby-954bhost #73 >> [ 15.775322] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown unknown >> [ 15.775323] RIP: 0010:error_entry+0x17/0x140 >> [ 15.775326] Code: f8 e9 98 fd ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 56 48 8b 74 24 08 48 89 7c 24 08 52 51 50 41 50 41 51 41 52 41 53 53 <55> 41 54 41 55 41 56 41 57 56 31 f6 31 d2 31 c9 45 31 c0 45 31 c9 >> [ 15.775328] RSP: 0000:fffffe2446b2b000 EFLAGS: 00010097 >> [ 15.775332] RAX: fffffe2446b2b0a8 RBX: 0000000000000000 RCX: ffffffffb3a00fed >> [ 15.775333] RDX: 0000000000000000 RSI: ffffffffb3a00b69 RDI: fffffe2446b2b0a8 >> [ 15.775336] RBP: fffffe2446b2b0a8 R08: 0000000000000000 R09: 0000000000000000 >> [ 15.775337] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 >> [ 15.775338] R13: 0000000000000000 R14: 000000000002dd80 R15: 0000000000000000 >> [ 15.775339] FS: 0000000000000000(0000) GS:ffff94b17dc00000(0000) knlGS:ffff94b17dc00000 >> [ 15.775340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 15.775341] CR2: fffffe2446b2aff8 CR3: 00080000167b8000 CR4: 00000000003506f0 >> [ 15.775342] Call Trace: >> [ 15.775352] > > > >> [ 15.775495] ? asm_exc_page_fault+0x22/0x30 >> [ 15.775496] ? restore_regs_and_return_to_kernel+0x22/0x22 >> [ 15.775497] ? exc_page_fault+0x11/0x120 >> [ 15.775499] ? asm_exc_page_fault+0x22/0x30 >> [ 15.775500] ? check_preemption_disabled+0x8/0xe0 >> [ 15.775502] ? __sev_es_ist_enter+0x13/0x100 >> [ 15.775503] ? exc_nmi+0x10e/0x150 >> [ 15.775505] ? end_repeat_nmi+0x16/0x67 >> [ 15.775506] ? asm_exc_double_fault+0x30/0x30 >> [ 15.775507] ? asm_exc_double_fault+0x30/0x30 >> [ 15.775508] ? asm_exc_double_fault+0x30/0x30 >> [ 15.775509] >> [ 15.775509] <#VC> >> [ 15.775510] ? __show_regs.cold+0x18e/0x23d >> [ 15.775511] >> [ 15.775511] <#DF> >> [ 15.775512] ? __die_body.cold+0x1a/0x1f >> [ 15.775513] ? die+0x26/0x40 >> [ 15.775517] ? handle_stack_overflow+0x44/0x60 >> [ 15.775518] ? exc_double_fault+0x14b/0x180 >> [ 15.775519] ? asm_exc_double_fault+0x1f/0x30 >> [ 15.775520] ? restore_regs_and_return_to_kernel+0x22/0x22 >> [ 15.775521] ? asm_exc_page_fault+0x9/0x30 >> [ 15.775522] ? error_entry+0x17/0x140 >> [ 15.775523] >> [ 15.775523] WARNING: stack recursion on stack type 6 >> [ 15.775524] Modules linked in: msr efivarfs >> [ 15.837935] ---[ end trace 0000000000000000 ]--- >> >> Signed-off-by: Alexey Kardashevskiy >> --- >> arch/x86/include/asm/debugreg.h | 29 ++++++++++++++++++++ >> arch/x86/kernel/nmi.c | 2 +- >> 2 files changed, 30 insertions(+), 1 deletion(-) >> >> diff --git a/arch/x86/include/asm/debugreg.h b/arch/x86/include/asm/debugreg.h >> index b049d950612f..396e2ea114dc 100644 >> --- a/arch/x86/include/asm/debugreg.h >> +++ b/arch/x86/include/asm/debugreg.h >> @@ -125,6 +125,35 @@ static __always_inline void local_db_restore(unsigned long dr7) >> set_debugreg(dr7, 7); >> } >> >> +/* noinline here inline __always_inline'd native_get_debugreg(int regno) */ >> +static noinline unsigned long native_get_debugreg7(void) >> +{ >> + unsigned long dr7; >> + asm("mov %%db7, %0" :"=r" (dr7)); >> + return dr7; >> +} >> + >> +static __always_inline unsigned long local_db_save_exc_nmi(void) >> +{ >> + unsigned long dr7; >> + >> + if (static_cpu_has(X86_FEATURE_HYPERVISOR) && !hw_breakpoint_active()) >> + return 0; >> + >> + dr7 = native_get_debugreg7(); >> + dr7 &= ~0x400; /* architecturally set bit */ >> + if (dr7) >> + set_debugreg(0, 7); >> + /* >> + * Ensure the compiler doesn't lower the above statements into >> + * the critical section; disabling breakpoints late would not >> + * be good. >> + */ >> + barrier(); >> + >> + return dr7; >> +} > > This is broken, and building with DEBUG_ENTRY=y would've told you. Huh, good to know. Is this it telling me so? vmlinux.o: warning: objtool: exc_nmi+0x73: call to native_get_debugreg7() leaves .noinstr.text section >> + >> #ifdef CONFIG_CPU_SUP_AMD >> extern void set_dr_addr_mask(unsigned long mask, int dr); >> #else >> diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c >> index cec0bfa3bc04..400b5b6b74f6 100644 >> --- a/arch/x86/kernel/nmi.c >> +++ b/arch/x86/kernel/nmi.c >> @@ -503,7 +503,7 @@ DEFINE_IDTENTRY_RAW(exc_nmi) >> */ >> sev_es_ist_enter(regs); >> >> - this_cpu_write(nmi_dr7, local_db_save()); >> + this_cpu_write(nmi_dr7, local_db_save_exc_nmi()); >> >> irq_state = irqentry_nmi_enter(regs); > > So what I don't get is why sev_es_ist_enter() doesn't cause us to make a > stack frame, that has an actual call in it (although admittedly > conditional). Is not the frame gone when sev_es_ist_enter() (which does not get inlined as per objdump's "ffffffff81bd4550 <__sev_es_ist_enter>: ") returned? -- Alexey