Messages in this thread | | | Date | Sun, 21 May 2023 15:01:11 -0700 | From | Drew Fustini <> | Subject | Re: riscv: boot failure for 3335068f8721 ("riscv: Use PUD/P4D/PGD pages for the linear mapping") |
| |
On Sat, May 20, 2023 at 10:22:36PM -0500, Samuel Holland wrote: > Hi Drew, > > On 5/20/23 21:05, Drew Fustini wrote: > > Hello, I tested 6.4-rc1 on an internal RISC-V SoC and observed a boot > > failure on a Store/AMO access fault (exception code 7) in __memset(). > > stval (e.g. badaddr) was set to 0xffffaf8000000000. This SoC is RV64GC > > with Sv48 so it seems that address is the start of the "direct mapping > > of all physical memory" [1]. > > > > The 6.3 release boots okay and the system is able to operate correctly > > with an Ubuntu 23.04 rootfs on eMMC. Therefore, I decided to bisect and > > I found the failure begins with 3335068f8721 ("riscv: Use PUD/P4D/PGD > > pages for the linear mapping"). The system boots okay with the prior > > commit 8589e346bbb6 ("riscv: Move the linear mapping creation in its > > own function"). > > > > The boot log [2] shows that the fault happens right after buildroot's > > init script [3] uses switch_root to execute init from the Ubuntu rootfs > > on the eMMC. > > > > DWARF4 is enabled in .config [4] and the decoded stack trace [5] shows: > > > > epc : __memset (/eng/dfustini/gitlab/linux/arch/riscv/lib/memset.S:67) > > > > From memset.S: > > > > Line 67: REG_S a1, 0(t0) > > > > From the oops: > > > > epc : ffffffff81122d6c ra : ffffffff80218504 sp : ffffaf8002e47500 > > gp : ffffffff82695010 tp : ffffaf8002e2ec00 t0 : ffffaf8000000000 > > t1 : 0000000000000080 t2 : 0000000000000001 s0 : ffffaf8002e47550 > > s1 : ffff8d8200000040 a0 : ffffaf8000000000 a1 : 0000000000000000 > > > > Thus I think it is trying to store 0x0 to 0xffffaf8000000000 which is > > the start of the direct map. From the boot log [2], OpenSBI shows: > > > > Domain0 Region00 : 0x0000000002080000-0x00000000020bffff M: (I,R,W) S/U: () > > Domain0 Region01 : 0x0000008000000000-0x000000800003ffff M: (R,W,X) S/U: () > > Domain0 Region02 : 0x0000000002000000-0x000000000207ffff M: (I,R,W) S/U: () > > Domain0 Region03 : 0x0000000000000000-0xffffffffffffffff M: (R,W,X) S/U: (R,W,X) > > > > The DDR memory on this SoC starts at 0x8000000000 with size 2GB. The > > memory node from the device tree [6]: > > > > memory@8000000000 { > > device_type = "memory"; > > reg = <0x80 0 0x00000000 0x80000000>; > > }; > > > > I think the direct map address 0xffffaf8000000000 would map to physical > > address 0x8000000000. Thus I think the attempted store in S-mode to that > > address would violate the PMP settings for Region01. > > > > I do not yet understand why this happens with 3335068f8721 ("riscv: Use > > PUD/P4D/PGD pages for the linear mapping") but not for the prior commit > > 8589e346bbb6 ("riscv: Move the linear mapping creation in its own > > function"). > > Where does Linux's DTB come from? It should be the one that was modified > by OpenSBI to add a reserved-memory node matching PMP Region01 > (fdt_reserved_memory_fixup()). > > Before this commit, Linux ignored the first 2 MiB of physical RAM. So if > OpenSBI was loaded in this region, you could get away with ignoring the > firmware-provided DTB; now you actually need to use it, as intended.
The address of the dtb is passed by the boot code to OpenSBI. I had been using OpenSBI master from Jan 9: 001106d ("docs: Update domain's region permissions and requirements"). The kernel receives the device tree from OpenSBI but I had never actually dumped it from sysfs.
I checked out the prior kernel commit 8589e346bbb6 ("riscv: Move the linear mapping creation in its own function") and ran "dtc -I fs /sys/firmware/devicetree/base/" to dump the device tree [1]. This showed that the reserved-memory node was blank.
Jessica pointed out to me on #riscv irc that this was fixed in OpenSBI on Jan 21 with: a990309 ("lib: utils: Fix reserved memory node for firmware memory"). Therefore, I updated to the current OpenSBI master: 33f1722 ("lib: sbi: Document sbi_ecall_extension members") from May 15. The device tree that OpenSBI passes to the kernel now has "mmode_resv0@80,0" and "mmode_resv1@80,20000".
Furthermore, my system now boots okay with 3335068f8721 ("riscv: Use PUD/P4D/PGD pages for the linear mapping") so the problem was just that I had been using an OpenSBI that was slightly too old.
Thanks, Drew
[1] https://gist.github.com/pdp7/71ca465997274e11953b26861e36144f [2] https://gist.github.com/pdp7/90b4632146fc55625735fa288d80532b
| |