Dasharo v0.9.0, apu4, apu6: Call Trace at boot time

[532910]

Component

Dasharo firmware

Device

PC Engines APU4

Dasharo version

v0.9.0

Dasharo Tools Suite version

No response

Brief summary

Call Trace at boot time

How reproducible

every time

How to reproduce

try to boot

Expected behavior

no call trace

Actual behavior

call trace during boot

Screenshots

No response

Additional context

[   23.149358] ------------[ cut here ]------------
[   23.153996] WARNING: CPU: 2 PID: 1 at 0xffffffff815268e2
[   23.159322] Modules linked in:
[   23.162400] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.15.150 #0
[   23.168511] Hardware name: PC Engines apu4/apu4, BIOS Dasharo (coreboot+UEFI) v0.9.0 03/28/2024
[   23.177221] RIP: 0010:0xffffffff815268e2
[   23.181164] Code: c7 c6 60 4c 52 81 8b b8 b4 03 00 00 e8 17 75 be ff 41 89 c4 85 c0 0f 84 30 fe ff ff 48 8b 7b 18 e8 63 3f f1 ff e9 83 fd ff ff <0f> 0b 48 8b 1b 48 81 fb f0 d5 4d 82 0f 85 3b fd ff ff eb 97 8b 7d
[   23.199935] RSP: 0018:ffffc9000002fd98 EFLAGS: 00010246
[   23.205179] RAX: 0000000f08ab75f5 RBX: ffff88810004a000 RCX: 0000000000000002
[   23.212332] RDX: 000000000000270f RSI: 0000000000000002 RDI: 0000000f08ab4ee6
[   23.219477] RBP: ffffc9000002fe18 R08: 0000000000000002 R09: 00000000000026fd
[   23.226624] R10: 0000000000000000 R11: ffffffff824ae548 R12: 00000000001e8480
[   23.233774] R13: 0000000080000000 R14: 000ffffffffffff8 R15: 0800000000000000
[   23.240925] FS:  0000000000000000(0000) GS:ffff88812ad00000(0000) knlGS:0000000000000000
[   23.249027] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   23.254785] CR2: 0000000000000000 CR3: 000000000540a000 CR4: 00000000000406e0
[   23.261937] Call Trace:
[   23.264402]  <TASK>
[   23.266516]  ? 0xffffffff81a88afd
[   23.269842]  ? 0xffffffff81a88c27
[   23.273182]  ? 0xffffffff810bd8d4
[   23.276507]  ? 0xffffffff815268e2
[   23.279837]  ? 0xffffffff813f6f56
[   23.283173]  ? 0xffffffff81ad8b28
[   23.286502]  ? 0xffffffff81ad8c38
[   23.289830]  ? 0xffffffff81c00b4b
[   23.293167]  ? 0xffffffff815268e2
[   23.296495]  ? 0xffffffff81526863
[   23.299825]  ? 0xffffffff81109add
[   23.303159]  ? 0xffffffff81a8e777
[   23.306488]  ? 0xffffffff81526375
[   23.309819]  0xffffffff82595e81
[   23.312979]  ? 0xffffffff825676be
[   23.316311]  0xffffffff82596198
[   23.319464]  0xffffffff825676d3
[   23.322627]  0xffffffff810030af
[   23.325786]  0xffffffff825593cc
[   23.328939]  ? 0xffffffff81adcd50
[   23.332272]  0xffffffff81adcd65
[   23.335430]  0xffffffff81003c32
[   23.338588]  </TASK>
[   23.340803] ---[ end trace 4829f5ca0413b3cc ]---

[532910]

debian stable:

[    4.906918] Freeing initrd memory: 54952K
[   25.217673] ------------[ cut here ]------------
[   25.222315] WARNING: CPU: 1 PID: 1 at drivers/iommu/amd/init.c:958 enable_iommus_vapic+0x30e/0x370
[   25.231330] Modules linked in:
[   25.234414] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 6.1.0-20-amd64 #1  Debian 6.1.85-1
[   25.242524] Hardware name: PC Engines apu6/apu6, BIOS Dasharo (coreboot+UEFI) v0.9.0 03/28/2024
[   25.251238] RIP: 0010:enable_iommus_vapic+0x30e/0x370
[   25.256317] Code: 44 24 08 65 48 2b 04 25 28 00 00 00 75 68 49 8b bf a0 00 00 00 48 83 c4 10 31 f6 5b 5d 41 5c 41 5d 41 5e 41 5f e9 72 d0 c2 ff <0f> 0b e9 a1 fd ff ff 48 8b 15 bc 6d 9c 02 48 c1 ea 05 83 e2 07 48
[   25.275088] RSP: 0018:ffffba470002fd70 EFLAGS: 00010246
[   25.280345] RAX: 0000000d23688e93 RBX: 00000000001e8480 RCX: 0000000000000001
[   25.287497] RDX: 00000000000026d6 RSI: 0000000000000001 RDI: 0000000d236867bd
[   25.294648] RBP: 0000000080000000 R08: 0000000000000001 R09: 00000000000026ad
[   25.301800] R10: ffff930aaacb78d8 R11: ffffffff90ed4448 R12: 000ffffffffffff8
[   25.308948] R13: 0800000000000000 R14: 2000000000000000 R15: ffff930a8005a000
[   25.316103] FS:  0000000000000000(0000) GS:ffff930aaac80000(0000) knlGS:0000000000000000
[   25.324205] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   25.329969] CR2: 0000000000000000 CR3: 0000000044010000 CR4: 00000000000406e0
[   25.337124] Call Trace:
[   25.339595]  <TASK>
[   25.341709]  ? __warn+0x7d/0xc0
[   25.344879]  ? enable_iommus_vapic+0x30e/0x370
[   25.349349]  ? report_bug+0xe2/0x150
[   25.352958]  ? handle_bug+0x41/0x70
[   25.356477]  ? exc_invalid_op+0x13/0x60
[   25.360340]  ? asm_exc_invalid_op+0x16/0x20
[   25.364558]  ? enable_iommus_vapic+0x30e/0x370
[   25.369028]  iommu_go_to_state+0xe23/0x14af
[   25.373252]  ? iommu_setup+0x279/0x279
[   25.377026]  amd_iommu_init+0x10/0x46
[   25.380718]  ? iommu_setup+0x279/0x279
[   25.384496]  pci_iommu_init+0xe/0x36
[   25.388095]  do_one_initcall+0x59/0x220
[   25.391965]  kernel_init_freeable+0x221/0x286
[   25.396360]  ? rest_init+0xd0/0xd0
[   25.399812]  kernel_init+0x16/0x130
[   25.403319]  ret_from_fork+0x22/0x30
[   25.406929]  </TASK>
[   25.409153] ---[ end trace 0000000000000000 ]---

bpo kernel:

[    5.040290] Freeing initrd memory: 60436K
[   25.279455] ------------[ cut here ]------------
[   25.284093] WARNING: CPU: 2 PID: 1 at drivers/iommu/amd/init.c:995 amd_iommu_enable_interrupts+0x59d/0x640
[   25.293787] Modules linked in:
[   25.296866] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.6.13+bpo-amd64 #1  Debian 6.6.13-1~bpo12+1
[   25.305843] Hardware name: PC Engines apu6/apu6, BIOS Dasharo (coreboot+UEFI) v0.9.0 03/28/2024
[   25.314553] RIP: 0010:amd_iommu_enable_interrupts+0x59d/0x640
[   25.320330] Code: 0f 85 62 fe ff ff 49 8b 87 b8 00 00 00 31 f6 48 89 c7 e8 b6 a0 b0 ff 49 8b bf d0 00 00 00 31 f6 e8 a8 a0 b0 ff e9 0b fc ff ff <0f> 0b e9 04 fc ff ff 48 8b 7b 18 89 44 24 04 e8 4f f9 ea ff 8b 44
[   25.339095] RSP: 0018:ffffa577c002fd78 EFLAGS: 00010246
[   25.344340] RAX: 0000000d4b675635 RBX: 00000000001e8480 RCX: 0000000000000002
[   25.351487] RDX: 00000000000026d6 RSI: 0000000000000002 RDI: 0000000d4b672f5f
[   25.358636] RBP: 0000000080000000 R08: 0000000000000002 R09: 00000000000026ae
[   25.365787] R10: 0000000000000000 R11: 0000000000000100 R12: 000ffffffffffff8
[   25.372934] R13: 0800000000000000 R14: 2000000000000000 R15: ffff98d94005a800
[   25.380087] FS:  0000000000000000(0000) GS:ffff98d96ad00000(0000) knlGS:0000000000000000
[   25.388186] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   25.393951] CR2: 0000000000000000 CR3: 0000000038a20000 CR4: 00000000000406e0
[   25.401102] Call Trace:
[   25.403567]  <TASK>
[   25.405687]  ? amd_iommu_enable_interrupts+0x59d/0x640
[   25.410850]  ? __warn+0x81/0x130
[   25.414115]  ? amd_iommu_enable_interrupts+0x59d/0x640
[   25.419276]  ? report_bug+0x171/0x1a0
[   25.422967]  ? handle_bug+0x41/0x70
[   25.426485]  ? exc_invalid_op+0x17/0x70
[   25.430342]  ? asm_exc_invalid_op+0x1a/0x20
[   25.434558]  ? amd_iommu_enable_interrupts+0x59d/0x640
[   25.439720]  ? amd_iommu_enable_interrupts+0x4d1/0x640
[   25.444893]  state_next+0xa8f/0x1910
[   25.448505]  amd_iommu_init+0x21/0x80
[   25.452192]  ? __pfx_pci_iommu_init+0x10/0x10
[   25.456574]  pci_iommu_init+0x12/0x40
[   25.460259]  do_one_initcall+0x5d/0x320
[   25.464129]  kernel_init_freeable+0x320/0x470
[   25.468515]  ? __pfx_kernel_init+0x10/0x10
[   25.472644]  kernel_init+0x1a/0x1c0
[   25.476163]  ret_from_fork+0x34/0x50
[   25.479768]  ? __pfx_kernel_init+0x10/0x10
[   25.483892]  ret_from_fork_asm+0x1b/0x30
[   25.487850]  </TASK>
[   25.490043] ---[ end trace 0000000000000000 ]---

[macpijan]

@532910 The first trace was from OpenWRT, right? Can you specify which version?

@PLangowski IIRC, you have tested booting with OpenWRT, can you provide more details?

[532910]

OpenWRT is the latest stable of course: 23.05.3, exactly this image:
https://downloads.openwrt.org/releases/23.05.3/targets/x86/64/openwrt-23.05.3-x86-64-generic-squashfs-combined-efi.img.gz

Moreover, while debian boots and looks like works fine, openwrt doesn't boot at all. Sounds like another issue, I'll provide more details a bit later.

[macpijan]

Any more details here?

[miczyg1]

First strace hit an undefined instruction right after a jump, so I have no idea why it happened:

0:  c7 c6 60 4c 52 81       mov    esi,0x81524c60
6:  8b b8 b4 03 00 00       mov    edi,DWORD PTR [rax+0x3b4]
c:  e8 17 75 be ff          call   0xffffffffffbe7528
11: 41 89 c4                mov    r12d,eax
14: 85 c0                   test   eax,eax
16: 0f 84 30 fe ff ff       je     0xfffffffffffffe4c
1c: 48 8b 7b 18             mov    rdi,QWORD PTR [rbx+0x18]
20: e8 63 3f f1 ff          call   0xfffffffffff13f88
25: e9 83 fd ff ff          jmp    0xfffffffffffffdad
2a: 0f 0b                   ud2 <------------------------------------------------
2c: 48 8b 1b                mov    rbx,QWORD PTR [rbx]
2f: 48 81 fb f0 d5 4d 82    cmp    rbx,0xffffffff824dd5f0
36: 0f 85 3b fd ff ff       jne    0xfffffffffffffd77
3c: eb 97                   jmp    0xffffffffffffffd5
3e: 8b                      .byte 0x8b
3f: 7d                      .byte 0x7d

The other traces at least say that it is something related to IOMMU.

[ma-neumann]

I guess issue #1134 is related, as I experienced the same segfault on an APU2 running Ubuntu. Please see this answer more specifically.

Adding amd_iommu_intr=legacy [1] to the kernel options in GRUB (i.e. in /etc/boot/grub.cfg on OpenWRT) disables using the vAPIC by the IOMMU which (temporarily) fixes this segfault.

I hope this issue is properly fixed when the fixes for disabling the vAPIC on APU of @miczyg1 get released: Dasharo/coreboot#592

[1] https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html