kernel: more informative process crashes

[nick1udwig]

https://discord.com/channels/1186394868336038080/1186684706700410962/1197287693332267114

[dr-frmr]

One thing we should try adjusting for this is the profile.release for a Rust process..

Currently we have this for all of them:

[profile.release]
panic = "abort"
opt-level = "s"
lto = true

panic = "abort" in particular could be stealing useful debug info.

[nick1udwig]

A good thing to play around with for sure. We set it that way to -- iirc -- reduce binary size. But we might be able to get much more useful info out, and I don't recall ever comparing binary sizes, so it may be an out-of-date optimization.

[zerosnacks]

Using panic instead of abort will yield much better stack tracing results at the cost of an increased binary size

Enabling RUST_BACKTRACE by default could help here as well: paradigmxyz/reth@0520c62

[nick1udwig]

Thanks for the recommendations! It sounds like its probably best to go this route as the default since it'll make dev a lot nicer. We can add a docs page on how to "productionize" if the binary size seems like it might be an issue.

[nick1udwig]

I may be doing something wrong, but using panic = "unwind" gives the same output on panic in my testing:

for panic = "abort"

Tue 1/23 13:25 process chess:chess:sys ended with error: error while executing at wasm backtrace:
    0: 0x3bd5b - panic_abort::__rust_start_panic::abort::ha48e12f63e2dac9a
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/panic_abort/src/lib.rs:96:17              - __rust_start_panic
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/panic_abort/src/lib.rs:38:5              - rust_panic
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/panicking.rs:833:25
    1: 0x38ae1 - std::panicking::rust_panic_with_hook::hbc32d19b84163502
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/panicking.rs:803:5
    2: 0x3c368 - std::panicking::begin_panic_handler::{{closure}}::h1a93a373504e7446
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/panicking.rs:651:13
    3: 0x3c2ce - std::sys_common::backtrace::__rust_end_short_backtrace::hd83d17399e1aaae5
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/sys_common/backtrace.rs:171:18
    4: 0x143b7 - rust_begin_unwind
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/panicking.rs:647:5
    5: 0xd36d - core::panicking::panic_fmt::h54b5ced41b1a7aaa
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/core/src/panicking.rs:72:14
    6: 0x4c54 - <unknown>!init

Caused by:
    wasm trap: wasm `unreachable` instruction executed
Tue 1/23 13:25 skipping OnExit::Restart for process chess:chess:sys due to crash

and for panic = "unwind":

Tue 1/23 13:23 process chess:chess:sys ended with error: error while executing at wasm backtrace:
    0: 0x3c059 - panic_abort::__rust_start_panic::abort::ha48e12f63e2dac9a
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/panic_abort/src/lib.rs:96:17              - __rust_start_panic
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/panic_abort/src/lib.rs:38:5              - rust_panic
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/panicking.rs:833:25
    1: 0x38eaa - std::panicking::rust_panic_with_hook::hbc32d19b84163502
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/panicking.rs:803:5
    2: 0x3c666 - std::panicking::begin_panic_handler::{{closure}}::h1a93a373504e7446
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/panicking.rs:651:13
    3: 0x3c5cc - std::sys_common::backtrace::__rust_end_short_backtrace::hd83d17399e1aaae5
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/sys_common/backtrace.rs:171:18
    4: 0x1455c - rust_begin_unwind
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/std/src/panicking.rs:647:5
    5: 0xd3ec - core::panicking::panic_fmt::h54b5ced41b1a7aaa
                    at /rustc/a2d9d73e608f1b24eba840c4fd2d68dbe3b65e01/library/core/src/panicking.rs:72:14
    6: 0x4d36 - <unknown>!init

Caused by:
    wasm trap: wasm `unreachable` instruction executed
Tue 1/23 13:23 skipping OnExit::Restart for process chess:chess:sys due to crash

As I say, I may be doing something wrong, but if not, then we may indeed have to resort to the method discussed in the link.

EDIT: In addition, wasm binaries are indeed slightly larger with "unwind"; surprising that a larger binary does nothing new?

[zerosnacks]

I think you are simulating a panic inside of a WASM module which is being caught by wasmtime's coredump trap. This is different from a panic inside of a regular Rust program as the WASM module relies on the inclusion of its own unwind table and that depends on how the WASM module was compiled.

bytecodealliance/wasmtime#4431 (comment) states wasmtime uses its own stack walker in favor of system unwind for performance reasons

[nick1udwig]

#204