fix(desktop): use WEBKIT_DMABUF_RENDERER_FORCE_SHM for NVIDIA/AppImage (#3654)

[Chessing234]

Summary

• Heuristic and --safe-rendering now set WEBKIT_DMABUF_RENDERER_FORCE_SHM=1 instead of WEBKIT_DISABLE_DMABUF_RENDERER=1
• Legacy DISABLE_DMABUF stays owned so operators can still set =0/=1 and take over the decision
• Linux troubleshooting docs updated to match (WEBKIT_DISABLE_DMABUF_RENDERER crashes WebKitGTK on NVIDIA instead of falling back to shared memory #3654)

Test plan

• unit tests in webkit_rendering::tests
• On NVIDIA + WebKitGTK 2.52: workspace switch no longer SIGSEGVs where the distro NVIDIA guard does not fire (Debian/Ubuntu proprietary-NVIDIA may still crash — WEBKIT_DISABLE_DMABUF_RENDERER crashes WebKitGTK on NVIDIA instead of falling back to shared memory #3654 stays open for that path)
• WEBKIT_DISABLE_DMABUF_RENDERER=0 still stands the heuristic down

[Chessing234]

nvidia/appimage webkit shm force — small linux desktop fix if anyone on the oss team is in that area.

[kwsantiago]

Tested this branch on the hardware from #3654: Ubuntu 24.04 x86_64, WebKitGTK 2.52.3, GeForce RTX 3050 Ti (driver 580.173.02), X11, proprietary NVIDIA driver. Local build, no environment overrides at all.

The heuristic does the right thing:

buzz-desktop: WEBKIT_DMABUF_RENDERER_FORCE_SHM=1 — NVIDIA GPU

It still SIGSEGVs on the first workspace switch, at the same crash site as #3654 (AcceleratedBackingStore::update with a null this).

The reason is downstream of this change. Debian and Ubuntu ship disable-nvidia-dmabuf.patch in webkit2gtk, which inserts an isNVIDIA() check into rendererBufferTransportMode() that returns before mode.add(SharedMemory). On a machine where that check fires, the transport mode is empty no matter which of the two variables is set, so the backing store is still never created.

What works here is this branch plus pinning EGL to Mesa, which makes the distro patch's isNVIDIA() false:

__EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/50_mesa.json

Worth noting the difference from @orestes74's report: Mesa EGL failed on their ARM64 hardware and FORCE_SHM alone was enough, which suggests the distro patch's check does not fire there. So the two cases need different things.

Not asking for a change to this PR. It is the right fix, it is what the module's own docs describe, and it helps every case where the distro patch does not trigger. Flagging only so #3654 is not closed as fully resolved by it: on Debian and Ubuntu with the proprietary driver, users will still hit the crash until the distro patch is fixed. That side is reported to Debian separately.

[wpfleger96]

🤖 Thanks for the fix. I verified the core claim at WebKit source (tag webkitgtk-2.52.5): WEBKIT_DISABLE_DMABUF_RENDERER returns before SharedMemory is added to the transport set (the #3654 null-deref), while WEBKIT_DMABUF_RENDERER_FORCE_SHM is read after it, so it only excludes the hardware dmabuf path. This is the right replacement, and the Plan/user-takeover semantics are preserved — keeping DISABLE_DMABUF in OWNED so =0/=1 still stands the heuristic down is the right call.

Requesting changes for one trivial CI blocker and two doc/UX issues (inline).

One scope note that doesn't need a code change: this is a mitigation, not a universal fix. On machines where WebKit's EGL platform-extension gate fails, the transport set ends up empty regardless of any env var (consistent with the "FORCE_SHM alone wasn't enough" report on #3654), and some distro NVIDIA dmabuf patches can bail out before SharedMemory is added. So I'd frame this as fixing the paths our heuristic covers and not close #3654 from this PR alone.

[orestes74]

Tested this branch (Chessing234:fix/webkit-force-shm-3654) on DGX Spark (NVIDIA GB10, Grace Blackwell, ARM64), Ubuntu 24.04, WebKitGTK 2.52.3, X11, PCI ID 10de:2e12. No environment variable overrides.

Heuristic correctly logs:

buzz-desktop: WEBKIT_DMABUF_RENDERER_FORCE_SHM=1 — NVIDIA GPU

App renders correctly on first frame (previously needed a devtools-open workaround to force a repaint on this build/patch combo — not needed with this PR). Workspace switching repeated several times with no SIGSEGV.

This matches @orestes74's ARM64 report — confirming FORCE_SHM alone is sufficient on this hardware, no Mesa EGL pin needed.

Separately, unrelated observation: I occasionally see the window fail to repaint when raising it from behind other windows (not a workspace switch). Happens rarely (a few times a day), no crash log or coredump captured yet, so not clear if it's the same root cause or something else. Will file separately with more detail if I can pin it down.

[Chessing234]

rustfmt blank-line fix is on tip with dco — needs workflow approval to re-run rust lint.

[Chessing234]

@wpfleger96 addressed the review notes on tip — DISABLE_DMABUF leave warning, FORCE_SHM takeover test, RDNA4 recipe marked unverified, and the ≥2.44 note. rustfmt/dco already fixed earlier. ready for another look / merge when you have a minute (also needs workflow approval on the fork).