Skip to content

Fix tray crash-loop: source x64 VC++ runtime from VS install at publish time - #713

Merged
shanselman merged 4 commits into
masterfrom
copilot/fix-tray-crash-loops
Jun 7, 2026
Merged

Fix tray crash-loop: source x64 VC++ runtime from VS install at publish time#713
shanselman merged 4 commits into
masterfrom
copilot/fix-tray-crash-loops

Conversation

Copilot AI commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Bundled msvcp140.dll/vcruntime140.dll from VCRuntime.CefSharp.140 NuGet (v1.0.5) are 14.29 — too old for onnxruntime 1.26. App-local DLL search order shadows the system runtime, so sherpa-onnx-c-api fails init (0x8007045A), the OfflineTts finalizer calls into the dead DLL, and the tray crash-loops on startup.

Changes

  • src/Directory.Build.targets — Generalized ResolveOpenClawVCRuntimeArm64FromVSInstallResolveOpenClawVCRuntimeFromVSInstall. x64 publish now resolves DLLs from VS install (same as ARM64 already did), replacing the stale NuGet items. NuGet retained only for local dotnet build dev convenience. Legacy alias target kept for backward compat.
<!-- At publish time, replace stale NuGet DLLs with current VS install DLLs -->
<ItemGroup Condition="'$(OpenClawVCRuntimeArch)' == 'x64'">
  <OpenClawVCRuntimeFiles Remove="@(OpenClawVCRuntimeFiles)" />
  <OpenClawVCRuntimeFiles Include="$(_OpenClawVsInstallRoot)\VC\Redist\MSVC\*\x64\Microsoft.VC*.CRT\vcruntime140*.dll" />
  <OpenClawVCRuntimeFiles Include="$(_OpenClawVsInstallRoot)\VC\Redist\MSVC\*\x64\Microsoft.VC*.CRT\msvcp140*.dll" />
</ItemGroup>
  • scripts/Test-ReleaseNativeDependencies.ps1 — Added Add-VCRuntimeVersionFloorErrors with a minimum version floor of 14.38.33130.0 (VS 2022 17.8, first CRT compatible with onnxruntime ≥ 1.20). Prevents silent regression to stale DLLs.

  • tests/OpenClaw.Tray.Tests/ReleaseSigningWorkflowTests.cs — Assertions updated for new target name + version floor presence.

  • docs/RELEASING.md — Updated to reflect both architectures now source from VS install.

@clawsweeper

clawsweeper Bot commented Jun 7, 2026

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge. Reviewed June 7, 2026, 6:19 PM ET / 22:19 UTC.

Summary
The PR changes release MSBuild targets to source x64 VC++ runtime DLLs from Visual Studio at publish time, expands the native dependency verifier, and updates release docs/tests.

Reproducibility: yes. The linked report at #703 plus current source give a high-confidence Windows x64 path: publish with the app-local 14.29 VC runtime, launch tray/TTS, and observe the Sherpa/ONNX initialization crash; I did not execute that Windows repro here.

Review metrics: 2 noteworthy metrics.

  • Changed surface: 4 files, +265/-37. The PR changes build targets, release validation, tests, and release docs in one release-critical path.
  • Release verifier expansion: 1 PowerShell verifier changed. The modified verifier gates x64 build, release artifact, and installer payload validation.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦪 silver shellfish
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Remove or justify native LoadLibrary probing of the managed Sherpa wrapper.
  • [P1] Add redacted Windows x64 terminal/log or recording proof from a published payload showing tray startup and Sherpa/ONNX initialization after the fix.
  • [P1] Report the required repo validation results once the PR is ready, including any Windows-specific release payload verifier output.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body/comments do not include terminal output, logs, screenshots, recordings, or linked artifacts showing the after-fix Windows x64 published tray and TTS/native stack behavior; contributors should add redacted proof and update the PR body so ClawSweeper re-reviews automatically, or ask a maintainer to comment @clawsweeper re-review if it does not.

Mantis proof suggestion
A desktop proof run could materially show the x64 published tray starts and stays running instead of crash-looping after native TTS initialization. A maintainer can ask Mantis to capture proof by posting a new PR comment that starts with the OpenClaw Mantis account mention, followed by:

visual task: verify the x64 published tray starts without a VC runtime or Sherpa/ONNX crash-loop and stays running after TTS initialization.

Risk before merge

  • [P1] No redacted Windows x64 after-fix proof is posted showing a published tray payload starts and initializes the Sherpa/ONNX stack without the reported crash-loop.
  • [P1] The x64 publish path now requires a Visual Studio install with the VC redist component, which can break existing local publish workflows that relied on NuGet-provided runtime DLLs.
  • [P1] The expanded release verifier can block release automation if the Win32 LoadLibrary probe treats the managed Sherpa wrapper assembly as a native DLL.

Maintainer options:

  1. Fix the verifier and require x64 proof (recommended)
    Remove the managed Sherpa wrapper from the Win32 native probe path or prove the probe is valid, then add redacted Windows x64 publish/tray/TTS proof before merge.
  2. Accept the stricter publish prerequisite
    Maintainers can intentionally accept the new Visual Studio requirement for x64 publish if the docs and error text are the intended upgrade path.
  3. Pause until release validation is proven
    Keep the draft unmerged if Windows build/release validation cannot show that the new verifier avoids false failures on the real payload.

Next step before merge

  • [P1] This needs contributor/maintainer handling for real Windows proof, the verifier bug, and the x64 publish compatibility decision before merge.

Security
Cleared: The diff touches release validation and executes a generated local probe, but I found no concrete secret exposure, permission broadening, unpinned action, or new third-party download execution.

Review findings

  • [P2] Remove managed Sherpa from the native load probe — scripts/Test-ReleaseNativeDependencies.ps1:105
Review details

Best possible solution:

Land a narrow release fix that sources x64 VC runtime DLLs from a current Microsoft redist source, keeps native probes scoped to native DLLs, makes the x64 publish prerequisite an explicit maintainer-approved choice, and includes redacted Windows x64 publish/tray startup proof.

Do we have a high-confidence way to reproduce the issue?

Yes. The linked report at #703 plus current source give a high-confidence Windows x64 path: publish with the app-local 14.29 VC runtime, launch tray/TTS, and observe the Sherpa/ONNX initialization crash; I did not execute that Windows repro here.

Is this the best way to solve the issue?

No, not as submitted. The MSBuild direction is plausible, but the verifier should avoid native-probing the managed Sherpa wrapper and the PR needs real Windows x64 after-fix proof before merge.

Full review comments:

  • [P2] Remove managed Sherpa from the native load probe — scripts/Test-ReleaseNativeDependencies.ps1:105
    sherpa-onnx.dll is later loaded with Assembly.LoadFrom as the managed wrapper, but this list sends it through the Win32 LoadLibrary probe first. That can fail for reasons unrelated to the native dependency chain and block release validation before the isolated TTS probe runs, so keep the native probe to actual native DLLs and let the reflection probe cover the managed wrapper.
    Confidence: 0.82

Overall correctness: patch is incorrect
Overall confidence: 0.78

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against e0a4b7c7c28d.

Label changes

Label justifications:

  • P1: The PR targets a reported tray startup crash-loop and changes release-critical packaging validation.
  • merge-risk: 🚨 compatibility: Merging changes x64 publish from NuGet-sourced runtime DLLs to a Visual Studio redist requirement, which can break existing local publish environments without VS.
  • merge-risk: 🚨 automation: The diff changes release validation behavior and can cause build/release gates to fail if the native probe is too broad.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body/comments do not include terminal output, logs, screenshots, recordings, or linked artifacts showing the after-fix Windows x64 published tray and TTS/native stack behavior; contributors should add redacted proof and update the PR body so ClawSweeper re-reviews automatically, or ask a maintainer to comment @clawsweeper re-review if it does not.
Evidence reviewed

What I checked:

  • Current x64 publish source: Current main still includes x64 VC runtime files from VCRuntime.CefSharp.140 1.0.5 before publish, while only ARM64 resolves from a Visual Studio install. (src/Directory.Build.targets:24, e0a4b7c7c28d)
  • ONNX/Sherpa dependency surface: The tray app references org.k2fsa.sherpa.onnx 1.13.0 and shared audio references Microsoft.ML.OnnxRuntime 1.26.0, matching the reported native stack. (src/OpenClaw.Tray.WinUI/OpenClaw.Tray.WinUI.csproj:70, e0a4b7c7c28d)
  • Release verifier gate: The workflow runs Test-ReleaseNativeDependencies.ps1 against x64 build artifacts and release payloads, so false verifier failures would block release automation. (.github/workflows/ci.yml:394, e0a4b7c7c28d)
  • PR head native probe bug: At PR head, Get-NativeLoadProbeFiles includes sherpa-onnx.dll in the Win32 LoadLibrary probe list, while the later TTS probe loads the same file as a managed assembly via Assembly.LoadFrom. (scripts/Test-ReleaseNativeDependencies.ps1:105, 4eb848b09bba)
  • Missing real behavior proof: The PR body and comments describe the intended fix but do not include terminal output, logs, screenshots, recordings, or linked artifacts showing an after-fix Windows x64 publish/tray/TTS run.
  • Canonical bug context: The related report at tray crash-loops on startup — bundled vc++ runtime older than onnxruntime needs #703 gives Event Log errors, identifies MSVCP140.dll 14.29.30142.1, and says replacing the app-local runtime with 14.51 fixes the crash-loop.

Likely related people:

  • Vincent Koc: Added the release native dependency verifier, VC runtime bundling targets, release docs, and test coverage that this PR changes. (role: introduced behavior; confidence: high; commits: 7485ce2af9da; files: scripts/Test-ReleaseNativeDependencies.ps1, src/Directory.Build.targets, docs/RELEASING.md)
  • Keith Mahoney: Recently changed the same release-runtime path to source ARM64 VC runtime files from Visual Studio and update release workflow/docs/tests. (role: recent area contributor; confidence: high; commits: f839cf53aa0c; files: src/Directory.Build.targets, .github/workflows/ci.yml, docs/RELEASING.md)
  • shanselman: Has recent release workflow/docs history and is requested as reviewer/assignee on this release packaging PR. (role: adjacent owner; confidence: medium; commits: c64484e392fe, 528f625edc5b, 435294415bea; files: docs/RELEASING.md, .github/workflows/ci.yml, tests/OpenClaw.Tray.Tests/ReleaseSigningWorkflowTests.cs)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

…nnxruntime crash

The VCRuntime.CefSharp.140 NuGet ships 14.29 DLLs that are too old for
onnxruntime 1.26. When these sit app-locally they shadow the system
runtime, causing sherpa-onnx-c-api to fail init (0x8007045A) and the
OfflineTts finalizer to crash-loop.

Both x64 and ARM64 publish now resolve VC++ runtime DLLs from the
Visual Studio install via vswhere (ARM64 already did this). The NuGet
is retained only as a dev-time convenience for local x64 builds.

Also adds a minimum version floor (14.38) in the release validation
script to prevent regressions.

Co-authored-by: shanselman <2892+shanselman@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix tray crash loops on startup due to VC++ runtime version mismatch Fix tray crash-loop: source x64 VC++ runtime from VS install at publish time Jun 7, 2026
Copilot AI requested a review from shanselman June 7, 2026 18:44
@clawsweeper clawsweeper Bot added the rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. label Jun 7, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 Urgent regression or broken agent/channel workflow affecting real users now. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 7, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@shanselman

Copy link
Copy Markdown
Collaborator

Validation/proof after the native TTS probe update:

  • Local required validation passed in an isolated PR-head worktree:
    • ./build.ps1
    • dotnet test ./tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj --no-restore
    • dotnet test ./tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj --no-restore
  • x64 publish verifier passed against a real published payload with current app-local Microsoft VC runtime DLLs.
  • Negative verifier proof passed: removing onnxruntime.dll fails the release verifier.
  • Negative verifier proof passed: removing the whole Sherpa/ONNX TTS stack fails the release verifier.
  • CI is green, including Build and Test/build (win-x64), which runs scripts/Test-ReleaseNativeDependencies.ps1 -PayloadPath publish -RequireAppLocalVCRuntime, and Build and Test/build (win-arm64).

@shanselman
shanselman marked this pull request as ready for review June 7, 2026 22:29
@shanselman
shanselman merged commit 8bf605c into master Jun 7, 2026
14 checks passed
@shanselman
shanselman deleted the copilot/fix-tray-crash-loops branch June 7, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P1 Urgent regression or broken agent/channel workflow affecting real users now. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tray crash-loops on startup — bundled vc++ runtime older than onnxruntime needs

2 participants