Skip to content

Commit 8bf605c

Browse files
CopilotshanselmanCopilot
authored
Fix tray crash-loop: source x64 VC++ runtime from VS install at publish time (#713)
* Initial plan * fix: source x64 VC++ runtime from VS install at publish time to fix onnxruntime crash The VCRuntime.CefSharp.140 NuGet ships 14.29 DLLs that are too old for onnxruntime 1.26. When these sit app-locally they shadow the system runtime, causing sherpa-onnx-c-api to fail init (0x8007045A) and the OfflineTts finalizer to crash-loop. Both x64 and ARM64 publish now resolve VC++ runtime DLLs from the Visual Studio install via vswhere (ARM64 already did this). The NuGet is retained only as a dev-time convenience for local x64 builds. Also adds a minimum version floor (14.38) in the release validation script to prevent regressions. Co-authored-by: shanselman <2892+shanselman@users.noreply.github.com> * test(release): probe TTS native stack Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(release): run TTS probe under dotnet Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: shanselman <2892+shanselman@users.noreply.github.com> Co-authored-by: Scott Hanselman <scott@hanselman.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent fd31e90 commit 8bf605c

4 files changed

Lines changed: 265 additions & 37 deletions

File tree

‎docs/RELEASING.md‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -99,16 +99,21 @@ reviewed deliberately.
9999

100100
CI also checks native runtime dependencies before release packaging. Both the
101101
x64 and ARM64 portable payloads must ship `vcruntime140.dll` next to every
102-
`libsodium.dll` copy. The x64 build leg sources its loose VC runtime DLLs from
103-
the `VCRuntime.CefSharp.140` NuGet package; the ARM64 build leg sources its
104-
DLLs from the Visual Studio install on the `windows-11-arm` runner (resolved
105-
via `vswhere` in `src\Directory.Build.targets`). The release job must
106-
Authenticode-verify Microsoft's x64 and ARM64 Visual C++ Runtime
107-
redistributables before passing the architecture-matching redistributable to
108-
Inno. The installer runs the redistributable before launching the tray so
109-
clean or stale Windows hosts can repair the runtime before Ed25519 device keys
110-
are generated or loaded, and it skips the post-install tray launch if the
111-
runtime installer fails.
102+
`libsodium.dll` copy. Both build legs source their loose VC runtime DLLs from
103+
the Visual Studio install on the CI runner (resolved via `vswhere` in
104+
`src\Directory.Build.targets`). This ensures the bundled CRT is new enough for
105+
`onnxruntime` — the `VCRuntime.CefSharp.140` NuGet is only used as a dev-time
106+
convenience for local `dotnet build` (not publish). The release validation
107+
script enforces a minimum VC++ runtime version floor (currently 14.38) to
108+
prevent regressions, and the x64 verifier load-probes the native TTS stack
109+
(`onnxruntime.dll`, `sherpa-onnx.dll`, and `sherpa-onnx-c-api.dll`) from the
110+
published payload so app-local runtime mismatches are caught before release.
111+
The release job must Authenticode-verify Microsoft's x64 and ARM64 Visual C++
112+
Runtime redistributables before passing the
113+
architecture-matching redistributable to Inno. The installer runs the
114+
redistributable before launching the tray so clean or stale Windows hosts can
115+
repair the runtime before Ed25519 device keys are generated or loaded, and it
116+
skips the post-install tray launch if the runtime installer fails.
112117

113118
The current Azure Artifact Signing resource is:
114119

‎scripts/Test-ReleaseNativeDependencies.ps1‎

Lines changed: 215 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ public static class OpenClawNativeDependencyProbe {
4242
4343
[DllImport("kernel32", SetLastError=true, CharSet=CharSet.Unicode)]
4444
public static extern bool SetDllDirectory(string lpPathName);
45+
46+
[DllImport("kernel32", SetLastError=true)]
47+
public static extern bool FreeLibrary(IntPtr hModule);
4548
}
4649
"@
4750
}
@@ -90,6 +93,206 @@ function Get-VCRuntimeFiles {
9093
) | Sort-Object FullName -Unique
9194
}
9295

96+
function Get-NativeLoadProbeFiles {
97+
param(
98+
[Parameter(Mandatory = $true)]
99+
[string]$Directory
100+
)
101+
102+
@(
103+
Get-ChildItem -LiteralPath $Directory -File -Filter "libsodium.dll"
104+
Get-ChildItem -LiteralPath $Directory -File -Filter "onnxruntime.dll"
105+
Get-ChildItem -LiteralPath $Directory -File -Filter "sherpa-onnx.dll"
106+
Get-ChildItem -LiteralPath $Directory -File -Filter "sherpa-onnx-c-api.dll"
107+
) | Sort-Object FullName -Unique
108+
}
109+
110+
function Add-NativeLoadProbeErrors {
111+
param(
112+
[Parameter(Mandatory = $true)]
113+
[System.IO.FileInfo]$File
114+
)
115+
116+
if (-not $shouldProbeNativeLoad) {
117+
return
118+
}
119+
120+
[OpenClawNativeDependencyProbe]::SetDllDirectory($File.DirectoryName) | Out-Null
121+
Push-Location $File.DirectoryName
122+
try {
123+
$handle = [OpenClawNativeDependencyProbe]::LoadLibrary($File.Name)
124+
$lastError = [Runtime.InteropServices.Marshal]::GetLastWin32Error()
125+
} finally {
126+
Pop-Location
127+
[OpenClawNativeDependencyProbe]::SetDllDirectory($null) | Out-Null
128+
}
129+
130+
if ($handle -eq [IntPtr]::Zero) {
131+
$errors.Add("Native dependency $(Get-RelativePath -Root $payloadRoot -Path $File.FullName) failed to load with app-local dependencies (Win32 error $lastError).")
132+
return
133+
}
134+
135+
[OpenClawNativeDependencyProbe]::FreeLibrary($handle) | Out-Null
136+
}
137+
138+
function Add-TtsNativeStackProbeErrors {
139+
if (-not $shouldProbeNativeLoad) {
140+
return
141+
}
142+
143+
$requiredFiles = @(
144+
"Microsoft.ML.OnnxRuntime.dll"
145+
"onnxruntime.dll"
146+
"sherpa-onnx.dll"
147+
"sherpa-onnx-c-api.dll"
148+
)
149+
150+
$filesByName = @{}
151+
foreach ($fileName in $requiredFiles) {
152+
$file = Get-ChildItem -LiteralPath $payloadRoot -Recurse -File -Filter $fileName | Select-Object -First 1
153+
if ($file) {
154+
$filesByName[$fileName] = $file
155+
} else {
156+
$errors.Add("Missing $fileName for TTS native stack probe.")
157+
}
158+
}
159+
160+
if ($filesByName.Count -ne $requiredFiles.Count) {
161+
return
162+
}
163+
164+
$dotnet = Get-Command dotnet -ErrorAction SilentlyContinue
165+
if (-not $dotnet) {
166+
$errors.Add("Cannot run isolated TTS native stack probe because dotnet was not found.")
167+
return
168+
}
169+
170+
$probeProject = @'
171+
<Project Sdk="Microsoft.NET.Sdk">
172+
<PropertyGroup>
173+
<OutputType>Exe</OutputType>
174+
<TargetFramework>net10.0</TargetFramework>
175+
<ImplicitUsings>enable</ImplicitUsings>
176+
<Nullable>enable</Nullable>
177+
</PropertyGroup>
178+
</Project>
179+
'@
180+
181+
$probeProgram = @'
182+
using System.Reflection;
183+
184+
if (args.Length != 1)
185+
{
186+
Console.Error.WriteLine("Expected payload root argument.");
187+
return 2;
188+
}
189+
190+
var payloadRoot = Path.GetFullPath(args[0]);
191+
Directory.SetCurrentDirectory(payloadRoot);
192+
193+
AppDomain.CurrentDomain.AssemblyResolve += (_, eventArgs) =>
194+
{
195+
var assemblyName = new AssemblyName(eventArgs.Name).Name;
196+
if (string.IsNullOrWhiteSpace(assemblyName))
197+
{
198+
return null;
199+
}
200+
201+
var candidate = Path.Combine(payloadRoot, assemblyName + ".dll");
202+
return File.Exists(candidate) ? Assembly.LoadFrom(candidate) : null;
203+
};
204+
205+
var sherpaAsm = Assembly.LoadFrom(Path.Combine(payloadRoot, "sherpa-onnx.dll"));
206+
var versionType = sherpaAsm.GetType("SherpaOnnx.VersionInfo", true)!;
207+
var version = versionType.GetProperty("Version")!.GetValue(null)?.ToString();
208+
if (string.IsNullOrWhiteSpace(version))
209+
{
210+
throw new InvalidOperationException("SherpaOnnx.VersionInfo.Version returned an empty version.");
211+
}
212+
213+
var onnxAsm = Assembly.LoadFrom(Path.Combine(payloadRoot, "Microsoft.ML.OnnxRuntime.dll"));
214+
var ortType = onnxAsm.GetType("Microsoft.ML.OnnxRuntime.OrtEnv", true)!;
215+
var instanceMethod = ortType.GetMethod(
216+
"Instance",
217+
BindingFlags.Public | BindingFlags.Static,
218+
binder: null,
219+
types: Type.EmptyTypes,
220+
modifiers: null);
221+
var env = instanceMethod is not null
222+
? instanceMethod.Invoke(null, null)
223+
: ortType.GetProperty("Instance", BindingFlags.Public | BindingFlags.Static)?.GetValue(null);
224+
225+
if (env is null)
226+
{
227+
throw new InvalidOperationException("Microsoft.ML.OnnxRuntime.OrtEnv did not initialize.");
228+
}
229+
230+
Console.WriteLine($"TTS native stack probe passed (Sherpa {version}, ONNX Runtime initialized).");
231+
return 0;
232+
'@
233+
234+
$probeDir = Join-Path ([System.IO.Path]::GetTempPath()) ("openclaw-tts-native-probe-{0}" -f [Guid]::NewGuid().ToString("N"))
235+
try {
236+
New-Item -ItemType Directory -Path $probeDir | Out-Null
237+
$projectPath = Join-Path $probeDir "OpenClaw.TtsNativeProbe.csproj"
238+
$programPath = Join-Path $probeDir "Program.cs"
239+
Set-Content -LiteralPath $projectPath -Value $probeProject -Encoding UTF8
240+
Set-Content -LiteralPath $programPath -Value $probeProgram -Encoding UTF8
241+
242+
$buildOutput = & $dotnet.Source build $projectPath -c Release -v:q 2>&1
243+
$buildExitCode = $LASTEXITCODE
244+
if ($buildExitCode -ne 0) {
245+
$tail = @($buildOutput | Select-Object -Last 12) -join " "
246+
$errors.Add("TTS native stack probe build failed with exit code $buildExitCode. $tail")
247+
return
248+
}
249+
250+
$probeDll = Join-Path $probeDir "bin\Release\net10.0\OpenClaw.TtsNativeProbe.dll"
251+
$output = & $dotnet.Source $probeDll $payloadRoot 2>&1
252+
$exitCode = $LASTEXITCODE
253+
} finally {
254+
try { if (Test-Path -LiteralPath $probeDir) { Remove-Item -LiteralPath $probeDir -Recurse -Force } } catch { }
255+
}
256+
257+
if ($exitCode -ne 0) {
258+
if ($exitCode -lt 0) {
259+
$errors.Add("TTS native stack probe crashed with exit code $exitCode. This usually means the app-local Sherpa/ONNX native dependency chain failed to initialize.")
260+
return
261+
}
262+
263+
$tail = @($output | Select-Object -Last 12) -join " "
264+
$errors.Add("TTS native stack probe failed with exit code $exitCode. $tail")
265+
}
266+
}
267+
268+
# onnxruntime >= 1.20 is built with a VS 2022 toolchain that requires
269+
# VC++ runtime 14.38+. Shipping older DLLs app-locally shadows the system
270+
# runtime and causes 0x8007045A DllNotFoundException at startup.
271+
# Floor: 14.38.33130.0 (VS 17.8, the first 14.38 release).
272+
$script:VCRuntimeMinVersion = [version]"14.38.33130.0"
273+
274+
function Add-VCRuntimeVersionFloorErrors {
275+
param(
276+
[Parameter(Mandatory = $true)]
277+
[System.IO.FileInfo]$File
278+
)
279+
280+
if (-not $runningOnWindows) {
281+
return
282+
}
283+
284+
$vi = [System.Diagnostics.FileVersionInfo]::GetVersionInfo($File.FullName)
285+
if (-not $vi -or -not $vi.FileVersion) {
286+
$errors.Add("Cannot read file version from $(Get-RelativePath -Root $payloadRoot -Path $File.FullName).")
287+
return
288+
}
289+
290+
$fileVer = [version]::new($vi.FileMajorPart, $vi.FileMinorPart, $vi.FileBuildPart, $vi.FilePrivatePart)
291+
if ($fileVer -lt $script:VCRuntimeMinVersion) {
292+
$errors.Add("VC++ runtime $(Get-RelativePath -Root $payloadRoot -Path $File.FullName) is version $fileVer, which is older than the minimum $($script:VCRuntimeMinVersion) required by onnxruntime. Update the VC++ Redistributable or Visual Studio install.")
293+
}
294+
}
295+
93296
$libsodiumFiles = @(
94297
Get-ChildItem -LiteralPath $payloadRoot -Recurse -File -Filter libsodium.dll |
95298
Sort-Object FullName
@@ -108,23 +311,23 @@ foreach ($libsodium in $libsodiumFiles) {
108311
if ($RequireAppLocalVCRuntime) {
109312
foreach ($runtimeFile in Get-VCRuntimeFiles -Directory $libsodium.DirectoryName) {
110313
Add-MicrosoftSignatureErrors -File $runtimeFile
314+
Add-VCRuntimeVersionFloorErrors -File $runtimeFile
111315
}
112316
}
317+
}
113318

114-
if ($shouldProbeNativeLoad) {
115-
[OpenClawNativeDependencyProbe]::SetDllDirectory($libsodium.DirectoryName) | Out-Null
116-
Push-Location $libsodium.DirectoryName
117-
try {
118-
$handle = [OpenClawNativeDependencyProbe]::LoadLibrary("libsodium.dll")
119-
$lastError = [Runtime.InteropServices.Marshal]::GetLastWin32Error()
120-
} finally {
121-
Pop-Location
122-
}
319+
if ($shouldProbeNativeLoad) {
320+
$probeFiles = @(
321+
Get-ChildItem -LiteralPath $payloadRoot -Recurse -Directory |
322+
ForEach-Object { Get-NativeLoadProbeFiles -Directory $_.FullName }
323+
Get-NativeLoadProbeFiles -Directory $payloadRoot
324+
) | Sort-Object FullName -Unique
123325

124-
if ($handle -eq [IntPtr]::Zero) {
125-
$errors.Add("libsodium.dll failed to load from $(Get-RelativePath -Root $payloadRoot -Path $libsodium.FullName) (Win32 error $lastError).")
126-
}
326+
foreach ($probeFile in $probeFiles) {
327+
Add-NativeLoadProbeErrors -File $probeFile
127328
}
329+
330+
Add-TtsNativeStackProbeErrors
128331
}
129332

130333
if ($RequireInstallerVCRedist) {

‎src/Directory.Build.targets‎

Lines changed: 29 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -8,32 +8,32 @@
88
</PropertyGroup>
99

1010
<!--
11-
x64 ships loose VC runtime DLLs sourced from the VCRuntime.CefSharp.140
12-
NuGet package, so devs don't need a Visual Studio install to build/publish.
13-
14-
ARM64 is resolved at publish time from the local Visual Studio install via
15-
ResolveOpenClawVCRuntimeArm64FromVSInstall — the existing NuGet ships no
16-
ARM64 binaries and Microsoft does not publish a comparable NuGet for the
17-
loose ARM64 redist DLLs.
11+
Both x64 and ARM64 resolve loose VC++ runtime DLLs from the local Visual
12+
Studio install at publish time (via ResolveOpenClawVCRuntimeFromVSInstall).
13+
This ensures onnxruntime and other native dependencies always get a current
14+
CRT — the VCRuntime.CefSharp.140 NuGet ships 14.29 DLLs that are too old
15+
for onnxruntime ≥ 1.20.
1816
17+
For local x64 *builds* (not publish), the NuGet DLLs are still copied to
18+
the output directory as a convenience so devs without VS can iterate.
1919
Local 'dotnet build -r win-arm64' intentionally does NOT require a VS
2020
install (so cross-compile sanity checks keep working on x64 dev boxes).
2121
The VS-install requirement only kicks in at publish time, which is what
22-
the release pipeline actually runs on the windows-11-arm runner.
22+
the release pipeline actually runs.
2323
-->
2424
<ItemGroup Condition="'$(OpenClawVCRuntimeArch)' == 'x64'">
2525
<OpenClawVCRuntimeFiles Include="$(OpenClawVCRuntimePackageRoot)x64\*.dll" />
2626
</ItemGroup>
2727

28-
<Target Name="ResolveOpenClawVCRuntimeArm64FromVSInstall"
29-
Condition="'$(OpenClawVCRuntimeArch)' == 'arm64'">
28+
<Target Name="ResolveOpenClawVCRuntimeFromVSInstall"
29+
Condition="'$(OpenClawVCRuntimeArch)' != ''">
3030

3131
<PropertyGroup>
3232
<_OpenClawVsWherePath>$(MSBuildProgramFiles32)\Microsoft Visual Studio\Installer\vswhere.exe</_OpenClawVsWherePath>
3333
</PropertyGroup>
3434

3535
<Error Condition="!Exists('$(_OpenClawVsWherePath)')"
36-
Text="vswhere.exe not found at '$(_OpenClawVsWherePath)'. Publishing for win-arm64 requires a Visual Studio install with the C++ Redistributable (Microsoft.VisualStudio.Component.VC.Redist.14.Latest) component. Install Visual Studio 2022 or later with that component, or publish for win-x64 instead." />
36+
Text="vswhere.exe not found at '$(_OpenClawVsWherePath)'. Publishing requires a Visual Studio install with the C++ Redistributable (Microsoft.VisualStudio.Component.VC.Redist.14.Latest) component." />
3737

3838
<Exec Command="&quot;$(_OpenClawVsWherePath)&quot; -latest -products * -requires Microsoft.VisualStudio.Component.VC.Redist.14.Latest -property installationPath"
3939
ConsoleToMSBuild="true"
@@ -42,7 +42,7 @@
4242
</Exec>
4343

4444
<Error Condition="'$(_OpenClawVsInstallRoot)' == '' or !Exists('$(_OpenClawVsInstallRoot)')"
45-
Text="vswhere did not find a Visual Studio install with the C++ Redistributable (Microsoft.VisualStudio.Component.VC.Redist.14.Latest) component (got '$(_OpenClawVsInstallRoot)'). Install that component via the Visual Studio Installer (Individual Components tab), or publish for win-x64 instead." />
45+
Text="vswhere did not find a Visual Studio install with the C++ Redistributable (Microsoft.VisualStudio.Component.VC.Redist.14.Latest) component (got '$(_OpenClawVsInstallRoot)'). Install that component via the Visual Studio Installer (Individual Components tab)." />
4646

4747
<!--
4848
Glob the loose CRT DLLs. The Microsoft.VC*.CRT folder name carries the VS
@@ -52,23 +52,37 @@
5252
vcruntime140*.dll + msvcp140*.dll to match the file set the x64 NuGet
5353
ships (concrt140/vccorlib140/vcruntime140_threads aren't needed by
5454
libsodium and aren't in the x64 payload either).
55+
56+
For x64 this replaces the stale NuGet-sourced items so publish always
57+
ships current DLLs from the VS install.
5558
-->
56-
<ItemGroup>
59+
<ItemGroup Condition="'$(OpenClawVCRuntimeArch)' == 'x64'">
60+
<OpenClawVCRuntimeFiles Remove="@(OpenClawVCRuntimeFiles)" />
61+
<OpenClawVCRuntimeFiles Include="$(_OpenClawVsInstallRoot)\VC\Redist\MSVC\*\x64\Microsoft.VC*.CRT\vcruntime140*.dll" />
62+
<OpenClawVCRuntimeFiles Include="$(_OpenClawVsInstallRoot)\VC\Redist\MSVC\*\x64\Microsoft.VC*.CRT\msvcp140*.dll" />
63+
</ItemGroup>
64+
65+
<ItemGroup Condition="'$(OpenClawVCRuntimeArch)' == 'arm64'">
5766
<OpenClawVCRuntimeFiles Include="$(_OpenClawVsInstallRoot)\VC\Redist\MSVC\*\arm64\Microsoft.VC*.CRT\vcruntime140*.dll" />
5867
<OpenClawVCRuntimeFiles Include="$(_OpenClawVsInstallRoot)\VC\Redist\MSVC\*\arm64\Microsoft.VC*.CRT\msvcp140*.dll" />
5968
</ItemGroup>
6069

6170
<Error Condition="'@(OpenClawVCRuntimeFiles)' == ''"
62-
Text="No ARM64 VC++ Runtime DLLs were found under '$(_OpenClawVsInstallRoot)\VC\Redist\MSVC\*\arm64\Microsoft.VC*.CRT\'. Install the 'C++ ARM64/ARM64EC build tools' and 'C++ Redistributable Update' Visual Studio components." />
71+
Text="No $(OpenClawVCRuntimeArch) VC++ Runtime DLLs were found under '$(_OpenClawVsInstallRoot)\VC\Redist\MSVC\*\$(OpenClawVCRuntimeArch)\Microsoft.VC*.CRT\'. Install the C++ Redistributable Update Visual Studio component." />
6372
</Target>
6473

74+
<!-- Legacy alias so existing CI log searches and docs still find the target. -->
75+
<Target Name="ResolveOpenClawVCRuntimeArm64FromVSInstall"
76+
DependsOnTargets="ResolveOpenClawVCRuntimeFromVSInstall" />
77+
6578
<Target Name="CopyOpenClawVCRuntimeToOutput"
6679
AfterTargets="Build"
6780
Condition="'$(OpenClawVCRuntimeArch)' == 'x64' and '@(OpenClawVCRuntimeFiles)' != ''">
6881
<!--
6982
x64-only on purpose. ARM64 publishes its runtime through the publish-only
7083
target below so that local 'dotnet build -r win-arm64' does not require
7184
a Visual Studio install.
85+
Uses the NuGet-sourced DLLs (the VS resolution only runs at publish).
7286
-->
7387
<Copy SourceFiles="@(OpenClawVCRuntimeFiles)"
7488
DestinationFolder="$(TargetDir)"
@@ -77,7 +91,7 @@
7791

7892
<Target Name="CopyOpenClawVCRuntimeToPublish"
7993
AfterTargets="Publish"
80-
DependsOnTargets="ResolveOpenClawVCRuntimeArm64FromVSInstall"
94+
DependsOnTargets="ResolveOpenClawVCRuntimeFromVSInstall"
8195
Condition="'$(PublishDir)' != '' and '$(OpenClawVCRuntimeArch)' != ''">
8296
<Copy SourceFiles="@(OpenClawVCRuntimeFiles)"
8397
DestinationFolder="$(PublishDir)"

0 commit comments

Comments
 (0)