Async Refactor

[timcassell]

Core changes:

1. Added BenchmarkRunner.RunAsync/BenchmarkSwitcher.Run*Async APIs.
2. Added cooperative cancellation support.
3. Async benchmarks are now properly awaited instead of synchronously blocked
   • This follows the suggested implementation in ValueTask-returning benchmarks are not being called correctly #1595 (comment)
   • Allows async benchmarks to work properly in-process with UI frameworks (currently it deadlocks).
4. Interfaces changed to support async.
5. IterationSetup/Cleanup now support async methods.
6. IClock is passed to the WorkloadActionNoUnroll etc. benchmark methods and they pass back ClockSpan, instead of the Engine starting and stopping the clock.
   • This is to avoid measuring workload setup before the core loop and async continuations for more accurate measurements. (It also helps to simplify the jit stage.)

Behavior changes:

1. await Task.Yield() continues on the current synchronization context if it exists, or the current task scheduler. Previously this meant it continued on a ThreadPool thread, now it means it will likely run on the same thread via the new BenchmarkSynchronizationContext. This should be benign.

Breaking changes:

1. IHost
   • Removed unused void Write(string message)
   • Added CancellationToken CancellationToken { get; } and ValueTask Yield()
   • Changed void SendSignal(HostSignal hostSignal) to ValueTask SendSignalAsync(HostSignal hostSignal)
2. IEngine
   • RunResults Run() changed to ValueTask<RunResults> RunAsync()
3. EngineParameters
   • WorkloadActionNoUnroll etc. changed from Action<long> to Func<long, IClock, ValueTask<ClockSpan>>
   • GlobalSetupAction etc. changed from Action to Func<ValueTask>
4. IExecutor
   • ExecuteResult Execute(ExecuteParameters executeParameters) changed to ValueTask<ExecuteResult> ExecuteAsync(ExecuteParameters executeParameters, CancellationToken cancellationToken)
5. IGenerator
   • Changed GenerateResult GenerateProject(BuildPartition buildPartition, ILogger logger, string rootArtifactsFolderPath) to ValueTask<GenerateResult> GenerateProjectAsync(BuildPartition buildPartition, ILogger logger, string rootArtifactsFolderPath, CancellationToken cancellationToken)
6. IBuilder
   • Changed BuildResult Build(GenerateResult generateResult, BuildPartition buildPartition, ILogger logger) to ValueTask<BuildResult> BuildAsync(GenerateResult generateResult, BuildPartition buildPartition, ILogger logger, CancellationToken cancellationToken)
7. IDiagnoser
   • Changed void Handle(HostSignal signal, DiagnoserActionParameters parameters) to ValueTask HandleAsync(HostSignal signal, DiagnoserActionParameters parameters, CancellationToken cancellationToken)
8. IDiagnoser, IToolchain, IValidator
   • Changed IEnumerable<ValidationError> Validate(ValidationParameters validationParameters) to IAsyncEnumerable<ValidationError> ValidateAsync(ValidationParameters validationParameters);
9. IExporter
   • Removed ExportToLog and ExportToFiles
   • Added ValueTask ExportAsync(Summary summary, ILogger logger, CancellationToken cancellationToken)

Other Changes:

1. Added support for custom awaitable returns
   • Not supported in InProcessNoEmitToolchain
   • Does not include extension await
2. New attributes
   • [AsyncCallerType] allows users to override the type used in the async method that calls their benchmark method.
     • Not supported in InProcessNoEmitToolchain
   • [BenchmarkCancellation] allows benchmarks to respond to cancellation requests from the user.
   • [AggressivelyOptimizeMethods] instructs the assembly weaver to apply AggressiveOptimization to all methods in the annotated type and nested types.
     • This is intended for internal use, but users can also use it. It's needed because C# doesn't allow to apply attributes to compiler-generated async state machines.
3. Simplified EngineJitStage
4. ReturnValueValidator is now able to validate async results.
5. Anonymous pipe IPC changed to TCP loopback.
6. Wasm toolchain now supports IPC for all JS engines (WebSocket or StdOut+File depending on the engine; configurable).
7. Added new analyzers and a code fixer.

Fixes #2442
Fixes #2159
Fixes #2299
Fixes #2162
Unblocks #1808

[timcassell]

@VSadov Should we emit runtime-async methods for this? Is it supported already? If so, what's the pattern?

[timcassell]

@copilot review

[timcassell]

I did the minimum changes required for proper async benchmarks, but I'm not sure if we want to take it further. We could make all interface methods async (IGenerator.GenerateProjectAsync, IBuilder.BuildAsync, etc), and we could maybe add CancellationToken support (I'm not sure the best way to signal cancellation to the child process, though).

@stephentoub Can we get your input here?

[stephentoub]

@stephentoub Can we get your input here?

Input on what specifically?

[timcassell]

@stephentoub Can we get your input here?

Input on what specifically?

On the whole async API design. More specifically my previous comment about how much of the surface we should make async and whether it even makes sense to add cancelation token support.

[timcassell]

@pavelsavara Can you sanity check the wasm changes here? Claude had the genius idea to use file-based IPC for JS shells that don't support WebSocket, so I was able to use that and still keep v8 as the default engine. (I don't expect a full review, this PR is too massive.)

[timcassell]

🤖 This review was generated by Claude.

PR #2958 Review: Async Refactoring

Overview

This is a massive PR (300+ files, 18K+ additions) that converts BenchmarkDotNet's entire pipeline to async. The core goals are well-motivated: proper async benchmark awaiting (fixing deadlocks with UI frameworks), cooperative cancellation, and async RunAsync/RunAllAsync APIs.

The architecture is sound. The key design decisions — BenchmarkSynchronizationContext for thread affinity, WorkloadContinuerAndValueTaskSource for zero-alloc async loops, TCP loopback replacing anonymous pipes, IClock passed into workload methods — are all well-reasoned.

Engine & Core Pipeline

• The async flow through BenchmarkRunnerClean → Engine.RunAsync() → workload actions is clean and correct.
• ConfigureAwait(true) usage in Engine.cs is intentional and appropriate — it ensures continuations run on the BenchmarkSynchronizationContext thread for consistent measurements.
• Exception handling in RunAsync() properly preserves the original exception when cleanup also throws (Dry jobs can eat iteration failures #1045).

IPC Changes (Anonymous Pipes → TCP Loopback)

• Binding to IPAddress.Loopback with port 0 and backlog of 1 is appropriate for single-process IPC.
• The signal/acknowledgment protocol between TcpHost and Broker is well-designed, with proper handling of the Cancel signal allowing AfterAll to still be received.
• Broker.ProcessDataCore correctly uses a linked CancellationTokenSource for AcceptConnection and properly scopes the cancellation registration.
• Socket error handling (ConnectionReset, Shutdown, OperationAborted) covers the expected failure modes.

InProcess Emit (IL Emission)

This is the most impressive part of the PR. The async state machine emission is correct and matches Roslyn-generated patterns:

• State initialized to -1, set to -2 on completion/exception — correct per C# compiler convention.
• AsyncTaskMethodBuilder lifecycle (Create → Start → SetResult/SetException) is properly implemented.
• Value type vs reference type awaiter handling (Call vs Callvirt, ldloca vs ldloc) is correct.
• The WorkloadContinuerAndValueTaskSource integration for reusable async state machines in the benchmark loop is sophisticated and well-executed.
• Setup/cleanup emission correctly wraps synchronous methods in completed ValueTask for uniform API.

Validators, Diagnosers, Exporters

• IValidator.ValidateAsync returning IAsyncEnumerable<ValidationError> is a clean async streaming interface.
• BenchmarkCancellationValidator is thorough — correctly inspects fields and properties with comprehensive error messages.
• Exporters make good use of async file I/O and await using with ProcessCleanupHelper and async channels for streaming process output.

Analyzers & Code Fixers

• AsyncBenchmarkAnalyzer and BenchmarkCancellationAttributeAnalyzer provide good compile-time guidance for users adopting the new async APIs.
• Comprehensive validation of [BenchmarkCancellation] attribute usage (type checks, accessibility, mutability).

Testing

Good coverage with RunAsyncTests, CancellationTokenTests, CustomTaskAndAwaitableTests, and CallerThreadTests covering the key new functionality, plus analyzer test coverage.

Minor Observations

None of these affect the overall design or implementation:

• Broker doesn't dispose IpcListener itself — ownership is managed by the outer scope in Executor.cs, but the implicit ownership model could be documented.
• TcpClient.Connect in IpcHelper has no explicit timeout, relying on the OS default.
• IHost.Yield() purpose (yielding back to JS in WASM) could be documented on the interface.

Verdict

The async refactoring is well-architected and the implementation is solid for a change of this scope. The IL emission work is particularly impressive. No issues found.