Speed up end-to-end C# Azure SDK regeneration

[live1206]

Problem

The provider-based reference map significantly improves C# generator performance, including large management-plane SDKs, but the end-to-end Azure SDK regeneration in packages/http-client-csharp/eng/pipeline/publish.yml does not show a comparable wall-clock improvement and can still approach the 90-minute job timeout.

Related work:

• Use provider C# reference map with pre-write internalization #10976
• Fix C# reference-map accessibility for hidden union types #11288

Current behavior

The full data-plane + management-plane regeneration currently covers approximately:

• 243 SDK projects
• 202 distinct service directories
• 197 management-plane SDK projects across 188 service directories

Submit-AzureSdkForNetPr.ps1 processes service directories serially. For every service it starts a new command similar to:

dotnet msbuild eng/service.proj /restore /t:GenerateCode /p:Trace=true /p:ServiceDirectory=<service>

This repeats substantial work outside the optimized generator phase:

• MSBuild process startup, evaluation, and restore for each service directory
• restore graph discovery that includes tests, samples, perf, stress, and other non-codegen projects by default
• serial InstallTspClient traversal and repeated npm ci work
• repeated client plugin builds; child generation receives SkipTspClientInstall=true but not SkipBuildPlugin=true
• tsp-client update and TypeSpec/spec synchronization for every SDK
• trace/debug logging and generated-file disk I/O

The outer service loop itself has no bounded parallelism. The job is capped at 90 minutes, so runs that reach the cap do not provide a useful completed-runtime comparison.

The generator microbenchmark in #10976 improved from 847.4 ms to 544.5 ms. That saves only about 74 seconds when naively multiplied across 243 typical generations, which can be hidden by orchestration variance. The large SDK measurements are much more significant—Network, DataFactory, and AppService together showed nearly 48 minutes of local savings—so a completed post-#11288 full run should still be measurably faster if it exercises the same optimized path.

Proposed work

1. Add phase and per-service timing with an end-of-job summary that separates setup, restore, tsp-client install/sync, TypeSpec compilation, C# generator execution, plugin build, and file writing.
2. Install tsp-client once per job and reuse it across SDK generation.
3. Build the client plugin once and pass SkipBuildPlugin=true to child generation where safe.
4. Restrict restore and MSBuild traversal to actual code-generation projects.
5. Avoid repeating restore/spec synchronization when inputs and dependencies are shared or already prepared.
6. Add bounded service-level parallelism. Resolve shared plugin-output conflicts generically rather than maintaining service-specific serialization exceptions.
7. Re-evaluate trace/debug logging volume in the full regeneration path.

Acceptance criteria

• Capture a completed baseline and optimized run using the same TypeSpec commit, Azure SDK commit, parameters, agent image, and cache state.
• Report a phase-level timing breakdown and per-service timings in pipeline output.
• The full data-plane + management-plane regeneration completes comfortably below the 90-minute timeout with a clear wall-clock improvement over baseline.
• Generated output is identical to the equivalent sequential regeneration.
• Parallel execution does not introduce shared-output races or flaky generation.

[github-actions]

🎯 Issue Triage

This is a feature/improvement request to optimize the end-to-end C# Azure SDK regeneration pipeline (publish.yml), which currently risks hitting a 90-minute timeout due to serial processing, repeated installs, and redundant MSBuild restores across ~243 SDK projects.

Category: Feature Request
Area: emitter:client:csharp

🔍 Analysis

• The generator itself was optimized in Use provider C# reference map with pre-write internalization #10976 and Fix C# reference-map accessibility for hidden union types #11288 (provider-based reference map), but pipeline orchestration overhead negates the wall-clock benefit at scale.
• Key bottlenecks: serial service-directory loop in Submit-AzureSdkForNetPr.ps1, repeated npm ci / tsp-client installs, per-service MSBuild restores including non-codegen projects, and no bounded parallelism.
• The proposed work covers 7 distinct optimization areas: timing instrumentation, shared installs, plugin build deduplication, narrow restore scope, parallelism with race-free output, and logging reduction.
• Acceptance criteria are well-defined and testable (baseline vs. optimized run comparison, phase-level timing, correctness parity).

📝 Suggested next steps

• Capture a completed baseline run (post-Fix C# reference-map accessibility for hidden union types #11288) to establish a timing benchmark before any orchestration changes.
• Implement timing/phase instrumentation first (item 1) to guide prioritization of the remaining items.
• Address the highest-impact items (single tsp-client install, single plugin build, bounded parallelism) iteratively, validating generated output parity after each.

Generated by Agentic Triage

[jorgerangel-msft]

FYI - Our local regen preview script https://github.com/microsoft/typespec/blob/main/packages/http-client-csharp/eng/scripts/RegenPreview.ps1 already allows for parallel generation.

[live1206]

FYI - Our local regen preview script https://github.com/microsoft/typespec/blob/main/packages/http-client-csharp/eng/scripts/RegenPreview.ps1 already allows for parallel generation.

After our perf improvement PR merged, the regen CI didn't get any overall runtime reduce, this is not about the local regen.

[jorgerangel-msft]

@live1206 my intention was to highlight that maybe we can utilize the existing script somehow in CI since it already handles a lot of what is outlined in this issue's description.