Skip to content

[http-client-csharp] Add Tier 1 in-memory response cache to playground-server - #10718

Merged
JoshLove-msft merged 18 commits into
mainfrom
copilot/improve-csharp-emitter-performance
Jun 3, 2026
Merged

[http-client-csharp] Add Tier 1 in-memory response cache to playground-server#10718
JoshLove-msft merged 18 commits into
mainfrom
copilot/improve-csharp-emitter-performance

Conversation

Copilot AI commented May 18, 2026

Copy link
Copy Markdown
Contributor

The playground's C# /generate endpoint spawns a fresh dotnet subprocess on every request, with no caching anywhere in the pipeline. Identical replays (undo/redo, share-link reloads, demo specs hit by many users) pay the full cost. This implements Item 3, Tier 1 of the perf plan: a container-local in-memory response cache.

Changes

  • playground-server/GenerationCache.cs (new)IGenerationCache + MemoryGenerationCache backed by IMemoryCache. ComputeKey(generatorName, codeModel, configuration, generatorVersion) returns a SHA-256 hex over a length-prefixed concatenation (avoids cross-boundary collisions like "Foo"+"Bar" vs "FooBar"+""). Entry Size = body byte length; backing cache SizeLimit = 256 MB. Entries are bounded by SizeLimit-driven compaction only — there is no TTL, since a new generator binary deploy implicitly invalidates the cache via the version-keyed entries.
  • playground-server/Program.cs — registers AddMemoryCache + IGenerationCache singleton. Reads the generator DLL's FileVersionInfo.FileVersion once at startup and folds it into the key, so a deploy of a new binary implicitly invalidates every entry. /generate short-circuits on hit (returns cached bytes, skips the subprocess) and serializes-once-then-caches on miss. Responses carry X-Cache: HIT|MISS. Cache hits are recorded in telemetry: a hit emits the PlaygroundGenerate event with outcome cache_hit, and every request carries a cacheStatus (hit|miss) property through the existing telemetry pipeline.
  • playground-server.Tests/ (new NUnit project, 14 tests) — key determinism & format; sensitivity to each of the four components (including version → invalidation); length-prefix unambiguity; null-argument guards; get/miss/set/overwrite round-trip; SizeLimit compaction.

Cache flow

var cacheKey = MemoryGenerationCache.ComputeKey(generatorName, body.CodeModel!, body.Configuration!, generatorVersion);
if (cache.TryGet(cacheKey, out var cached) && cached is not null)
{
    request.HttpContext.Response.Headers["X-Cache"] = "HIT";
    telemetryProperties["cacheStatus"] = "hit";
    TrackGenerateEvent("cache_hit");
    return Results.Bytes(cached.Body, cached.ContentType); // skips dotnet subprocess
}
request.HttpContext.Response.Headers["X-Cache"] = "MISS";
telemetryProperties["cacheStatus"] = "miss";
// ...run generator, then:
var responseBytes = JsonSerializer.SerializeToUtf8Bytes(new GenerateResponse(files), GenerateJsonContext.Default.GenerateResponse);
cache.Set(cacheKey, new CachedGenerationResponse(responseBytes, "application/json"));
return Results.Bytes(responseBytes, "application/json");

Out of scope

Tier 2 file-based cache and Items 1/2/4/5 of the issue. Generator internals are untouched; production tsp codegen is unaffected.

…d-server

Agent-Logs-Url: https://github.com/microsoft/typespec/sessions/693fd6ec-31da-4c6f-a4be-4324b946a6f4

Co-authored-by: jorgerangel-msft <102122018+jorgerangel-msft@users.noreply.github.com>
@microsoft-github-policy-service microsoft-github-policy-service Bot added the emitter:client:csharp Issue for the C# client emitter: @typespec/http-client-csharp label May 18, 2026
Copilot AI changed the title [WIP] Enhance performance of C# emitter in TypeSpec playground [http-client-csharp] Add Tier 1 in-memory response cache to playground-server May 18, 2026
Copilot AI requested a review from jorgerangel-msft May 18, 2026 20:22
Comment thread packages/http-client-csharp/playground-server/Program.cs Outdated
Agent-Logs-Url: https://github.com/microsoft/typespec/sessions/656e84a4-17f1-4c10-87de-4f80c2df8412

Co-authored-by: jorgerangel-msft <102122018+jorgerangel-msft@users.noreply.github.com>
@JoshLove-msft

Copy link
Copy Markdown
Contributor

@copilot resolve merge conflicts

Comment thread packages/http-client-csharp/playground-server/GenerationCache.cs Outdated
…p-emitter-performance

# Conflicts:
#	packages/http-client-csharp/playground-server/Program.cs

Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
@JoshLove-msft

Copy link
Copy Markdown
Contributor

is there a way we can test this in the playground before we deploy it?

Kicked off a pipeline run

JoshLove-msft and others added 4 commits June 2, 2026 09:24
Adds four GenerationCacheTests cases that drive ComputeKey with a
multi-KB code model/configuration shaped like real generator input:
determinism, key change on a semantic code-model edit, key change on a
configuration edit, and a cache round-trip. Addresses PR review feedback
requesting coverage with a realistic code model.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Follows the generator test projects' TestData convention: the realistic
code model and configuration now live in
TestData/GenerationCacheTests/*.json (copied to the output directory) and
are loaded via a ReadTestData helper, rather than as inlined string
constants in the test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread packages/http-client-csharp/playground-server/GenerationCache.cs Outdated
…ackage versions

- Split GenerationCache.cs into CachedGenerationResponse.cs,
  IGenerationCache.cs and MemoryGenerationCache.cs (one public type per file).
- Make the memory cache SizeLimit configurable via app settings
  (GenerationCache:SizeLimitBytes) with the 256 MB default, surfaced in a
  new appsettings.json.
- Drop hardcoded package versions from playground-server.Tests.csproj and
  consume the central versions from generator/Packages.Data.props instead
  (adding a Microsoft.Extensions.Caching.Memory entry there).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@JoshLove-msft
JoshLove-msft enabled auto-merge June 2, 2026 18:55
which@7.0.0 raised its Node engine floor to ^22.22.2 || ^24.15.0, which
broke CI: the emitter pipeline's primary Node (24.x) resolves to 24.14.1
on the hosted agents, failing the engine-strict pnpm install. which@6.0.1
(engines: ^20.17.0 || >=22.9.0) is API-compatible and satisfied by every
Node version in the build matrix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
JoshLove-msft and others added 2 commits June 2, 2026 12:36
The pinned major-version specs resolved to a stale cached Node on the
agents (24.14.1), which failed engine-strict checks for dependencies
requiring >=24.15.0. checkLatest fetches the newest matching release.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
JoshLove-msft and others added 2 commits June 3, 2026 09:40
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@JoshLove-msft
JoshLove-msft added this pull request to the merge queue Jun 3, 2026
Merged via the queue into main with commit 388961b Jun 3, 2026
53 checks passed
@JoshLove-msft
JoshLove-msft deleted the copilot/improve-csharp-emitter-performance branch June 3, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

emitter:client:csharp Issue for the C# client emitter: @typespec/http-client-csharp eng

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Brainstorm perf improvements for running c# emitter in playground

5 participants