Use JSON instead of YAML for typespec-python code model serialization

[msyyc]

Problem

Today @typespec/http-client-python exchanges the code model between the TypeScript emitter and the Python generator as YAML. This is very slow and dominates end-to-end generation time, especially under Pyodide where PyYAML runs as pure Python.

Measured on the Azure network spec (large real-world spec):

Stage	YAML (js-yaml / PyYAML)
JS serialize	~1000 ms
Python load	~11000 ms
Python dump	~5500 ms
File size	14.4 MB

Total serialization overhead: ~29 s.

Switching to a faster JS YAML library (yaml/eemeli) does not help — it is ~1.8x slower to serialize and ~70x slower to parse than js-yaml, and it rejects the model by default due to the alias-count guard. The bottleneck is the Python side, which only a non-YAML format fixes.

Proposal

Replace YAML with a reference-preserving JSON format, using the same $id/$ref convention as the C# emitter (System.Text.Json ReferenceHandler.Preserve):

• First time an object is seen: { "$id": "N", ...properties }
• First time an array is seen: { "$id": "N", "$values": [...] }
• Any later reference to an already-seen node: { "$ref": "N" }

The code model is a cyclic/shared object graph (on the network spec: 35,175 objects and 32,075 arrays, of which 3,739 objects and 2,855 arrays are shared, plus 1,022 cycle back-edges). Because $id must be registered before recursing into a node's children, cycles are encoded as a $ref back to the enclosing node, so cycles are preserved (the C# writer drops them, which the Python model cannot tolerate). The Python decoder then reconstructs a graph with identical shared identity and cycles, so the generator's model layer needs zero changes.

Expected result: ~29 s -> ~0.2 s and file size 14.4 MB -> 8.8 MB.

Work items

• Add emitter/src/code-model-serializer.ts — cycle-safe $id/$ref serializer
• Add generator/pygen/_codemodel_json.py — matching Python loads/dumps decoder/encoder
• Drop js-yaml/@types/js-yaml (emitter) and PyYAML/types-PyYAML (generator) dependencies
• Rename the emit-yaml-only emitter option to emit-codemodel-only
• Update CI regenerate scripts, batch runner, mypy config, and ARCHITECTURE.md

Validation criteria

• Full package build (emitter + pygen wheel)
• End-to-end: compile the network spec -> $id/$ref code model -> generator produces correct output
• Python codec reconstructs an identical graph (structural metrics match) and round-trips stably
• Generator unit tests, lint, format, and mypy pass

Reference

POC: #11178

[github-actions]

🎯 Issue Triage

This is a performance improvement proposal to replace YAML with a reference-preserving JSON format ($id/$ref) for the code model exchange between the TypeScript emitter and Python generator in @typespec/http-client-python, reducing serialization overhead from ~29s to ~0.2s.

Category: Feature Request

Area: emitter:client:python

🔍 Analysis

• The bottleneck is Python-side YAML deserialization (~11s load + ~5.5s dump) on large specs like Azure network
• The code model contains cyclic/shared object graphs which require a reference-preserving format — standard JSON is insufficient
• Proposed $id/$ref convention is compatible with the C# emitter's ReferenceHandler.Preserve approach
• A POC PR already exists: POC: Use JSON instead of YAML for typespec-python code model serialization #11178
• Work items are clearly defined: new serializer in TS, decoder in Python, dependency removals, option renames, and doc/CI updates
• Zero changes needed to the generator's model layer since graph identity is preserved

📝 Suggested next steps

• Review the POC at POC: Use JSON instead of YAML for typespec-python code model serialization #11178 to validate correctness of cycle handling
• Implement emitter/src/code-model-serializer.ts and generator/pygen/_codemodel_json.py
• Remove js-yaml / PyYAML dependencies
• Rename emit-yaml-only option to emit-codemodel-only
• Validate end-to-end on the network spec

Generated by Agentic Triage