Add constrained autoregressive surface realizer (VSS3-05, SLM-73)

docs/design/verified-scope-solver.md

@@ -484,6 +484,30 @@ the closure.
   certificate.
 - Semantic fields never take the autoregressive path.
 
+## Implemented constrained autoregressive surface realizer (VSS3-05 / SLM-73)
+
+[`models/surface_autoregressor.py`](../../src/slm_training/models/surface_autoregressor.py)
+adds a small standalone causal byte-decoder with printable-ASCII byte tokens,
+grammar-aware `IdentifierConstraint`, permissive `DecorativeConstraint`, prompt
+context encoding, and constrained greedy/top-k generation. It is kept separate
+from the main TwoTower/diffusion paths so it cannot become a second program
+generator.
+
+[`dsl/neural_surface_realizer.py`](../../src/slm_training/dsl/neural_surface_realizer.py)
+wraps the model as a `SurfaceRealizer`. It accepts only `SURFACE_ONLY` slots,
+only `INTERNAL_IDENTIFIER` and `DECORATIVE_TEXT` kinds, generates each slot
+under its own constraint mask, and falls back to the deterministic baseline
+per slot on dead end, invalid proposal, model error, or missing model.
+`realize_surface_and_verify` still canonicalizes and re-verifies the final
+program with the pack oracle. The neural realizer is a caller-supplied
+`SurfaceRealizer`, not a new pack slot; the default pipeline keeps
+`DeterministicSurfaceRealizer`.
+
+This is fixture wiring only (`tests/test_models/test_surface_autoregressor.py`,
+`tests/test_dsl/test_neural_surface_realizer.py`, and
+`tests/test_data/test_surface_rows.py`). The deterministic baseline remains the
+default; no full train/eval run, checkpoint, or ship gate is claimed.
+
 ## Relationship to existing implementation
 
 | Symbol (file:line) | Existing role | Disposition under this contract |

docs/design/vss3-05-autoregressive-surface-realizer.md

@@ -0,0 +1,140 @@
+# VSS3-05: Constrained autoregressive surface realizer with deterministic fallback
+
+**Issue:** SLM-73
+
+**Status:** wiring / fixture evidence. No trained production checkpoint, no
+full-corpus train/eval run, and no `--ship-gates` claim. Deterministic surface
+realization remains the default and baseline.
+
+## What was added
+
+A small, standalone causal byte-autoregressor for surface-only slots, kept
+outside the main TwoTower/diffusion program generator so it cannot accidentally
+become a second generator path.
+
+`src/slm_training/models/surface_autoregressor.py`:
+
+- `SurfaceByteVocab` — printable-ASCII byte vocabulary (`<pad>`, `<bos>`,
+  `<eos>`, `<unk>`, `B:xx`). Any lowercase identifier or decorative ASCII text
+  can be spelled token-by-token without a corpus-bound tokenizer.
+- `IdentifierConstraint` and `DecorativeConstraint` — constrained next-token
+  sets for OpenUI-style internal identifiers and decorative text. Enforce
+  grammar, reserved-word avoidance, peer uniqueness, and byte budgets at decode
+  time.
+- Small causal decoder (`CausalSelfAttention`, `SurfaceTransformerBlock`,
+  `SurfaceContextEncoder`, `SurfaceAutoregressor`) with prompt context encoding,
+  cross-attention, weight tying, and constrained greedy/top-k generation.
+- `train_surface_autoregressor` — tiny fixture trainer for (prompt, target)
+  pairs, used by tests and future harnesses.
+- Save/load and `from_records` for API symmetry.
+
+`src/slm_training/dsl/neural_surface_realizer.py`:
+
+- `NeuralSurfaceRealizerConfig` — runtime knobs (`model`, `model_path`,
+  `device`, `max_bytes`, `temperature`, `top_k`, `seed`, `fallback_to_deterministic`).
+- `NeuralSurfaceRealizer` — a `SurfaceRealizer` backed by the AR model.
+  - Only accepts slots already classified as `SURFACE_ONLY`.
+  - Only supports `INTERNAL_IDENTIFIER` and `DECORATIVE_TEXT` kinds.
+  - Generates each slot independently under its own constraint mask.
+  - Falls back to `DeterministicSurfaceRealizer` per slot on dead end, invalid
+    proposal, model error, or when no model is configured.
+  - Rejects non-`SURFACE_ONLY` slots before the model is invoked.
+
+`src/slm_training/harnesses/model_build/config.py`:
+
+- Default-off AR fields for API symmetry and future harness wiring:
+  `surface_realizer`, `surface_ar_enabled`, `surface_ar_d_model`,
+  `surface_ar_n_layers`, `surface_ar_n_heads`, `surface_ar_max_bytes`,
+  `surface_ar_temperature`, `surface_ar_top_k`, `surface_ar_fallback`,
+  `surface_ar_verify_retry`. Deterministic realization remains the default.
+
+`src/slm_training/dsl/schema.py`:
+
+- Added `"surface_realization"` to `TASK_TOKENS` so derived training records can
+  carry the correct task label.
+
+`src/slm_training/data/progspec/surface_rows.py`:
+
+- `derive_surface_realization_records(spec, *, realizer, opaque_bindings,
+  include_authorities)` — derives one `ExampleRecord` per realized surface slot
+  from a verified `ProgramSpec`, preserving `split` and `split_group_id`.
+  Defaults to the deterministic baseline when `realizer` is `None` and only
+  emits authorities in `include_authorities` (`SURFACE_ONLY` and
+  `OPAQUE_USER_VALUE` by default).
+
+Regression tests:
+
+- `tests/test_models/test_surface_autoregressor.py` — byte vocab round-trip,
+  identifier/decorative constraints, fixture overfit, save/load, `from_records`
+  (torch-only; skipped where torch is unavailable).
+- `tests/test_dsl/test_neural_surface_realizer.py` — neural realizer plugged into
+  `realize_surface_and_verify`, trained-identifier verification, no-model
+  deterministic fallback, dead-end fallback, disabled fallback error,
+  authority/kind rejection. The no-model and guard paths are torch-free; the
+  model-backed paths are torch-only.
+- `tests/test_data/test_surface_rows.py` — `derive_surface_realization_records`
+  emits one record per surface slot, respects `include_authorities`, inherits
+  split/group (torch-free), and can use a trained neural realizer (torch-only).
+
+## What is intentionally not added
+
+- The AR realizer is **not** registered as a full `ModelPlugin` in
+  `factory.py`. That would create a second program generator, which this issue
+  forbids.
+- The `ModelBuildConfig` fields are wiring placeholders; no train/eval harness
+  automatically instantiates or trains the AR realizer yet.
+- No production checkpoint, no full-corpus run, and no ship-gate claim.
+
+## Design boundaries preserved
+
+- `DeterministicSurfaceRealizer` remains the default.
+- Autoregression is allowed only for slots the pack classifier marks as
+  `SURFACE_ONLY`.
+- Each slot is generated under a hard constraint mask; the final program is
+  still canonicalized and re-verified by `pack.oracle`.
+- Fallback is per-slot and transparent in assignment provenance.
+- Fixture-only; meaningful-parse and full ship gates remain the production bar.
+
+## End-to-end example
+
+Input (solved semantic IR, placeholders preserved):
+
+```text
+root = Stack([title], "column")
+title = TextContent(":hero.title")
+```
+
+A tiny fixture-trained model maps the binder slot prompt
+`kind=internal_identifier authority=surface_only slot_id=openui:binder:title
+symbol=title max=64` to the identifier `title`. `realize_surface_and_verify`
+substitutes `title` for the semantic symbol, splices the opaque content
+`:user.title`, canonicalizes, and verifies:
+
+```text
+root = Stack([v0], "column")
+v0 = TextContent(":user.title")
+```
+
+The assignment carries the model-chosen value and provenance:
+
+```text
+SurfaceAssignment(
+    slot_id='openui:binder:title',
+    value='title',
+    provenance='autoregressive',
+)
+```
+
+If the model is absent or fails, the same slot falls back to deterministic
+`v0` with provenance `autoregressive_fallback:<reason>:deterministic:canonical_name`.
+
+## Caveats
+
+- The model is a scratch causal decoder; real capacity and corpus training are
+  future matrix work.
+- `DecorativeConstraint` accepts all printable ASCII, but OpenUI V1 decorative
+  text slots are currently classified as `OPAQUE_USER_VALUE`, so the AR path for
+  decorative text is wired but not exercised end-to-end by the default pack.
+- Constrained generation is greedy/top-k; sampling temperature > 0 is supported
+  but not ship-gated.
+- No checkpoint, no full `--ship-gates` run, and no production claim.

src/slm_training/data/progspec/surface_rows.py

@@ -0,0 +1,104 @@
+"""Surface-realization training rows for VSS3-05 (SLM-73).
+
+Derives ``ExampleRecord`` rows from verified ``ProgramSpec`` roots after split
+assignment. Each row represents one surface slot realization target. Split and
+group identity are inherited from the parent spec so leakage checks remain
+intact.
+"""
+
+from __future__ import annotations
+
+import hashlib
+from typing import Any
+
+from slm_training.data.progspec.schema import ProgramSpec, emit_record
+from slm_training.dsl.surface import (
+    SurfaceAuthority,
+    SurfaceRealizer,
+    realize_surface_and_verify,
+    resolve_surface_slot_extractor,
+)
+
+
+def _semantic_fingerprint(spec: ProgramSpec) -> str:
+    """Stable fingerprint of the solved semantic IR."""
+    payload = f"{spec.program_family_id}:{spec.lineage_id}:{spec.canonical_openui}"
+    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]
+
+
+def _surface_prompt(spec: ProgramSpec, slot: Any) -> str:
+    """Short prompt describing the slot for training/conditioning."""
+    parts = [
+        f"kind={slot.kind.value}",
+        f"symbol={slot.semantic_symbol_id or ''}",
+        f"slot={slot.slot_id}",
+    ]
+    return " ".join(parts)
+
+
+def derive_surface_realization_records(
+    spec: ProgramSpec,
+    *,
+    realizer: SurfaceRealizer | None = None,
+    opaque_bindings: dict[str, Any] | None = None,
+    include_authorities: frozenset[str] | None = None,
+) -> tuple[Any, ...]:
+    """Derive one ``ExampleRecord`` per realized surface slot.
+
+    The deterministic baseline is used when ``realizer`` is ``None``. Only slots
+    whose authority is in ``include_authorities`` are emitted (default:
+    ``SURFACE_ONLY`` and ``OPAQUE_USER_VALUE``).
+    """
+    from slm_training.dsl.pack import get_pack
+
+    include_authorities = include_authorities or frozenset(
+        {SurfaceAuthority.SURFACE_ONLY.value, SurfaceAuthority.OPAQUE_USER_VALUE.value}
+    )
+    pack = get_pack("openui")
+    result = realize_surface_and_verify(
+        spec,
+        pack=pack,
+        realizer=realizer,
+        opaque_bindings=opaque_bindings or {},
+        semantic_ir_fingerprint=_semantic_fingerprint(spec),
+        prior_status="verified",
+    )
+    if result.status not in {"solved", "verified"}:
+        return ()
+
+    assignment_by_slot = {a.slot_id: a for a in result.assignments}
+    # Main resolves the OpenUI extractor via the pack_id registry; the pack does
+    # not expose a ``surface_slot_extractor`` attribute directly.
+    extractor = resolve_surface_slot_extractor(pack)
+    if extractor is None:
+        return ()
+    slots = extractor(spec.canonical_openui)
+    records: list[Any] = []
+    for slot in slots:
+        if slot.authority.value not in include_authorities:
+            continue
+        assignment = assignment_by_slot.get(slot.slot_id)
+        if assignment is None:
+            continue
+        prompt = _surface_prompt(spec, slot)
+        record_id = f"{spec.id}_surface_{slot.slot_id}_{hashlib.sha256(assignment.value.encode()).hexdigest()[:8]}"
+        record = emit_record(
+            spec,
+            prompt=prompt,
+            task="surface_realization",
+            openui=result.source,
+            record_id=record_id,
+            source="surface_realization",
+            determinacy="deterministic",
+            tier="Silver",
+            provenance={
+                "surface_slot_id": slot.slot_id,
+                "surface_assignment": assignment.to_dict(),
+                "surface_result_status": result.status,
+            },
+        )
+        records.append(record)
+    return tuple(records)
+
+
+__all__ = ["derive_surface_realization_records"]