DSL Diffusion SLM Research Program — Wave 1 foundations (E3 lineage, A1 emptiness probe)#259
Merged
Tyler-R-Kendrick merged 15 commits intoJul 17, 2026
Merged
Conversation
24 new Adjacent-tagged sources for the Tracks A-G research program (sketch-then-fill, macro/library induction, binding theory, negative results on identifier anonymization and alpha-equivalence hashing, diffusion adaptation baselines, reasoning baselines, self-improvement loops) in resources/autoresearch/dsl-program-sources.json, loadable via scripts/autoresearch.py --source-manifest. research-lineage.md gains a program section with cluster boundaries and explicit novelty positioning; regression test guards manifest size, tags, and no-overlap with earlier manifests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
New evals/emptiness_probe.py scores the model's fully-masked reconstruction
NLL of the gold populated program vs a deterministic minimal-valid program,
decomposing empty-preference into total NLL (length-bias-exposed, what a
score-ranking constrained decoder compares) and per-token NLL (length-
controlled). Verdict separates decode-time length-bias/constraint-distortion
(fix in Track A) from content-modeling failure (Tracks B/C/D). CLI
scripts/probe_emptiness.py with honest AgentV envelope (diagnostic_not_ship).
Fixture-grade run over the committed playground_demo checkpoint documented in
docs/design/iter-e248-emptiness-probe-20260716.{md,json}; the real E224+
verdict requires the local frontier checkpoints (gitignored). MODEL_CARD
history notes the diagnostic. Four unit tests; full changed-file suite green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 24 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (54)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Adds a per-suite component_type_recall floor to DEFAULT_SHIP_GATES so a compression- or decode-driven change cannot green the gates with shorter-but-emptier output on syntax alone: the trivial/empty program recovers ~0 of the gold component types and now fails. Floors sit at or below each suite's structural bar (density must be at least as present as structure); the change only makes the policy stricter, never weaker. The metric is surfaced in the gates 'actual' block for transparency. Gate tables in adversarial-review.md and the honest-ship-eval skill updated in lockstep. Three focused tests: floor present for every suite, valid+ structural+low-recall now fails, populated run still passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
New evals/semantic_bits.py measures a corpus's information content over
grammar choice points only, tokenizer-independently: corpus unigram
description length over the ProductionCodec choice stream vs the surface
lexeme stream, with bits_per_decision (empirical entropy) and a
params_per_bit(n) helper for the B3 capacity ladder. CLI
scripts/measure_semantic_bits.py.
On the committed eval corpus (n=19): externalizing the grammar cuts total
choice bits 1.88x (production 4.32 vs surface 5.40 bits/decision, 1.51x
fewer tokens) — the quantitative form of 'externalized grammar shrinks the
learning target'. Documented in
docs/design/iter-e249-semantic-bits-20260716.{md,json}. 7 unit tests.
Also completes the E2 semantic-density gate fixtures: two AgentV ship
tests construct a passing smoke suite and now supply component_type_recall
(a metric real scoreboards emit) so smoke still passes all bars; their
fail-closed intent is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
Extends the compiler completion forest from proving legality to proving minimum content: build_completion_forest gains a min_content gate at the EOS-admission chokepoint so a grammatically complete but empty/underfull layout is not a legal completion while the grammar still offers a way to add content (guarded to never dead-end). Content is measured by emitted_component_count via the tokenizer's compiler-derived component symbol space (no AST parse / Node bridge). New TwoTowerConfig.decode_min_content (0 off | >0 floor | -1 auto from distinct slot-contract roots via _effective_min_content), threaded through ModelBuildConfig and factory so any matrix row can set it. Recorded risk: hard floors can force hallucinated components — A3 coverage-energy remask is the softer sibling; the v9 E-row comparison measures which wins. Tests: forest withholds/admits EOS by floor with no dead-end; _effective_min_content auto/fixed/off; end-to-end config threading; iter-e250 doc. test_compiler_decode + test_lattice_search green (38); full changed-file suite 272 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
…grade) Executes the previously plan-only V9 compiler-lattice campaign end-to-end through scripts/run_quality_matrix.py on CPU with the honest five-suite scoreboard and per-row AgentV publication. New --eval-checkpoint flag: the V9 rows declare initialization=eval_only (decode-policy comparison over identical checkpoint lineage), but the matrix classifier silently reclassified them to parent/scratch, retraining a model per row. The flag routes declared eval-only rows through one frozen checkpoint; regression tests cover the declaration and the routing. Campaign: E240 trained as the explicit scratch control (800 steps, fixture v1 corpus, 108 records); E241-E247 evaluated eval-only from E240's frozen checkpoint. All rows fail honest gates (syntax/meaningful parse 0.0 from placeholder-policy rejections - genuine fixture-scale capacity, bridge verified). Diagnostics: triggered rows reproduce greedy byte-for-byte with zero bottoms/rollbacks; always-on PTRM (E244) fires 1680 triggers at ~3x latency and lowers structural similarity on every suite - selective over always-on stochasticity at fixture scale. Evidence: iter-e240-e247-lattice-campaign-20260716.md + merged campaign JSON; V9 section of quality-experiment-matrix.md now carries measured results; MODEL_CARD history row appended. Wiring evidence only - no ship claim; the frontier-scale run on local E224+ checkpoints (GPU, rico_held 1500) stays open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132PYCBHb1t1pjhDRzLrPov
Adds select_remask_coverage_indices (parallel_decode) mirroring the CoRe instability policy but ranking by a per-position content-coverage deficit, and wires it as remask_policy='coverage' in the TwoTower MaskGIT remask dispatch. TwoTowerModel._coverage_deficit scores non-content (filler) positions by 1 - content_fraction from the tokenizer's compiler-derived component/sym/bind spaces, so a content-sparse layout preferentially remasks filler for another decode pass toward missing inventory content; dense rows and kind_ids-less tokenizers degrade gracefully to confidence remasking. Soft sibling of A4's hard min-content contract; no new config field, no extra forward. Tests: pure-policy ranking/fallback/budget/BOS + model-level deficit scoring (content 0, filler >0). Regression across v6/v7 remask + compiler decode green (58); full changed-file suite 198 passed. Doc iter-e251. No checkpoint/ship claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
…atch New dsl/canonicalize.py maps each layout equivalence class (binder renaming, statement order, style literals) to one representative string via a confluent production-codec round-trip (encode_openui -> decode_productions): canonical statement order, De Bruijn-style binder pool, style stripped, validated through the official parser, idempotent. Honestly a normal-form canonicalizer, not an e-graph/equality-saturation engine and not a semantic simplifier (no default elision / container flattening). Documents the context-sensitive alpha-equivalence caveat (2401.02948) for C3. evals/canonical_match.py consumes it: canonical vs surface exact-match with a canonicalization_rescued count, beside surface match, never a ship gate. Feeds D1 (forward-process simplifier) and C3 (repeated-span fingerprint). 9 tests (alpha-equivalence, idempotence, validation, rescue accounting); full changed-file suite 240 passed. Doc iter-e252. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
New data/diffusion/simplify.py pre-canonicalizes ExampleRecord targets (via the D2 canonicalizer) before the corruption pipeline tokenizes and masks them, so every noised intermediate the denoiser sees is a noised canonical tree — the reverse model learns one canonical form per layout equivalence class instead of surface variation it should ignore. Reverse- loop simplification would instead push mid-trajectory states off- distribution. Never drops a record (unparseable targets pass through); placeholders and all fields preserved. simplify_records reports distinct_canonical_targets (the equivalence-class count = the cleaner learning target). Tests: canonical/valid/idempotent target, alpha-variants collapse to one target, pass-through on unparseable, placeholder preservation, corpus collapse stats. 5 tests; full changed-file suite 225 passed. Doc iter-e253. Forward-vs-post-hoc X-row comparison is a GPU-host grammar- matrix run, not claimed here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
Scopes the third DSL target (patterns/algorithms/data-structures) with the oracle problem front-and-center: unlike OpenUI (parser+renderer) and GraphQL (graphql-js+schema), a pattern DSL has no free validity oracle, so 'correct' is a modeling decision made before any training. Recommends shipping the first pack on a typed well-formedness oracle (reward labeled well_formed_not_behavioral, analogous to OpenUI syntax!=meaningful), with executable property tests as a documented gate upgrade. Maps the DSL into the existing program machinery (typed-AST generation, placeholders, De Bruijn scope, D2 canonicalizer, D1 forward simplification) and the F1 pack contract. Design-only; no code, corpus, or ship claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
Scopes the fourth DSL family (expert nomenclatures / ontologies) and the pack-contract variant it needs: unlike the CFG packs, a nomenclature is a flat 10^4-10^6 concept graph whose validity oracle is an ontology- consistency check, not a parser, and whose constrained decode is a graph-walk (mask to concepts reachable by a legal relation edge), not a DFA. Frames it as the strongest test of schema-as-symbol-table (F2) and dynamic pseudo-embeddings (C2), with a small openly-licensed pilot ontology first (not SNOMED) so the fixture loop runs on CPU. Design-only; no ontology, code, or ship claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
New models/hf_denoiser.py adapts the pretrained SmolLM2-135M causal LM into the denoiser tower (denoiser_backend=hf): full bidirectional visibility via an explicit 4D attention mask (only DiffuLLaMA's drop-the-causal-mask move; no annealing/shift/training recipe), fresh OpenUI-vocab embeddings with weight-tied lm_head, and context-tower hiddens prepended as projected prefix states. Implements the exact DenoiserTower interface so training, every decode path, and checkpoint round-trips work unchanged. V10 matrix rows: matched pair differing only in denoiser_backend - E255 from-scratch control (1.1M) vs E256 adaptation (135M), parallel MaskGIT decode on both. Fixture-grade run (200 CPU steps, fixture v1 corpus, honest five-suite scoreboard + AgentV): both fail honest gates; the adaptation trails the scratch control on every secondary signal (loss 8.51 vs 3.75), and an lr=3e-5 probe converges even less (9.72), so the deficit is budget, not just LR. B4 verdict stays OPEN - fixture scale can neither confirm nor kill the from-scratch assumption; the decisive run needs GPU-scale matched-compute arms with per-arm LR selection. Tests: bidirectionality, encode/project candidate equivalence, backbone gradient flow + checkpoint round-trip, V10 matched-pair registration; also makes the checkout-relative suite-loader fallback test hermetic (it failed whenever a real outputs/data/eval/v1 had been built). Evidence: iter-e255-e256-b4-ar-adaptation-20260716.md + v10 campaign JSONs; quality-experiment-matrix.md V10 section; research-lineage DiffuLLaMA tag Adjacent->Adapted; MODEL_CARD history row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132PYCBHb1t1pjhDRzLrPov
bind_encoding=relative on the lexer-native tokenizer: definition sites become nameless <BINDDEF> tokens (statement position IS the identity - the model never invents or even selects a binder name) and reference sites carry signed statement deltas <BINDREL_+/-k> (OpenUI allows forward references, so classic most-recent-binder De Bruijn indices do not apply). Decode inverts deltas positionally; out-of-scope offsets decode to a never-defined oob<k> name that stream_check flags unresolved - scope legality is the verifier's job, never silently repaired. New tokens keep TokenKind.BIND, so the grammar gate's NAME expansion and factorized embeddings work unchanged. Encoding persisted in the tokenizer sidecar; plumbed through TwoTowerConfig/ModelBuildConfig/Experiment with absolute as the default (zero change to existing rows). Also fixes a real alpha-invariance bug the C1 property test exposed in the D2 canonicalizer: production_codec._statement_order read binder references out of string-literal contents, so placeholder text like ":form.title" aliased the binders form/title and canonical statement order leaked original names. String literals are now stripped before reference scanning; regression test added and a correction note appended to the D2 iter doc. V10 row E257 (matched vs E255, only bind_encoding differs). Fixture run (200 CPU steps, fixture v1 corpus, honest suites + AgentV): syntax parse 0.667/0.6/0.25/0.5 vs 0.0 on the matched absolute control, loss 3.27 vs 3.75, decode p50 1-8s vs ~15s; meaningful parse 0.0 on both - failures shift to empty_root_stack, i.e. the valid-but-empty wall (Track A) is untouched, as expected. Wiring evidence only; frontier E-row and C1xA interaction stay open. Round-trip property test over every committed seed program via the lang-core bridge plus 7 unit tests; full changed-file suite green (320 passed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132PYCBHb1t1pjhDRzLrPov
Both branches claimed the v10 quality-matrix name: main's exact-state local-preference rows (E248-E254) keep v10; this branch's B4/C1 representation rows (E255-E257) become v11 (script, tests, docs retitled; historical result-JSON filenames unchanged). Code conflicts resolved as unions (config fields, factory wiring, compiler_draft helpers, twotower backend switch + isolated aux init); production_codec keeps main's AST-based _statement_order. MODEL_CARD history and quality-experiment-matrix keep both campaigns' rows. passing_evaluation() fixture now supplies component_type_recall so the RL-readiness gate test matches this branch's E2 semantic-density floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132PYCBHb1t1pjhDRzLrPov
Keeps #274's repaired E252 counterfactual pipeline (local_decisions evidence outputs, train_preference --evidence-out, corrected E252 docs) and this branch's v11 rows, A4 min-content decode, and HF denoiser backend switch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132PYCBHb1t1pjhDRzLrPov
Tyler-R-Kendrick
deleted the
claude/dsl-diffusion-slm-autoresearch-dg16mx
branch
July 17, 2026 00:56
Tyler-R-Kendrick
added a commit
that referenced
this pull request
Jul 17, 2026
Salvaged from the superseded B4/C1/C2 branch (old head 9e86c13; the lever implementations there were superseded by the E255-E257 work merged via #259). Both defects are latent on main because no merged row activates runtime_symbol_features yet; both fire the moment one does: - evaluate_loss_suites now clears request-local runtime symbol features on entry: the teacher-forced suites carry no request context, and stale per-batch features from the last training batch crash the batched NLL forward (batch-dim mismatch) or silently bias it when dimensions happen to line up - DenoiserTower.project accepts [D]/[N,D] hidden slices with features active (compiler/tree scorers project slices): shape-polymorphic for a single active request, fail-closed when multi-request features cannot be attributed to a sliced row Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014AWAXqSioAB3sYXKxgJ3rG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the first items of the DSL Diffusion SLM Research Program (Linear project
DSL Diffusion SLM Research Program, team slm-training). Work continues on this branch/PR across the program's waves; this is the running PR.Context
The program builds DSL-specific diffusion SLMs (OpenUI first, then GraphQL, a design-patterns DSL, and expert nomenclatures) inside the existing autoresearch loop. The binding constraint from
docs/MODEL_CARD.md(E224–E236) is that syntax validity is solved (parse-syntax 1.0) but meaningful parse ≈ 0 — models emit valid-but-empty layouts. Wave 1 lays the measurement/diagnosis foundation before any representation or decode changes.Changes so far
E3 — Research-lineage intake (SLM-33) — commit
9d46032src/slm_training/resources/autoresearch/dsl-program-sources.json: 24 prior-art sources (all tagged Adjacent), loadable viascripts/autoresearch.py --source-manifest, feeding the hypothesizer'sEvidenceSnapshot.docs/design/research-lineage.mdgains a "DSL diffusion research program (Tracks A–G)" section with cluster boundaries, engaged negative results, and explicit novelty positioning.Adjacenttags, and zero URI overlap with the earlier committed manifests.A1 — Emptiness probe (SLM-20) — commit
6924a2asrc/slm_training/evals/emptiness_probe.py: scores the model's fully-masked reconstruction NLL of the gold populated program vs a deterministic minimal-valid program, decomposing empty-preference into total NLL (length-bias-exposed — what a score-ranking constrained decoder compares) and per-token NLL (length-controlled). Verdict separateslength_bias_constraint_distortion(decode-time fix) fromcontent_modeling_failure(representation/training).scripts/probe_emptiness.pyCLI with an honest AgentV envelope (claim=diagnostic_not_ship).playground_democheckpoint documented indocs/design/iter-e248-emptiness-probe-20260716.{md,json}; MODEL_CARD history notes the diagnostic.Verification
tests/test_autoresearch/test_harness.py) and 4 emptiness-probe tests (tests/test_evals/test_emptiness_probe.py)..githooks/check-changed: 161 passed, 1 skipped.python -m scripts.repo_policyok;git diff --checkclean.Honesty
All runs here are fixture/CPU-scratch — wiring evidence only, not ship claims. No ship gate is weakened and nothing is promoted. The real A1 verdict for the E224+ wall requires running the probe against the local frontier checkpoints (gitignored, absent from a fresh clone) on a GPU host.
🤖 Generated with Claude Code
https://claude.ai/code/session_015M3mERHzsfxy561zTeJfNA
Generated by Claude Code