feat(efs4-04): SLM-140 causal synthesis manifest, loader, CLI, tests and docs#543
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 40 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (40)
📝 WalkthroughWalkthroughAdds an Evidence-First Semantic SLM campaign manifest, typed synthesis harness, CLI, evidence graph outputs, causal diagnosis reports, validation tests, and related design metadata and version-registry updates. ChangesEFS4-04 causal synthesis
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant SynthesisHarness
participant DesignManifests
participant OutputFiles
CLI->>SynthesisHarness: Load campaign manifest
SynthesisHarness->>DesignManifests: Scan and match iter-*.json results
DesignManifests-->>SynthesisHarness: Result manifests and hypothesis states
SynthesisHarness-->>CLI: Validated synthesis and evidence graph
CLI->>OutputFiles: Write JSON, Markdown, Mermaid, and DOT outputs
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Tyler-R-Kendrick
force-pushed
the
agent/slm-140
branch
from
July 20, 2026 04:16
9db244d to
be3f0f1
Compare
…and docs - Add EFS4-04 campaign manifest, result loader, evidence DAG, causal diagnosis, architecture dispositions, and Markdown/DOT/Mermaid renderers. - Add scripts/synthesize_efs_campaign.py CLI and pyproject entry point. - Add regression tests for the harness and CLI. - Generate committed design artifacts (json, md, graph) for the synthesis. - Update quality-experiment-matrix.md and research-lineage.md. - Bump harness.experiments and register harness.experiments.efs4_04_causal_synthesis. - Refresh version stamps and clean trailing whitespace in existing iter-* reports. Verification: - pytest: 2912 passed, 7 skipped - scripts.repo_policy: ok - scripts.verify_version_stamps --check: ok - .githooks/check-changed: ok - git diff --check: ok
Tyler-R-Kendrick
force-pushed
the
agent/slm-140
branch
from
July 20, 2026 04:18
be3f0f1 to
6892c63
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (3)
src/slm_training/harnesses/experiments/efs4_04_causal_synthesis.py (1)
65-76: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
EDGE_KINDSduplicates theEvidenceEdge.kindLiteral.The same eight edge-kind strings are maintained in two places. Deriving one from the other (or dropping the unused frozenset if it's not consumed elsewhere) would remove the duplication risk of the two lists silently drifting apart.
Also applies to: 186-198
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/slm_training/harnesses/experiments/efs4_04_causal_synthesis.py` around lines 65 - 76, Remove the duplicated edge-kind declaration by reusing the canonical `EvidenceEdge.kind` Literal for validation or membership checks, or remove `EDGE_KINDS` entirely if it has no consumers. Ensure all supported edge-kind values remain defined in exactly one authoritative location and update references accordingly.scripts/synthesize_efs_campaign.py (2)
161-166: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUnnecessary local re-export of
render_markdown.This wrapper only exists to justify a lazy import "to avoid a second import in
__main__" — but importing it at the top alongside the other harness imports is simpler and avoids a shadowing function with the same name as the thing it wraps.As per coding guidelines, "Do not add abstractions, dependencies, boilerplate, or files unless they are needed; prefer deletion, simple implementations, and the fewest files possible."♻️ Suggested simplification
from slm_training.harnesses.experiments.efs4_04_causal_synthesis import ( EvidenceFirstSemanticSynthesisV1, build_default_campaign_manifest, load_manifest, load_result_manifests, + render_markdown, render_dot, render_mermaid, save_manifest, save_synthesis, synthesize_campaign, validate_synthesis, ) @@ - - -def render_markdown(synthesis: EvidenceFirstSemanticSynthesisV1) -> str: - """Local re-export to avoid a second import in __main__.""" - from slm_training.harnesses.experiments.efs4_04_causal_synthesis import render_markdown as _render - - return _render(synthesis)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/synthesize_efs_campaign.py` around lines 161 - 166, Remove the local render_markdown wrapper in scripts/synthesize_efs_campaign.py and import render_markdown directly alongside the other harness imports. Update any callers to use that imported symbol, preserving the existing rendering behavior without the shadowing re-export.Source: Coding guidelines
111-126: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid importing the private
_matcheshelper across module boundaries.
_matchesis underscore-prefixed inefs4_04_causal_synthesis.py, signaling it's private/internal. Reaching into it from the CLI for--describecouples this script to an unstable internal detail that could be renamed/removed without notice.♻️ Suggested fix: export a small public helper instead
- if args.describe: - from slm_training.harnesses.experiments.efs4_04_causal_synthesis import _matches - - print(f"Campaign: {manifest.campaign_id}") + if args.describe: + print(f"Campaign: {manifest.campaign_id}")Add a public
matches_expected_ref(ref, manifest)wrapper (or rename_matchesto a public name and add it to the module's public exports) inefs4_04_causal_synthesis.py, then import that instead.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/synthesize_efs_campaign.py` around lines 111 - 126, Replace the private _matches import in the --describe flow with a public matches_expected_ref helper from efs4_04_causal_synthesis.py. Add that public wrapper there (or rename and publicly export the existing helper), then use it when checking each hypothesis’s expected_result_refs while preserving the current matching and reporting behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/design/iter-efs4-04-causal-synthesis-20260720.json`:
- Line 5: The committed synthesis artifact uses a pytest temporary directory
instead of the real docs/design inputs. Regenerate the artifact using the
repository’s actual manifest and docs-design paths, then update
generation_command and the corresponding evidence_graph so it contains the real
matched evidence shown in the companion .mmd graph.
- Around line 5-878: Regenerate the EFS4-04 synthesis artifacts using the real
manifest and docs/design directory, not the stale pytest temporary directory:
update docs/design/iter-efs4-04-causal-synthesis-20260720.json lines 5-878 by
running synthesize_efs_campaign with the repository manifest, then regenerate
docs/design/iter-efs4-04-causal-synthesis-20260720.md lines 3-6 and
docs/design/iter-efs4-04-causal-synthesis-graph.mmd lines 1-65 in the same run.
Ensure all three artifacts consistently include the available evidence,
populated hypothesis states and graph edges, honest gate results, and matching
version metadata; the graph site currently appears correct but must remain
synchronized with the regenerated JSON and Markdown.
In `@scripts/synthesize_efs_campaign.py`:
- Around line 145-156: Ensure the output-writing flow around save_synthesis
creates parent directories for out_md and, when args.graph_output is set, for
mmd_path and dot_path before calling write_text. Preserve the existing filenames
and rendering behavior while allowing custom output paths in directories that do
not yet exist.
In `@src/slm_training/harnesses/experiments/efs4_04_causal_synthesis.py`:
- Around line 640-651: Update checkpoint_refs construction and the corresponding
validation in validate_synthesis so path checks operate on the checkpoint value
rather than the key=value wrapper. Ensure local absolute paths and file://
references are both reachable and rejected, removing or revising the enclosing
scheme guard that excludes file:// values. Apply the same behavior to both
row-derived and top-level checkpoint fields.
- Around line 608-669: Update _resolve_hypothesis to select the most recent
matched manifest rather than the alphabetically first one, using source_path
ordering consistent with the iter-*-YYYYMMDD naming convention (or an existing
explicit timestamp/commit field if available). Preserve deterministic selection
and update the nearby comment to describe newest-evidence preference.
In `@src/slm_training/resources/versions.json`:
- Around line 1232-1256: Update the paths array for
harness.experiments.efs4_04_causal_synthesis to include
docs/design/iter-efs4-04-causal-synthesis-graph.mmd and
docs/design/iter-efs4-04-causal-synthesis-graph.dot, preserving all existing
artifact entries.
---
Nitpick comments:
In `@scripts/synthesize_efs_campaign.py`:
- Around line 161-166: Remove the local render_markdown wrapper in
scripts/synthesize_efs_campaign.py and import render_markdown directly alongside
the other harness imports. Update any callers to use that imported symbol,
preserving the existing rendering behavior without the shadowing re-export.
- Around line 111-126: Replace the private _matches import in the --describe
flow with a public matches_expected_ref helper from efs4_04_causal_synthesis.py.
Add that public wrapper there (or rename and publicly export the existing
helper), then use it when checking each hypothesis’s expected_result_refs while
preserving the current matching and reporting behavior.
In `@src/slm_training/harnesses/experiments/efs4_04_causal_synthesis.py`:
- Around line 65-76: Remove the duplicated edge-kind declaration by reusing the
canonical `EvidenceEdge.kind` Literal for validation or membership checks, or
remove `EDGE_KINDS` entirely if it has no consumers. Ensure all supported
edge-kind values remain defined in exactly one authoritative location and update
references accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 53ab612f-89d7-46e8-bc4b-88eb3dd74c05
⛔ Files ignored due to path filters (1)
docs/design/iter-efs4-04-causal-synthesis-graph.dotis excluded by!**/*.dot
📒 Files selected for processing (40)
docs/design/evidence-first-semantic-slm-campaign-v1.jsondocs/design/iter-cap2-03-state-local-action-heads-20260718.mddocs/design/iter-cap2-04-state-ablation-20260718.mddocs/design/iter-cap3-02-calibration-20260718.mddocs/design/iter-cap3-03-ternary-falsification-20260718.mddocs/design/iter-cap3-04-sensitivity-20260718.mddocs/design/iter-cap4-01-residual-quantization-20260718.mddocs/design/iter-cap4-02-adaptive-plane-routing-20260718.mddocs/design/iter-cap4-04-block-sparsity-20260718.mddocs/design/iter-e231-component-inventory-20260716.mddocs/design/iter-e232-role-component-plan-20260716.mddocs/design/iter-efs4-04-causal-synthesis-20260720.jsondocs/design/iter-efs4-04-causal-synthesis-20260720.mddocs/design/iter-efs4-04-causal-synthesis-graph.mmddocs/design/iter-ldi4-02-sae-decision-state-diagnostic-20260720.jsondocs/design/iter-ldi4-03-intervention-unification-20260720.jsondocs/design/iter-pretrained-denoiser-activation-20260719.mddocs/design/iter-scaffold-distillation-activation-20260719.mddocs/design/iter-sde0-02-metric-gaming-20260719.mddocs/design/iter-sde2-02-synthetic-integrity-gates-20260719.mddocs/design/iter-sde2-04-diversity-fingerprints-20260719.mddocs/design/iter-sde4-02-min-controller-capacity-20260720.jsondocs/design/iter-slm135-trailed-assumptions-20260720.jsondocs/design/iter-slm139-stochastic-recursive-width-20260720.jsondocs/design/iter-slm139-stochastic-recursive-width-20260720.mddocs/design/iter-slm144-plan-predictor-20260720.jsondocs/design/iter-slm145-plan-predictor-factors-20260720.jsondocs/design/iter-slm145-plan-predictor-factors-20260720.mddocs/design/iter-slm146-semantic-plan-compiler-20260720.jsondocs/design/iter-slm146-semantic-plan-compiler-20260720.mddocs/design/iter-spv0-03-semantic-regret-20260719.jsondocs/design/quality-experiment-matrix.mddocs/design/research-lineage.mdpyproject.tomlscripts/synthesize_efs_campaign.pysrc/slm_training/harnesses/experiments/__init__.pysrc/slm_training/harnesses/experiments/efs4_04_causal_synthesis.pysrc/slm_training/resources/versions.jsontests/test_harnesses/experiments/test_efs4_04_causal_synthesis.pytests/test_scripts/test_synthesize_efs_campaign.py
| "schema_version": "evidence_first_semantic_synthesis/v1", | ||
| "campaign_id": "evidence-first-semantic-slm-campaign", | ||
| "manifest_hash": "815b9582843ac61a09afb565a621b1037f0344016bac0d463d8a41454fd4a9fd", | ||
| "generation_command": "python -m scripts.synthesize_efs_campaign --manifest /tmp/pytest-of-codex/pytest-320/test_validate_only_with_existi0/manifest.json --docs-design /tmp/pytest-of-codex/pytest-320/test_validate_only_with_existi0", |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy lift
Committed synthesis artifact appears generated from an empty pytest tmp directory, not the real docs/design.
generation_command points at a pytest tmp_path from test_validate_only_with_existing_manifest_returns_zero, and evidence_graph has zero edges/result nodes — but the companion .mmd graph (same date) shows real matched evidence. See consolidated comment for full detail and required fix.
Also applies to: 258-878
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/design/iter-efs4-04-causal-synthesis-20260720.json` at line 5, The
committed synthesis artifact uses a pytest temporary directory instead of the
real docs/design inputs. Regenerate the artifact using the repository’s actual
manifest and docs-design paths, then update generation_command and the
corresponding evidence_graph so it contains the real matched evidence shown in
the companion .mmd graph.
Comment on lines
+5
to
+878
| "generation_command": "python -m scripts.synthesize_efs_campaign --manifest /tmp/pytest-of-codex/pytest-320/test_validate_only_with_existi0/manifest.json --docs-design /tmp/pytest-of-codex/pytest-320/test_validate_only_with_existi0", | ||
| "created_at": "2026-07-20T04:04:04.102502Z", | ||
| "version_stamp": { | ||
| "stamp_schema": "version_stamp/v1", | ||
| "code_commit": "685a506f42f6d246036f1c939c9cb1f0da89a8c4", | ||
| "code_dirty": true, | ||
| "components": { | ||
| "harness.experiments": "v29", | ||
| "harness.experiments.efs4_04_causal_synthesis": "v1" | ||
| }, | ||
| "stamped_at": "2026-07-20T04:04:04.102520+00:00" | ||
| }, | ||
| "hypotheses": [ | ||
| { | ||
| "hypothesis_id": "efs0-01-checkpoint-provenance", | ||
| "linear_issue": "SLM-103", | ||
| "claim": "Frontier checkpoints are durable, hash-verified, and resolvable from a fresh clone.", | ||
| "falsifier": "Checkpoints are missing, hashes do not match, or remote references are unresolvable.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs0-02-decode-invariance", | ||
| "linear_issue": "SLM-104", | ||
| "claim": "A canonical decoder/path exists and produces invariant semantic outcomes across decoder variations.", | ||
| "falsifier": "Decoder path changes still change the primary semantic metric or no canonical path can be selected.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs0-03-meaningful-v2", | ||
| "linear_issue": "SLM-105", | ||
| "claim": "Binding-aware meaningful v2 separates valid-but-empty/useless programs from genuinely useful outputs.", | ||
| "falsifier": "The metric can be gamed by minimal-valid, rare-omission, or inventory-free outputs.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs0-04-judge-independence", | ||
| "linear_issue": "SLM-106", | ||
| "claim": "Judge/human agreement and cross-family scoring show the semantic judge is independent and stable.", | ||
| "falsifier": "Judge conflicts or family-specific inflation exceed a tolerable rate.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs0-05-rejected-lever-readjudication", | ||
| "linear_issue": "SLM-107", | ||
| "claim": "Prior rejected levers remain negative under five seeds, paired tests, corrected decoder, and independent labels.", | ||
| "falsifier": "At least one prior rejection changes classification when confounds are removed.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs1-01-external-ceiling", | ||
| "linear_issue": "SLM-108", | ||
| "claim": "A 1–7B external model constrained by the same compiler achieves a nontrivial semantic ceiling above the tiny SLM.", | ||
| "falsifier": "The external constrained model is near zero, indicating a specification/representation/evaluator limit.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs1-02-exposure-ladder", | ||
| "linear_issue": "SLM-109", | ||
| "claim": "The frozen E228 recipe shows a clear exposure threshold at ≥100× cumulative token exposure.", | ||
| "falsifier": "Semantic metrics remain flat through ≥100× exposure while loss improves or saturates.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs1-03-empty-length-bias", | ||
| "linear_issue": "SLM-110", | ||
| "claim": "Valid-but-empty/minimal-shell outputs win because of length or mask-mass bias, and a principled correction helps.", | ||
| "falsifier": "Empty outputs are preferred by the model score, not the decoder, or corrections do not improve semantics.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs2-01-x22-scaling", | ||
| "linear_issue": "SLM-111", | ||
| "claim": "X22 valid-state tree-edit quality scales predictably with beam width and/or edit depth before saturating.", | ||
| "falsifier": "Quality is flat across beam width and depth while search coverage rises.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs2-02-trigger-telemetry", | ||
| "linear_issue": "SLM-112", | ||
| "claim": "Observe-only trigger predicates fire predictively before recoverable semantic failures under a non-greedy regime.", | ||
| "falsifier": "Triggers never fire or fire without predicting failure/recovery benefit.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs2-03-conflict-slice-repair", | ||
| "linear_issue": "SLM-113", | ||
| "claim": "Conflict-localized remasking repairs more failures with fewer edits than full remask or suffix rollback.", | ||
| "falsifier": "Slices are too imprecise, miss dependencies, or provide no cost/quality benefit at equal budgets.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs2-04-verifier-cascade", | ||
| "linear_issue": "SLM-115", | ||
| "claim": "A cached cheap-to-expensive verifier cascade preserves ≥95% of flat-stack pruning at ≤30% verifier cost.", | ||
| "falsifier": "The cascade loses >5% sound pruning, changes outcomes, or cannot reduce cost below 30–50%.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs3-01-solver-state-supervision", | ||
| "linear_issue": "SLM-118", | ||
| "claim": "A 50/50 gold/on-policy DAgger-style state mixture outperforms pure sources on held-out self-failure recovery.", | ||
| "falsifier": "The mixture provides no recovery gain, or rollout labels are too sparse/noisy to beat pure gold.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs3-02-corruption-curriculum", | ||
| "linear_issue": "SLM-120", | ||
| "claim": "Including 5–15% near-solved semantic corruptions improves local recovery and fixed-point stability without harming generation.", | ||
| "falsifier": "Near-solved mass produces no recovery gain, causes copying, or degrades full-generation quality.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs3-03-b3-capacity-v2", | ||
| "linear_issue": "SLM-124", | ||
| "claim": "The choice representation reaches a preregistered semantic target at smaller capacity/bytes than the surface representation.", | ||
| "falsifier": "After correcting the decoder, surface and choice capacity curves are equivalent or surface is better.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs3-04-candidate-selector", | ||
| "linear_issue": "SLM-127", | ||
| "claim": "A contract-grounded selector with calibrated abstention closes the pass@K-to-selected-pass@K gap.", | ||
| "falsifier": "Candidate pools contain useful programs but no selector beats simple baselines at the target risk.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs3-05-canonical-ast-dedup", | ||
| "linear_issue": "SLM-130", | ||
| "claim": "Canonical AST deduplication increases within-prompt unique hard-valid modes and semantic pass@K at fixed cost.", | ||
| "falsifier": "Deduplication changes only bookkeeping, does not increase unique valid modes, or removes semantically distinct candidates.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs3-06-ast-sketch-retrieval", | ||
| "linear_issue": "SLM-133", | ||
| "claim": "AST-sketch data balancing and/or choice-native retrieval improve semantic quality at fixed budget without leakage.", | ||
| "falsifier": "Sketch balancing or choice retrieval is equivalent to controls or harms OOD semantics/copying.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs4-01-trailed-assumptions", | ||
| "linear_issue": "SLM-135", | ||
| "claim": "A trailed, dependency-aware solver avoids false prunes that a monotone proposal-contingent state cannot recover from.", | ||
| "falsifier": "Production architecture never places proposal facts in irreversible state, or both policies recover identically.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs4-02-shared-recursive-denoiser", | ||
| "linear_issue": "SLM-138", | ||
| "claim": "A weight-shared recursive denoiser with deep supervision improves semantic/recovery metrics at matched block evaluations.", | ||
| "falsifier": "Recursive arms are equivalent or worse under matched cost and exposure.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| }, | ||
| { | ||
| "hypothesis_id": "efs4-03-stochastic-recursive-state", | ||
| "linear_issue": "SLM-139", | ||
| "claim": "A learned high-level stochastic recursive state increases valid semantic mode coverage and selected quality at matched cost.", | ||
| "falsifier": "Samples collapse to one mode, low-level noise matches/exceeds high-level width, or selector cannot convert width safely.", | ||
| "state": "MISSING", | ||
| "state_reason": "No committed result manifest matched the expected refs.", | ||
| "result_refs": [], | ||
| "checkpoint_refs": [], | ||
| "notes": [] | ||
| } | ||
| ], | ||
| "lineage_summary": { | ||
| "manifest_id": "evidence-first-semantic-slm-campaign", | ||
| "hypothesis_count": 21, | ||
| "terminal_state_counts": { | ||
| "MISSING": 21 | ||
| }, | ||
| "checkpoint_reference_count": 0 | ||
| }, | ||
| "evidence_graph": { | ||
| "nodes": [ | ||
| { | ||
| "node_id": "campaign:efs", | ||
| "kind": "causal_conclusion", | ||
| "label": "Evidence-First Semantic SLM Campaign", | ||
| "payload": { | ||
| "manifest_hash": "815b9582843ac61a09afb565a621b1037f0344016bac0d463d8a41454fd4a9fd", | ||
| "hypothesis_count": 21 | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs0-01-checkpoint-provenance", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-103 efs0-01-checkpoint-provenance", | ||
| "payload": { | ||
| "claim": "Frontier checkpoints are durable, hash-verified, and resolvable from a fresh clone.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs0-02-decode-invariance", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-104 efs0-02-decode-invariance", | ||
| "payload": { | ||
| "claim": "A canonical decoder/path exists and produces invariant semantic outcomes across decoder variations.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs0-03-meaningful-v2", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-105 efs0-03-meaningful-v2", | ||
| "payload": { | ||
| "claim": "Binding-aware meaningful v2 separates valid-but-empty/useless programs from genuinely useful outputs.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs0-04-judge-independence", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-106 efs0-04-judge-independence", | ||
| "payload": { | ||
| "claim": "Judge/human agreement and cross-family scoring show the semantic judge is independent and stable.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs0-05-rejected-lever-readjudication", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-107 efs0-05-rejected-lever-readjudication", | ||
| "payload": { | ||
| "claim": "Prior rejected levers remain negative under five seeds, paired tests, corrected decoder, and independent labels.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs1-01-external-ceiling", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-108 efs1-01-external-ceiling", | ||
| "payload": { | ||
| "claim": "A 1–7B external model constrained by the same compiler achieves a nontrivial semantic ceiling above the tiny SLM.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs1-02-exposure-ladder", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-109 efs1-02-exposure-ladder", | ||
| "payload": { | ||
| "claim": "The frozen E228 recipe shows a clear exposure threshold at ≥100× cumulative token exposure.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs1-03-empty-length-bias", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-110 efs1-03-empty-length-bias", | ||
| "payload": { | ||
| "claim": "Valid-but-empty/minimal-shell outputs win because of length or mask-mass bias, and a principled correction helps.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs2-01-x22-scaling", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-111 efs2-01-x22-scaling", | ||
| "payload": { | ||
| "claim": "X22 valid-state tree-edit quality scales predictably with beam width and/or edit depth before saturating.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs2-02-trigger-telemetry", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-112 efs2-02-trigger-telemetry", | ||
| "payload": { | ||
| "claim": "Observe-only trigger predicates fire predictively before recoverable semantic failures under a non-greedy regime.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs2-03-conflict-slice-repair", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-113 efs2-03-conflict-slice-repair", | ||
| "payload": { | ||
| "claim": "Conflict-localized remasking repairs more failures with fewer edits than full remask or suffix rollback.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs2-04-verifier-cascade", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-115 efs2-04-verifier-cascade", | ||
| "payload": { | ||
| "claim": "A cached cheap-to-expensive verifier cascade preserves ≥95% of flat-stack pruning at ≤30% verifier cost.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs3-01-solver-state-supervision", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-118 efs3-01-solver-state-supervision", | ||
| "payload": { | ||
| "claim": "A 50/50 gold/on-policy DAgger-style state mixture outperforms pure sources on held-out self-failure recovery.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs3-02-corruption-curriculum", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-120 efs3-02-corruption-curriculum", | ||
| "payload": { | ||
| "claim": "Including 5–15% near-solved semantic corruptions improves local recovery and fixed-point stability without harming generation.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs3-03-b3-capacity-v2", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-124 efs3-03-b3-capacity-v2", | ||
| "payload": { | ||
| "claim": "The choice representation reaches a preregistered semantic target at smaller capacity/bytes than the surface representation.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs3-04-candidate-selector", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-127 efs3-04-candidate-selector", | ||
| "payload": { | ||
| "claim": "A contract-grounded selector with calibrated abstention closes the pass@K-to-selected-pass@K gap.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs3-05-canonical-ast-dedup", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-130 efs3-05-canonical-ast-dedup", | ||
| "payload": { | ||
| "claim": "Canonical AST deduplication increases within-prompt unique hard-valid modes and semantic pass@K at fixed cost.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs3-06-ast-sketch-retrieval", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-133 efs3-06-ast-sketch-retrieval", | ||
| "payload": { | ||
| "claim": "AST-sketch data balancing and/or choice-native retrieval improve semantic quality at fixed budget without leakage.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs4-01-trailed-assumptions", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-135 efs4-01-trailed-assumptions", | ||
| "payload": { | ||
| "claim": "A trailed, dependency-aware solver avoids false prunes that a monotone proposal-contingent state cannot recover from.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs4-02-shared-recursive-denoiser", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-138 efs4-02-shared-recursive-denoiser", | ||
| "payload": { | ||
| "claim": "A weight-shared recursive denoiser with deep supervision improves semantic/recovery metrics at matched block evaluations.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "hyp:efs4-03-stochastic-recursive-state", | ||
| "kind": "hypothesis", | ||
| "label": "SLM-139 efs4-03-stochastic-recursive-state", | ||
| "payload": { | ||
| "claim": "A learned high-level stochastic recursive state increases valid semantic mode coverage and selected quality at matched cost.", | ||
| "state": "MISSING" | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "diagnosis:primary", | ||
| "kind": "causal_conclusion", | ||
| "label": "Primary diagnosis: insufficient_valid_evidence", | ||
| "payload": { | ||
| "primary": "insufficient_valid_evidence", | ||
| "primary_reason": "Core measurement issues (checkpoint provenance, decoder invariance, semantic metric, judge independence, re-adjudication) are not all POSITIVE. Without durable, invariant, independently measured evidence, no causal diagnosis of training, data, search, or architecture can be asserted.", | ||
| "counterfactual_evidence": "Counterfactual: if SLM-103/104/105/106/107 were all POSITIVE, the remaining NOT_RUN_BY_GATE states could be attributed to training-exposure or architecture limits rather than to measurement uncertainty.", | ||
| "secondary": [ | ||
| "measurement_limited" | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:compiler-owned-grammar-schema-binding-lattice-and-exact-closure", | ||
| "kind": "architecture_decision", | ||
| "label": "compiler-owned grammar/schema/binding lattice and exact closure -> ADOPT_AS_SAFETY_ONLY", | ||
| "payload": { | ||
| "item": "compiler-owned grammar/schema/binding lattice and exact closure", | ||
| "disposition": "ADOPT_AS_SAFETY_ONLY", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "cost", | ||
| "nogood", | ||
| "solver", | ||
| "trail", | ||
| "verifier" | ||
| ], | ||
| "next_action": "Keep as correctness infrastructure; do not claim quality improvement." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:reduced-product-cross-domain-propagation-claims", | ||
| "kind": "architecture_decision", | ||
| "label": "reduced-product/cross-domain propagation claims -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "reduced-product/cross-domain propagation claims", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "nogood", | ||
| "solver", | ||
| "trail" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:reversible-decisions-local-nogoods-and-certificate-backed-traili", | ||
| "kind": "architecture_decision", | ||
| "label": "reversible decisions, local nogoods, and certificate-backed trailing -> ADOPT_AS_SAFETY_ONLY", | ||
| "payload": { | ||
| "item": "reversible decisions, local nogoods, and certificate-backed trailing", | ||
| "disposition": "ADOPT_AS_SAFETY_ONLY", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "nogood", | ||
| "solver", | ||
| "trail" | ||
| ], | ||
| "next_action": "Keep as correctness infrastructure; do not claim quality improvement." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:free-form-typed-ast-topology-diffusion", | ||
| "kind": "architecture_decision", | ||
| "label": "free-form typed-AST/topology diffusion -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "free-form typed-AST/topology diffusion", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "capacity", | ||
| "choice_codec", | ||
| "representation" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:x22-all-valid-tree-edit-diffusion", | ||
| "kind": "architecture_decision", | ||
| "label": "X22/all-valid tree-edit diffusion -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "X22/all-valid tree-edit diffusion", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "search", | ||
| "x22" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:deterministic-shared-recursive-denoiser-deep-supervision", | ||
| "kind": "architecture_decision", | ||
| "label": "deterministic shared recursive denoiser/deep supervision -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "deterministic shared recursive denoiser/deep supervision", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "architecture", | ||
| "deep_supervision", | ||
| "recursive" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:request-local-recurrent-latent-persistence", | ||
| "kind": "architecture_decision", | ||
| "label": "request-local recurrent latent persistence -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "request-local recurrent latent persistence", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "architecture", | ||
| "deep_supervision", | ||
| "recursive" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:triggered-ptrm-inference-only-low-level-noise", | ||
| "kind": "architecture_decision", | ||
| "label": "triggered PTRM/inference-only low-level noise -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "triggered PTRM/inference-only low-level noise", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "ptrm", | ||
| "trigger" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:learned-gram-style-high-level-stochastic-state", | ||
| "kind": "architecture_decision", | ||
| "label": "learned GRAM-style high-level stochastic state -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "learned GRAM-style high-level stochastic state", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "architecture", | ||
| "gram", | ||
| "stochastic" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:candidate-ast-dedup-semantic-mode-tracking", | ||
| "kind": "architecture_decision", | ||
| "label": "candidate AST dedup/semantic mode tracking -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "candidate AST dedup/semantic mode tracking", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "dedup", | ||
| "diversity" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:contract-grounded-selector-and-abstention", | ||
| "kind": "architecture_decision", | ||
| "label": "contract-grounded selector and abstention -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "contract-grounded selector and abstention", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "abstention", | ||
| "selection" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:conflict-slice-remasking", | ||
| "kind": "architecture_decision", | ||
| "label": "conflict-slice remasking -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "conflict-slice remasking", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "remask", | ||
| "repair" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:cheap-to-expensive-verifier-cascade-cache", | ||
| "kind": "architecture_decision", | ||
| "label": "cheap-to-expensive verifier cascade/cache -> ADOPT_AS_SAFETY_ONLY", | ||
| "payload": { | ||
| "item": "cheap-to-expensive verifier cascade/cache", | ||
| "disposition": "ADOPT_AS_SAFETY_ONLY", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "cost", | ||
| "verifier" | ||
| ], | ||
| "next_action": "Keep as correctness infrastructure; do not claim quality improvement." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:choice-representation-capacity-conclusion", | ||
| "kind": "architecture_decision", | ||
| "label": "choice representation/capacity conclusion -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "choice representation/capacity conclusion", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "capacity", | ||
| "choice_codec", | ||
| "exposure", | ||
| "representation", | ||
| "training" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:gold-on-policy-mixed-supervision-and-nearly-solved-curriculum", | ||
| "kind": "architecture_decision", | ||
| "label": "gold/on-policy mixed supervision and nearly-solved curriculum -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "gold/on-policy mixed supervision and nearly-solved curriculum", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "corruption", | ||
| "curriculum", | ||
| "on_policy", | ||
| "supervision" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:ast-sketch-data-balancing-and-choice-native-retrieval", | ||
| "kind": "architecture_decision", | ||
| "label": "AST-sketch data balancing and choice-native retrieval -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "AST-sketch data balancing and choice-native retrieval", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "data", | ||
| "retrieval" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| }, | ||
| { | ||
| "node_id": "decision:content-floor-length-mask-mass-corrections", | ||
| "kind": "architecture_decision", | ||
| "label": "content-floor/length/mask-mass corrections -> NOT_RUN_BY_GATE", | ||
| "payload": { | ||
| "item": "content-floor/length/mask-mass corrections", | ||
| "disposition": "NOT_RUN_BY_GATE", | ||
| "supporting_experiments": [], | ||
| "falsifying_experiments": [], | ||
| "effect": "No proven semantic-quality effect under current evidence.", | ||
| "uncertainty": "Evidence is plan/fixture-grade; frontier runs are pending.", | ||
| "semantic_metric": "binding_aware_meaningful_v2", | ||
| "cost_metric": "wall_seconds / verifier_calls", | ||
| "safety_constraints": [ | ||
| "Must not weaken compiler-owned legality." | ||
| ], | ||
| "activation_conditions": [ | ||
| "content_floor", | ||
| "decode" | ||
| ], | ||
| "next_action": "Run the related EFS branch to a terminal state before disposition." | ||
| } | ||
| } | ||
| ], | ||
| "edges": [] | ||
| }, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy lift
Committed EFS4-04 synthesis artifacts were generated from a stale/empty test directory, not the real docs/design/.
The JSON's generation_command (--manifest .../test_validate_only_with_existi0/manifest.json --docs-design .../test_validate_only_with_existi0) matches the truncated pytest tmp_path for test_validate_only_with_existing_manifest_returns_zero in tests/test_scripts/test_synthesize_efs_campaign.py. That tmp directory contains no iter-*.json result fixtures, which is why every one of the 21 hypotheses resolves to MISSING and evidence_graph.edges is empty. The Markdown faithfully re-renders this same wrong state. However, the companion .mmd evidence graph for the identical date-stamp shows real result_* nodes (e.g. iter-efs-decode-invariance-20260718.json, iter-efs0-03-meaningful-v2-frontier-audit-20260717.json) with populated --supports--> edges — proof that real evidence exists in the actual docs/design/ and would change the causal diagnosis, architecture dispositions, and champion decision if the script were run correctly. As committed, the "official" EFS4-04 report is not an honest representation of the evidence actually available in this repository, which directly conflicts with the guideline that experiments "must leave durable evidence... matching Markdown measured results" and "honest gate results."
docs/design/iter-efs4-04-causal-synthesis-20260720.json#L5-L878: regenerate by runningpython -m scripts.synthesize_efs_campaign --manifest docs/design/evidence-first-semantic-slm-campaign-v1.json --docs-design docs/design ...(the real repo path, not a tmp fixture dir) and commit the corrected output.docs/design/iter-efs4-04-causal-synthesis-20260720.md#L3-L6: regenerate from the corrected JSON in the same run.docs/design/iter-efs4-04-causal-synthesis-graph.mmd#L1-L65: regenerate alongside the corrected JSON/MD so all three artifacts reflect one consistent run (this file already appears correct and can serve as the reference for what the corrected JSON/MD should show).
As per coding guidelines, "All experiments must leave durable evidence under docs/design/, including JSON, matching Markdown measured results or notes, recipe metadata, honest gate results, and a version_stamp."
📍 Affects 3 files
docs/design/iter-efs4-04-causal-synthesis-20260720.json#L5-L878(this comment)docs/design/iter-efs4-04-causal-synthesis-20260720.md#L3-L6docs/design/iter-efs4-04-causal-synthesis-graph.mmd#L1-L65
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/design/iter-efs4-04-causal-synthesis-20260720.json` around lines 5 -
878, Regenerate the EFS4-04 synthesis artifacts using the real manifest and
docs/design directory, not the stale pytest temporary directory: update
docs/design/iter-efs4-04-causal-synthesis-20260720.json lines 5-878 by running
synthesize_efs_campaign with the repository manifest, then regenerate
docs/design/iter-efs4-04-causal-synthesis-20260720.md lines 3-6 and
docs/design/iter-efs4-04-causal-synthesis-graph.mmd lines 1-65 in the same run.
Ensure all three artifacts consistently include the available evidence,
populated hypothesis states and graph edges, honest gate results, and matching
version metadata; the graph site currently appears correct but must remain
synchronized with the regenerated JSON and Markdown.
Source: Coding guidelines
Comment on lines
+608
to
+669
| def _resolve_hypothesis( | ||
| hypothesis: CampaignHypothesisSpec, | ||
| manifests: tuple[ResultManifest, ...], | ||
| ) -> HypothesisSynthesis: | ||
| """Resolve one hypothesis against the loaded result manifests.""" | ||
| matched = tuple(m for m in manifests if any(_matches(ref, m) for ref in hypothesis.expected_result_refs)) | ||
| if not matched: | ||
| return HypothesisSynthesis( | ||
| hypothesis_id=hypothesis.hypothesis_id, | ||
| linear_issue=hypothesis.linear_issue, | ||
| claim=hypothesis.claim, | ||
| falsifier=hypothesis.falsifier, | ||
| state="MISSING", | ||
| state_reason="No committed result manifest matched the expected refs.", | ||
| ) | ||
|
|
||
| # Prefer the strongest resolved manifest deterministically (alphabetical by path). | ||
| chosen = sorted(matched, key=lambda m: m.source_path)[0] | ||
| inferred = _manifest_state(chosen) | ||
| allowed = set(hypothesis.allowed_decisions) or set( | ||
| ("INVALIDATED", "INCONCLUSIVE", "EQUIVALENT", "NEGATIVE", "POSITIVE", "NOT_RUN_BY_GATE") | ||
| ) | ||
| if inferred not in allowed: | ||
| state: DecisionState = "CONTRADICTORY" | ||
| reason = ( | ||
| f"Inferred state {inferred!r} from {chosen.source_path} is not in " | ||
| f"preregistered allowed decisions {sorted(allowed)}." | ||
| ) | ||
| else: | ||
| state = inferred | ||
| reason = f"Resolved from {chosen.source_path} (schema={chosen.schema_version}, status={chosen.status})." | ||
|
|
||
| checkpoint_refs: list[str] = [] | ||
| for row in chosen.rows: | ||
| if isinstance(row, dict): | ||
| for key in ("checkpoint_sha256", "checkpoint_path", "checkpoint_remote_uri"): | ||
| val = row.get(key) | ||
| if val: | ||
| checkpoint_refs.append(f"{key}={val}") | ||
| # Also look for top-level checkpoint fields. | ||
| for key in ("checkpoint_sha256", "checkpoint_path", "checkpoint_remote_uri"): | ||
| val = chosen.raw.get(key) | ||
| if val: | ||
| checkpoint_refs.append(f"{key}={val}") | ||
|
|
||
| notes: list[str] = [] | ||
| if hypothesis.activation_gate and state == "NOT_RUN_BY_GATE": | ||
| notes.append(f"Activation gate not cleared: {hypothesis.activation_gate}") | ||
| if len(matched) > 1: | ||
| notes.append(f"Multiple result manifests matched: {[m.source_path for m in matched]}") | ||
|
|
||
| return HypothesisSynthesis( | ||
| hypothesis_id=hypothesis.hypothesis_id, | ||
| linear_issue=hypothesis.linear_issue, | ||
| claim=hypothesis.claim, | ||
| falsifier=hypothesis.falsifier, | ||
| state=state, | ||
| state_reason=reason, | ||
| result_refs=tuple(m.source_path for m in matched), | ||
| checkpoint_refs=tuple(sorted(set(checkpoint_refs))), | ||
| notes=tuple(notes), | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Alphabetical "chosen" manifest pick may select the stalest, not strongest, evidence.
sorted(matched, key=lambda m: m.source_path)[0] picks the alphabetically-first match when a hypothesis's expected_result_refs glob matches multiple committed result files. Given the repo's iter-*-YYYYMMDD*.json naming convention, sorting alphabetically by path is equivalent to sorting by date ascending, so a newer rerun of the same experiment will never override an earlier one — the comment "Prefer the strongest resolved manifest deterministically" doesn't reflect this. For a campaign whose entire premise is evidence-first causal diagnosis, silently preferring stale results over more recent reruns is a real correctness risk.
🐛 Prefer the most recent match by source_path (or an explicit source_commit/created_at field) instead of the alphabetically-first one
- # Prefer the strongest resolved manifest deterministically (alphabetical by path).
- chosen = sorted(matched, key=lambda m: m.source_path)[0]
+ # Prefer the most recently generated manifest deterministically; ties break by path.
+ chosen = sorted(matched, key=lambda m: m.source_path, reverse=True)[0]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _resolve_hypothesis( | |
| hypothesis: CampaignHypothesisSpec, | |
| manifests: tuple[ResultManifest, ...], | |
| ) -> HypothesisSynthesis: | |
| """Resolve one hypothesis against the loaded result manifests.""" | |
| matched = tuple(m for m in manifests if any(_matches(ref, m) for ref in hypothesis.expected_result_refs)) | |
| if not matched: | |
| return HypothesisSynthesis( | |
| hypothesis_id=hypothesis.hypothesis_id, | |
| linear_issue=hypothesis.linear_issue, | |
| claim=hypothesis.claim, | |
| falsifier=hypothesis.falsifier, | |
| state="MISSING", | |
| state_reason="No committed result manifest matched the expected refs.", | |
| ) | |
| # Prefer the strongest resolved manifest deterministically (alphabetical by path). | |
| chosen = sorted(matched, key=lambda m: m.source_path)[0] | |
| inferred = _manifest_state(chosen) | |
| allowed = set(hypothesis.allowed_decisions) or set( | |
| ("INVALIDATED", "INCONCLUSIVE", "EQUIVALENT", "NEGATIVE", "POSITIVE", "NOT_RUN_BY_GATE") | |
| ) | |
| if inferred not in allowed: | |
| state: DecisionState = "CONTRADICTORY" | |
| reason = ( | |
| f"Inferred state {inferred!r} from {chosen.source_path} is not in " | |
| f"preregistered allowed decisions {sorted(allowed)}." | |
| ) | |
| else: | |
| state = inferred | |
| reason = f"Resolved from {chosen.source_path} (schema={chosen.schema_version}, status={chosen.status})." | |
| checkpoint_refs: list[str] = [] | |
| for row in chosen.rows: | |
| if isinstance(row, dict): | |
| for key in ("checkpoint_sha256", "checkpoint_path", "checkpoint_remote_uri"): | |
| val = row.get(key) | |
| if val: | |
| checkpoint_refs.append(f"{key}={val}") | |
| # Also look for top-level checkpoint fields. | |
| for key in ("checkpoint_sha256", "checkpoint_path", "checkpoint_remote_uri"): | |
| val = chosen.raw.get(key) | |
| if val: | |
| checkpoint_refs.append(f"{key}={val}") | |
| notes: list[str] = [] | |
| if hypothesis.activation_gate and state == "NOT_RUN_BY_GATE": | |
| notes.append(f"Activation gate not cleared: {hypothesis.activation_gate}") | |
| if len(matched) > 1: | |
| notes.append(f"Multiple result manifests matched: {[m.source_path for m in matched]}") | |
| return HypothesisSynthesis( | |
| hypothesis_id=hypothesis.hypothesis_id, | |
| linear_issue=hypothesis.linear_issue, | |
| claim=hypothesis.claim, | |
| falsifier=hypothesis.falsifier, | |
| state=state, | |
| state_reason=reason, | |
| result_refs=tuple(m.source_path for m in matched), | |
| checkpoint_refs=tuple(sorted(set(checkpoint_refs))), | |
| notes=tuple(notes), | |
| ) | |
| def _resolve_hypothesis( | |
| hypothesis: CampaignHypothesisSpec, | |
| manifests: tuple[ResultManifest, ...], | |
| ) -> HypothesisSynthesis: | |
| """Resolve one hypothesis against the loaded result manifests.""" | |
| matched = tuple(m for m in manifests if any(_matches(ref, m) for ref in hypothesis.expected_result_refs)) | |
| if not matched: | |
| return HypothesisSynthesis( | |
| hypothesis_id=hypothesis.hypothesis_id, | |
| linear_issue=hypothesis.linear_issue, | |
| claim=hypothesis.claim, | |
| falsifier=hypothesis.falsifier, | |
| state="MISSING", | |
| state_reason="No committed result manifest matched the expected refs.", | |
| ) | |
| # Prefer the most recently generated manifest deterministically; ties break by path. | |
| chosen = sorted(matched, key=lambda m: m.source_path, reverse=True)[0] | |
| inferred = _manifest_state(chosen) | |
| allowed = set(hypothesis.allowed_decisions) or set( | |
| ("INVALIDATED", "INCONCLUSIVE", "EQUIVALENT", "NEGATIVE", "POSITIVE", "NOT_RUN_BY_GATE") | |
| ) | |
| if inferred not in allowed: | |
| state: DecisionState = "CONTRADICTORY" | |
| reason = ( | |
| f"Inferred state {inferred!r} from {chosen.source_path} is not in " | |
| f"preregistered allowed decisions {sorted(allowed)}." | |
| ) | |
| else: | |
| state = inferred | |
| reason = f"Resolved from {chosen.source_path} (schema={chosen.schema_version}, status={chosen.status})." | |
| checkpoint_refs: list[str] = [] | |
| for row in chosen.rows: | |
| if isinstance(row, dict): | |
| for key in ("checkpoint_sha256", "checkpoint_path", "checkpoint_remote_uri"): | |
| val = row.get(key) | |
| if val: | |
| checkpoint_refs.append(f"{key}={val}") | |
| # Also look for top-level checkpoint fields. | |
| for key in ("checkpoint_sha256", "checkpoint_path", "checkpoint_remote_uri"): | |
| val = chosen.raw.get(key) | |
| if val: | |
| checkpoint_refs.append(f"{key}={val}") | |
| notes: list[str] = [] | |
| if hypothesis.activation_gate and state == "NOT_RUN_BY_GATE": | |
| notes.append(f"Activation gate not cleared: {hypothesis.activation_gate}") | |
| if len(matched) > 1: | |
| notes.append(f"Multiple result manifests matched: {[m.source_path for m in matched]}") | |
| return HypothesisSynthesis( | |
| hypothesis_id=hypothesis.hypothesis_id, | |
| linear_issue=hypothesis.linear_issue, | |
| claim=hypothesis.claim, | |
| falsifier=hypothesis.falsifier, | |
| state=state, | |
| state_reason=reason, | |
| result_refs=tuple(m.source_path for m in matched), | |
| checkpoint_refs=tuple(sorted(set(checkpoint_refs))), | |
| notes=tuple(notes), | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/slm_training/harnesses/experiments/efs4_04_causal_synthesis.py` around
lines 608 - 669, Update _resolve_hypothesis to select the most recent matched
manifest rather than the alphabetically first one, using source_path ordering
consistent with the iter-*-YYYYMMDD naming convention (or an existing explicit
timestamp/commit field if available). Preserve deterministic selection and
update the nearby comment to describe newest-evidence preference.
Comment on lines
+640
to
+651
| checkpoint_refs: list[str] = [] | ||
| for row in chosen.rows: | ||
| if isinstance(row, dict): | ||
| for key in ("checkpoint_sha256", "checkpoint_path", "checkpoint_remote_uri"): | ||
| val = row.get(key) | ||
| if val: | ||
| checkpoint_refs.append(f"{key}={val}") | ||
| # Also look for top-level checkpoint fields. | ||
| for key in ("checkpoint_sha256", "checkpoint_path", "checkpoint_remote_uri"): | ||
| val = chosen.raw.get(key) | ||
| if val: | ||
| checkpoint_refs.append(f"{key}={val}") |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Local-absolute-path checkpoint check never fires due to the key=value ref format.
checkpoint_refs entries are built as f"{key}={val}" (e.g. "checkpoint_path=/abs/local/path"), but validate_synthesis checks ref.startswith(("file://", "/")) against that same string. Since the string always starts with the field name ("checkpoint_path=", etc.), the local-absolute-path prohibition can never trigger — this validation gate is effectively dead. Separately, the "file://" branch of that same check is unreachable regardless, since the enclosing "://" not in ref guard already excludes any ref containing "file://".
This defeats the intended safety check that prevents fixture/local checkpoint refs from being published as durable evidence, which matters given the repo policy that full HF-context checkpoints must sync to hf://buckets/....
🛡️ Strip the key prefix before checking the path shape
for syn in synthesis.hypotheses:
if syn.state == "CONTRADICTORY":
errors.append(f"{syn.hypothesis_id}: contradictory state vs preregistered decision contract")
for ref in syn.checkpoint_refs:
- if "://" not in ref and "/" in ref and not ref.startswith("hf://"):
- # local absolute paths are prohibited
- if ref.startswith(("file://", "/")):
- errors.append(f"{syn.hypothesis_id}: local absolute checkpoint ref {ref}")
+ _, _, value = ref.partition("=")
+ if value and not value.startswith("hf://") and (value.startswith("/") or value.startswith("file://")):
+ errors.append(f"{syn.hypothesis_id}: local absolute checkpoint ref {ref}")Also applies to: 1261-1268
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/slm_training/harnesses/experiments/efs4_04_causal_synthesis.py` around
lines 640 - 651, Update checkpoint_refs construction and the corresponding
validation in validate_synthesis so path checks operate on the checkpoint value
rather than the key=value wrapper. Ensure local absolute paths and file://
references are both reachable and rejected, removing or revising the enclosing
scheme guard that excludes file:// values. Apply the same behavior to both
row-derived and top-level checkpoint fields.
Comment on lines
+1232
to
1256
| "harness.experiments.efs4_04_causal_synthesis": { | ||
| "version": "v1", | ||
| "kind": "harness", | ||
| "paths": [ | ||
| "src/slm_training/harnesses/experiments/efs4_04_causal_synthesis.py", | ||
| "scripts/synthesize_efs_campaign.py", | ||
| "tests/test_harnesses/experiments/test_efs4_04_causal_synthesis.py", | ||
| "tests/test_scripts/test_synthesize_efs_campaign.py", | ||
| "docs/design/evidence-first-semantic-slm-campaign-v1.json", | ||
| "docs/design/iter-efs4-04-causal-synthesis-20260720.json", | ||
| "docs/design/iter-efs4-04-causal-synthesis-20260720.md" | ||
| ], | ||
| "history": [ | ||
| { | ||
| "version": "v1", | ||
| "date": "2026-07-20", | ||
| "note": "no-bump: add regression tests for the campaign manifest, synthesis loader, and CLI; harness behavior unchanged" | ||
| }, | ||
| { | ||
| "version": "v1", | ||
| "date": "2026-07-20", | ||
| "note": "initial registration of SLM-140 EFS4-04 Evidence-First Semantic SLM causal synthesis manifest and loader" | ||
| } | ||
| ] | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: confirm which EFS4-04 graph artifacts were committed and whether they're covered by any versions.json path prefix.
fd -g 'iter-efs4-04-causal-synthesis-graph*' docs/design
rg -n '"docs/design/' src/slm_training/resources/versions.json | rg -i 'efs4|graph'Repository: Tyler-R-Kendrick/slm-training
Length of output: 412
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the relevant versions.json area and search for any component that already claims the graph artifacts.
python3 - <<'PY'
from pathlib import Path
p = Path('src/slm_training/resources/versions.json')
lines = p.read_text().splitlines()
for start in (1180, 1230, 1260):
end = min(len(lines), start + 80)
print(f"\n--- versions.json lines {start+1}-{end} ---")
for i in range(start, end):
print(f"{i+1}: {lines[i]}")
PY
echo
echo "--- direct graph path search ---"
rg -n '"docs/design/iter-efs4-04-causal-synthesis-graph\.(mmd|dot)"' src/slm_training/resources/versions.json || true
echo
echo "--- any efs4-04 component paths mentioning graph ---"
rg -n '"harness\.experiments\.efs4_04_causal_synthesis"|docs/design/iter-efs4-04-causal-synthesis-graph' src/slm_training/resources/versions.jsonRepository: Tyler-R-Kendrick/slm-training
Length of output: 2425
Add the graph artifacts to harness.experiments.efs4_04_causal_synthesis.paths
docs/design/iter-efs4-04-causal-synthesis-graph.mmd and docs/design/iter-efs4-04-causal-synthesis-graph.dot are committed outputs of this harness/CLI but aren’t listed here, so they won’t be covered by the version-stamp staleness check.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/slm_training/resources/versions.json` around lines 1232 - 1256, Update
the paths array for harness.experiments.efs4_04_causal_synthesis to include
docs/design/iter-efs4-04-causal-synthesis-graph.mmd and
docs/design/iter-efs4-04-causal-synthesis-graph.dot, preserving all existing
artifact entries.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/synthesize_efs_campaign.pyCLI and pyproject entry point.quality-experiment-matrix.mdandresearch-lineage.md.harness.experimentsand registerharness.experiments.efs4_04_causal_synthesis.Verification
scripts.repo_policy: okscripts.verify_version_stamps --check: ok.githooks/check-changed: okgit diff --check: okSummary by CodeRabbit
New Features
Documentation
Tests