From 4007da49adb37dee8a7886dcc330ad951b4f7a1f Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 10 Jul 2026 19:29:20 -0400 Subject: [PATCH 1/5] Add conformance framework design plan (F0-F5) Design for turning the 38 ad hoc rust/conformance/ fixtures into a general-purpose, engine-agnostic HermitCrab conformance suite that lives in Machine (not the Rust port, which may move to its own repo): a new top-level conformance/ directory, an adapter CLI protocol any engine can implement, a C# harness with self-check + coverage-report modes, and four fixture categories (single-feature, negative, cross-cutting, pathological) instead of today's single-feature-only pattern. Per John's mandates: every word and parse carries a human-readable description of what it exercises, the coverage report is meant to actively drive which fixtures get added next, and pathological (complexity-stress, not feature-correctness) fixtures are tagged and excluded from default runs. First implementation pass scoped to F0-F3 (canonicalize the oracle CLI, migrate + standardize existing fixtures, build the harness, one worked example of each new category) -- F4 (coverage-driven backfill) and F5 (CI) are explicit follow-ups, not part of this PR. Co-Authored-By: Claude Sonnet 5 --- docs/conformance-framework-plan.md | 276 +++++++++++++++++++++++++++++ 1 file changed, 276 insertions(+) create mode 100644 docs/conformance-framework-plan.md diff --git a/docs/conformance-framework-plan.md b/docs/conformance-framework-plan.md new file mode 100644 index 00000000..f25cfebd --- /dev/null +++ b/docs/conformance-framework-plan.md @@ -0,0 +1,276 @@ +# HermitCrab Conformance Framework — design & implementation plan + +**Status:** approved by John 2026-07-10, implementation starting on branch `conformance-framework`. + +## 1. Motivation + +The Rust port of `SIL.Machine.Morphology.HermitCrab` built up 38 oracle-generated conformance +fixtures during Phase 2 (`rust/conformance/`), each a hand-minimized grammar + word list + a +signature-format `expected.tsv` generated by running the *live C# engine* as ground truth. This +worked well, but: + +- It lives under `rust/`, which is likely to be extracted to its own repo. The fixtures — and the + oracle that generates them — are not Rust-specific; they should outlive that split. +- Coverage was built two ways: mining C# commit history (`rust/conformance/HISTORY-MATRIX.md`) and + fixtures born from whichever bug someone happened to hit while porting a feature. There is no + systematic walk of HermitCrab's actual grammar-model surface, and no mechanism that tells you + *what's still missing* — an inventory pass for this plan found reduplication has zero fixtures, + affix-template slot semantics are untested in isolation, and several `MorphologicalOutputAction` + variants are only incidentally exercised. +- The oracle-generating tool itself (`BatchCommand`, in `SIL.Machine.Morphology.HermitCrab.Tool`) + doesn't exist on `master` — only in a worktree tied to an unmerged perf-optimization PR. +- Every fixture today is a single-feature positive case. There are no deliberate negative fixtures + (words that must NOT parse), no fixtures designed to exercise more than one mechanism at once + (cross-cutting), and no fixtures whose purpose is to stress the engine's complexity/performance + characteristics rather than pin one rule's correctness. + +## 2. Goals + +1. A **general-purpose, engine-agnostic conformance test suite** for HermitCrab, covering the full + scope of grammar constructs the engine supports — not just the three reference corpora + (Sena/Indonesian/Amharic) — with systematically-tracked coverage, not ad hoc/history-mined + coverage. +2. Structured so **any engine can be validated against it interchangeably** (the Rust port today; + anything else tomorrow) via a documented, minimal CLI adapter contract — not a shared library + dependency. +3. **Lives in `Machine`** (this repo), independent of where the Rust port ends up. +4. Every word (and every individual parse of a word with more than one) carries a **human-readable + description of what it exercises and why it's in the suite** — not just a fixture-level README. +5. The C# side can **run itself and produce a coverage report**, cross-referencing fixtures against + an explicit checklist of grammar-model constructs, so gaps are visible and actionable rather than + discovered by accident. +6. Coverage includes **negative fixtures** (words that correctly fail to parse), **cross-cutting + fixtures** (deliberate interactions between multiple constructs, not just isolated mechanisms), + and a clearly separated **pathological/stress category** — complexity-focused, not + feature-focused, and excluded from default runs since some are deliberately slow. + +## 3. Non-goals (for this pass) + +- Not migrating the Rust port itself, or deciding the future-repo split's mechanics — this plan only + needs the framework to be *ready* to be pulled in by whatever ends up consuming it. +- Not attempting literal 100% combinatorial coverage of every construct-pair interaction — see §7 for + what "coverage-complete enough" means in practice. +- Not replacing `rust/parity-out/`'s golden corpus-level regression diffing (Sena/Indonesian/Amharic + full-corpus parity) — that stays as-is; this framework is the systematic, per-mechanism layer + underneath it, not a replacement for real-corpus regression testing. + +## 4. Architecture + +### 4.1 Location + +A new top-level directory, `conformance/`, at the repo root — sibling to `src/`, `tests/`, `rust/`. +Existing fixtures move from `rust/conformance/` to `conformance/` via `git mv` (preserves history). +When `rust/` is eventually extracted, `conformance/` stays in `Machine` and gets pulled in by +whatever consumes it (git submodule or sparse checkout pinned to a tagged commit — a mechanics +decision for later, not blocking this plan). + +### 4.2 Fixture format (extends the existing one, doesn't replace it) + +Every fixture directory keeps the proven pieces: +- `grammar.xml` — the `HermitCrabInput` XML grammar. +- `words.txt` — one word per line (unchanged; still what an adapter is fed). +- `expected.tsv` — the existing 5-column `BatchCommand` signature format + (`idx\tword\tms\tstatus\tsignature`), engine-agnostic, order-independent. **This is the only file + an engine adapter's output is diffed against** — descriptions and metadata never leak into the + engine-facing contract. +- `script.txt` — the literal oracle-invocation script (already standard on newer fixtures; backfilled + onto older ones during migration). + +Two new files per fixture: + +- **`manifest.json`** — machine-readable metadata, the input to the coverage report: + ```json + { + "id": "rewrite/simultaneous-epenthesis", + "category": "single-feature", + "constructs": ["RewriteRule.Simultaneous", "Epenthesis"], + "tags": ["phonology"], + "pathological": false, + "oracle": { "worktree": ".worktrees/parse-opt", "commit": "ccf750e6" }, + "provenance": "P13, rust-optimizations-phase2.md" + } + ``` + `category` is one of `single-feature` | `negative` | `cross-cutting` | `pathological`. + `constructs` is a list of entries from the checklist in §6 — this is the field the coverage report + keys on. + +- **`words.md`** (or `.yaml` — see open question in §9) — **the per-word/per-parse description + mandate**. One entry per word in `words.txt`, and where a word has more than one expected parse, + one description per parse: + ```yaml + - word: "buibui" + exercises: > + Simultaneous-mode epenthesis inserts a vowel after every high vowel in one pass; the two + insertion sites (after each of the root's two high vowels) must not feed each other the way + an Iterative pass would. + parses: + - signature: "19|b(u)(b)(u)" + exercises: "the only valid analysis: unapplying both epenthesis sites in a single non-feeding pass yields root 19" + - word: "buuibui" + exercises: "negative case for the same rule -- this shape is NOT reachable by any unapplication order" + expect_fail: true + ``` + This is metadata for humans and the coverage report, never part of the engine-adapter contract — + keeping the adapter contract (§4.3) minimal is what makes "pull in another engine" cheap. + +### 4.3 The adapter protocol (the interchangeability mechanism) + +Documented in `conformance/PROTOCOL.md`. Any engine becomes conformance-testable by implementing: +``` + batch [--start N] +``` +producing the existing 5-column TSV with the existing signature algorithm (sorted, +`;`-joined `morph+morph|shape` tuples per distinct analysis). This is exactly what `hc-rs batch` +and C#'s `BatchCommand` already do today — the protocol doc formalizes what's currently only +implicit in `BatchCommand.cs`'s source, so a third engine could implement it without reading anyone's +code. + +### 4.4 The harness project + +`src/SIL.Machine.Morphology.HermitCrab.Conformance/` (new C# project): +- **Runner**: given `--fixtures --adapter ""`, runs every fixture through + the adapter, diffs against `expected.tsv`, reports pass/fail per fixture and per word. +- **Self-check mode** (no `--adapter`): runs fixtures against the live C# engine directly (via the + now-mainline `BatchCommand`, §5 F0) — this is what catches a fixture going stale when the oracle + itself changes, the lesson LT-22613 taught (the oracle can be wrong and need correcting too). +- **Coverage report**: cross-references every fixture's `manifest.json.constructs` against the full + checklist (§6), and reports, per construct: how many fixtures touch it, whether any are negative, + whether any are cross-cutting, and — for zero/thin-coverage constructs — flags them explicitly. + This is the mechanism John asked for: *"should be able to produce its own coverage report, and + that should be used to actively add grammars and words toward complete coverage"* — the report is + meant to be read by whoever (human or subagent) picks the next fixture to build, not just archived. +- **Pathological fixtures are excluded by default** (`category: pathological` in the manifest); an + explicit `--include-pathological` flag opts in. The runner should also support a per-fixture + timeout/step-cap read from the manifest for this category (see §4.5), so a pathological fixture + reports "within expected complexity budget" rather than hanging the whole suite. + +### 4.5 The four fixture categories + +- **`single-feature`** — today's existing pattern: one mechanism, isolated, minimal grammar. +- **`negative`** — a word that must correctly produce **zero parses** under the grammar. Just as + important as positive fixtures (an engine that over-generates is as wrong as one that + under-generates) and currently has zero dedicated coverage — every existing fixture only pins + words that *do* parse. +- **`cross-cutting`** — deliberately combines two or more constructs in one grammar (e.g. + reduplication interacting with a compounding rule, or a disjunctive affix template slot combined + with Simultaneous-mode epenthesis) — the interactions HISTORY-MATRIX shows are where C#'s own bugs + actually clustered historically, so this category is where new fixtures are highest-value. +- **`pathological`** — per John: **not** about pinning any specific feature's correctness. These + measure the engine's behavior under high combinatorial/complexity load — e.g. deep nested optional + affixes creating exponential candidate counts, overlapping Simultaneous rewrite match sites, + disjunctive-slot combinatorics, a compounding grammar with many co-occurring roots. Each carries in + its manifest an expected complexity budget (step-count ceiling and/or wall-clock bound) rather than + a specific parse signature — the assertion is "stays within budget" (or, for a deliberate + pathological *mis-parse* case, "still produces the mathematically correct answer despite the + adversarial shape," which is a real assertion, just an expensive one to check). Excluded from the + default run; a separate, explicitly-invoked slow-suite job. + +## 5. Phased implementation + +- **F0 — Canonicalize the oracle CLI.** Merge `BatchCommand` into mainline + `src/SIL.Machine.Morphology.HermitCrab.Tool` on this branch (it currently only exists in + `.worktrees/parse-opt`, tied to an unmerged PR). Blocks everything else — every fixture's + provenance depends on this tool being buildable without checking out a side branch. +- **F1 — Stand up `conformance/` and migrate.** `git mv` the 38 existing fixtures from + `rust/conformance/` to `conformance/` (preserve history). Backfill `script.txt` on the older + fixtures that lack it. Write `manifest.json` for every migrated fixture (`category: + single-feature` for all of them at this point — that's an accurate statement about what exists + today). Write `conformance/PROTOCOL.md`. +- **F2 — Build the harness** (`SIL.Machine.Morphology.HermitCrab.Conformance`): runner, self-check + mode, coverage report. Wire `rust/hc-parse/tests/*_conformance.rs` to read fixtures from the new + `conformance/` path (a path change, not a rewrite — the existing replay pattern is unaffected). +- **F3 — Worked examples of the three new categories.** Not full coverage yet — one or two fixtures + per new category, chosen to establish the pattern clearly: one `negative` fixture, one + `cross-cutting` fixture (a real interaction flagged as thin in the checklist, e.g. reduplication × + compounding), one `pathological` fixture (e.g. deeply nested optional affixes). Each with a full + `words.md` per-word/per-parse write-up, so later fixture-writing passes have a concrete template. +- **F4 — Run the coverage report, act on it.** With F0-F3 done, run the harness's coverage report + for real and use its output to drive the next fixture-writing pass(es) — reduplication, + affix-template slot semantics, individual `MorphologicalOutputAction` variants, natural-class + precision, boundary-marker edge cases, and negative/cross-cutting variants of already-covered + constructs. This is explicitly an ongoing, iterative process (per John: coverage reports "should + be used to actively add grammars and words towards complete coverage"), not a single pass — F4 + is the first iteration, not the last. +- **F5 — CI wiring.** Run the full corpus (minus `pathological/`) against the C# oracle itself in + Machine's CI — a real regression gate on HermitCrab, independent of any other engine. Document the + pull-in process for whatever eventually consumes `conformance/` (Rust-repo-to-be or otherwise). + +This plan's first PR covers **F0-F3**: the foundational skeleton, migration, harness, and one worked +example of each new category. F4 (the coverage-driven backfill loop) and F5 (CI) are explicitly +follow-up passes — the coverage report from F3's tiny worked set won't yet be "complete," and +shouldn't be treated as such. + +## 6. Grammar-model construct checklist (coverage-report keys) + +Built from a direct read of `src/SIL.Machine.Morphology.HermitCrab/` + `MorphologicalRules/` + +`PhonologicalRules/`. This is the initial key set for `manifest.json.constructs` / +the coverage report; expected to grow as gaps are found, not fixed in stone. + +| Construct | Existing coverage | +|---|---| +| Stratum (`Linear`/`Unordered` rule order) | unit-test only, no Path-A fixture | +| `AffixProcessRule`: prefix/suffix/circumfix/infix | covered (`affix-shapes/`) | +| `AffixProcessRule`: reduplication (`ReduplicationHint`) | **none** | +| `AffixProcessRule`: subtraction/truncation | covered (`affix-shapes/truncate`) | +| `RealizationalAffixProcessRule` | covered (`realizational/`) | +| `CompoundingRule` | covered, thin (2 fixtures) | +| `MorphologicalOutputAction`: `CopyFromInput`/`InsertSegments` | incidental only | +| `MorphologicalOutputAction`: `ModifyFromInput`/`InsertSimpleContext` | **none isolated** | +| `RewriteRule` Iterative — epenthesis/deletion/feature/expansion/merge | covered, deepest area | +| `RewriteRule` Simultaneous | covered (P13, 4 fixtures) | +| `MetathesisRule` | covered (`metathesis/`) | +| Affix template slots (obligatory/disjunctive/ordering) | **none isolated** — only incidental via real corpora | +| `NaturalClass`: `Segments` vs `FeatureNaturalClass`/`SegmentNaturalClass` precision | partially (`strrep-identity`) | +| Boundary markers (`CharacterDefinitionTable`) | covered (`loader/`) | +| `MorphemeCoOccurrenceRule`/`AllomorphCoOccurrenceRule` | covered (`cooccurrence/`) | +| MPR features/groups | covered (`mpr-groups/`) | +| Guesser/`LexicalGuess` | tracked separately (P11), not yet in this framework | +| Disjunctive allomorphs / free-fluctuation | covered (`allomorphy/`) | +| Stem names | covered (`realizational/stem-name`) | +| Tracing (`TraceType`) | out of `expected.tsv`'s scope entirely — separate mechanism (`trace_diff.py`) | + +Every row above currently has **zero negative fixtures and zero cross-cutting fixtures** — even +"covered" rows are covered only in the single-feature-positive sense. + +## 7. What "coverage-complete enough" means + +Not literal 100% combinatorial coverage (the construct-pair space is too large to be worth +exhausting). The practical bar: +1. Every leaf construct in §6 has at least one `single-feature` fixture (positive) and at least one + `negative` fixture where a meaningful negative case exists for that construct. +2. Every construct-pair that HISTORY-MATRIX shows historically interacting badly in C#'s own commit + history gets a `cross-cutting` fixture. +3. A curated (not exhaustive) set of `pathological` fixtures, one per identified major complexity + driver (Simultaneous rewrite overlap, deep optional-affix nesting, compounding combinatorics, + disjunctive-slot explosion, template-slot backtracking). +4. The coverage report shows no construct at zero coverage. + +## 8. Acceptance criteria for the F0-F3 PR + +- `BatchCommand` builds on `master`-equivalent mainline `src/SIL.Machine.Morphology.HermitCrab.Tool`. +- All 38 existing fixtures live under `conformance/`, each with a `manifest.json`, each verified to + still pass self-check mode against the live C# oracle (this is also the first real test of the + harness itself). +- `conformance/PROTOCOL.md` documents the adapter contract precisely enough that a third party could + implement it without reading `BatchCommand.cs`. +- The harness runs, self-check mode passes on all migrated fixtures, and `--adapter` mode + successfully validates against `hc-rs batch` (proving the Rust port still conforms after the + migration, and proving the adapter model actually works end-to-end). +- At least one fixture in each of `negative`/`cross-cutting`/`pathological`, each with a complete + `words.md`. +- The coverage report runs and produces a real (if still mostly "gap") report against §6's checklist. +- `rust/hc-parse/tests/*_conformance.rs` still pass, reading fixtures from the new path. + +## 9. Open questions (flag, don't block on) + +- `words.md` vs `words.yaml` for the per-word/per-parse descriptions — YAML is more + machine-parseable for the coverage report; Markdown reads better for humans browsing a fixture. + Default to YAML with a comment-friendly structure (a coverage report needs to parse this + reliably); revisit if it reads badly in practice. +- Exact distribution mechanism for pulling `conformance/` into a future separate Rust repo (git + submodule vs. sparse checkout vs. tagged-release archive) — deferred per John's "let's not push + right now, we may be starting a new repo instead." +- Whether `manifest.json`'s `constructs` values should be free-text (grows organically) or a closed + enum validated by the harness (stricter, catches typos, but needs updating whenever a new construct + is added) — default to free-text validated against §6's checklist as a soft warning, not a hard + error, so a genuinely new construct can still be recorded before the checklist is updated. From b04abfed9f6168d5de143c7d38589e3e19a71e66 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 10 Jul 2026 19:36:13 -0400 Subject: [PATCH 2/5] F0: canonicalize BatchCommand onto this branch's HermitCrab.Tool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports BatchCommand.cs verbatim from commit b9b77bfd (the parse-opt-worktree port that already stripped --rule-stats/--parallel, which depend on Morpher members from unmerged perf-optimization PRs not present on this branch). Registers it in Program.cs alongside parse/tracing/test/stats. Verified: dotnet build succeeds, and running `batch` against rust/conformance/rewrite/simultaneous-epenthesis/words.txt reproduces its checked-in expected.tsv's word/status/signature columns exactly (ms and the STARTED sentinel line are excluded from comparison by convention, same as b9b77bfd's own verification and rust/tools/parse_compare.py's semantics). Co-Authored-By: Claude Sonnet 5 F1: import conformance/ fixtures from the rust branch (cross-branch copy) This branch is based on origin/master, not rust -- the Rust port will never land in this repo, so the conformance framework shouldn't be built on top of it. The 38 fixtures under rust/conformance/ only exist in the rust branch's history; since master never had a rust/ directory, this can't be a same- branch git mv (there's nothing to rename from here). Content copied via `git checkout -- rust/conformance` then moved to the new top-level conformance/ path -- byte-identical content to what's on the rust branch, just without shared line history in this repo (the rust branch retains its own copy/history for that content). Co-Authored-By: Claude Sonnet 5 Add grounded F0-F3 implementation scoping notes after F0 + F1-move recon Per instruction to pause before the bulk of implementation: records what's actually done (F0 BatchCommand port + verify, F1 pure git-mv), concrete facts learned from real recon that refine the original plan (BatchCommand's actual CLI shape vs. the design doc's stated adapter protocol; exact 19-fixture script.txt gap list; 14 Rust test files needing a path-depth fix, not "a couple"; no step-count stat exists on this branch), and a small-step ordered plan with one commit per step for the remaining F1/F2/F3 work. Co-Authored-By: Claude Sonnet 5 Correct plan/notes docs after rebuilding onto origin/master Both docs originally described F1 as a same-branch git mv from rust/conformance/ to conformance/, written before this branch was rebuilt onto origin/master (rather than rust) per John's direction that the Rust port will never land in this repo. Corrected to describe the actual mechanism used: a cross-branch content copy, since this branch shares no history with rust/conformance/'s origin. Also re-scopes the implementation notes' "14 Rust test files need a path fix" item as a future rust-branch concern, not something this branch needs to (or can) address, since it no longer contains a rust/ directory at all. Co-Authored-By: Claude Sonnet 5 Expand plan: XAmple as second engine (non-phonology), bug-driven fixture growth policy - Goals 7-8: XAmple conformance via capability filtering; cheap deliberate growth - 4.3: engine capability profiles + per-engine grammar representation in PROTOCOL.md - New 4.6: XAmple design (grammar.xample/ parallels, shared expected.tsv, thin adapter) - New 8: fixtures-ride-along-with-bug-fixes policy - New F6 phase; F2 gains capability filtering; acceptance criteria + open questions updated Co-Authored-By: Claude Fable 5 F1 remainder: script.txt normalization, manifest.json for all 38 fixtures, PROTOCOL.md, README.md - Backfill script.txt onto the 19 fixtures that lacked it and normalize the 19 existing ones to a relative-path form (`batch words.txt expected.tsv`), replacing stale absolute paths baked in from whatever worktree originally generated them. - Add manifest.json to every one of the 38 fixtures (category: single-feature for all). `requires` is derived mechanically by grepping each grammar.xml for / elements (20 fixtures require phonology: all 15 rewrite/*, all 3 metathesis/*, plus affix-shapes/noncontiguous and loader/n2-default-symbol, both of which embed a real phonological rule alongside their primary construct). `constructs` values are drawn from the plan's section 6 checklist, cross-referenced against each fixture's README/grammar. `oracle`/`provenance` record the .worktrees/parse-opt @ ccf750e6 origin confirmed present in every README. - Add conformance/constructs.txt: the section-6 checklist as an editable data file, the coverage report's cross-reference key set. - Add conformance/PROTOCOL.md: the canonical 3-arg single-command adapter contract, the 5-column TSV format and BuildSignature algorithm, comparison semantics, capability profiles, per-engine grammar representation, and the aside on hc.dll's actual (non-conforming) CLI shape needing a one-line script-file wrapper. - Add conformance/README.md: orientation plus the section-8 growth policy and its two cheap-path recipes (new word, new fixture). Co-Authored-By: Claude Fable 5 Plan reframing per John: all-language parser oracle, usable outside Machine; XAmple-ready not XAmple-run Co-Authored-By: Claude Fable 5 Reframe README.md/PROTOCOL.md per John: all-language parser oracle, engine-agnostic Per e502965d's plan reframing: conformance/ is meant to be usable outside Machine entirely, with HermitCrab positioned as the founding/reference oracle rather than the subject being tested. README.md now opens with the "all-language parser oracle" framing and states explicitly that conformance/ is self-contained (no dependency on Machine's build, layout, or C#). PROTOCOL.md's opening reframes the same way: it's a standalone spec any parser can implement, with HermitCrab's C# engine credited as the implementation that generated the ground truth and first implemented the protocol, not a privileged consumer of it. No content/mechanism changes -- wording only. Co-Authored-By: Claude Fable 5 F2: conformance harness (runner, self-check, adapter mode, capability filtering, coverage report) New src/SIL.Machine.Morphology.HermitCrab.Conformance console project, wired into Machine.sln: - Fixture/FixtureManifest: discovers every conformance/**/manifest.json and deserializes it. - SelfCheckEngine: loads grammar.xml via XmlLanguageLoader, builds a Morpher, parses every word in-process, computing signatures via the new SIL.Machine.Morphology.HermitCrab.Tool.SignatureFormat helper (BatchCommand.BuildSignature extracted verbatim into its own public class so the oracle and the harness share one implementation instead of two copies that could drift). - AdapterEngine: shells out to an external `--adapter "