refactor(p2m): terminology migration to maximize broad-developer recognition

[Chang Liu (changliu2)]

Summary

Sweeping terminology migration for the public preview, rebased onto current main. The new vocabulary is final — no follow-up rename PR is planned. prompt / scenario remains the canonical multi-turn discriminator.

Because the diff is large (200+ files), this description categorizes the surface-level changes so a reviewer can verify scope without scrolling the file list.

Authoring UX (most important user-facing change)

The behavior spec is now a single self-contained YAML file. There is no longer a companion <name>.md file loaded by the runtime.

# Before                       # After
concept:                       behavior:
  name: my_concept               name: my_behavior
# + companion my_concept.md      description: |
#   (markdown loaded at runtime)   Free-form prose that fully replaces
                                   the old companion .md. Authors put
                                   the entire behavior definition here.

The 6 stock companion .md files were deleted. Authors now write the behavior definition as free-form text directly under behavior.description. No magic filename lookup, no second file to keep in sync.

What the user authors (YAML)

Before	After
concept.name + companion <name>.md file	behavior.name + behavior.description (inline; free-form prose)
pipeline.policy	pipeline.systematize
pipeline.seeds	pipeline.test_set
pipeline.rollout	pipeline.inference
factors: [...]	test_set.stratify.dimensions: [...]
kind: prompt | scenario (per-record JSONL key)	type: prompt | scenario
examples/concepts/ (authoring directory)	examples/behavior_specs/
test_set.tool_source: per_seed	test_set.tool_source: per_test_case (the legacy per_seed value is still accepted but emits a DeprecationWarning)

Pipeline stages (final)

systematize -> test_set -> inference -> judge

What p2m generates (artifacts)

Stage	Artifacts
systematize	systematization.json, taxonomy.json
test_set	test_set.jsonl, stratification.json
inference	inference_set.jsonl (each row = one test-case execution; shape covers chat conversations AND agent action sequences)
judge	scores.jsonl, metrics.json

Per-record discriminator (FINAL)

type: prompt    # single-turn case
type: scenario  # multi-turn case

prompt and scenario are the canonical labels. No further rename to single_turn/multi_turn is planned.

CLI

assert is added as a CLI alias alongside the existing p2m entrypoint. Both work; p2m run keeps working unchanged. (assert is fine as a CLI script name because it never appears as a Python identifier — the package itself is still p2m. import assert would be a SyntaxError.)

[project.scripts]
p2m    = "p2m.cli:cli"
assert = "p2m.cli:cli"   # new alias

The dead p2m analysis taxonomy-logs subcommand was removed. It depended on p2m.analysis.analyze_policies, which was deleted from main (commit 103edd5, "Sync to main") and crashed on invocation. The legacy .eval-archive format it consumed is no longer used anywhere; the subcommand was unreachable.

Renamed internal identifiers (Python)

The seeds_* prefix split into two prefixes that map onto a meaningful distinction. The split is intentional and parallels how every test framework distinguishes the suite (the file you run) from the case (one row inside it).

Before	After	What it refers to
auditor (variable, parameter, class)	tester	—
rollout (variable, parameter, module)	inference	—
concept_*	behavior_*	—
sub_risk_* / failure_mode_* (categorization context)	behavior_category_*	—
seeds_* (collection: file, stage, prompts)	test_set_*	The artifact / stage / prompt set (e.g., test_set.jsonl, pipeline.test_set, prompts/test_set_*.md)
seeds_* (record-level: id, count, schema, template)	test_case_*	One row inside the test set (test_case_id, prompt_test_case_count, TEST_CASE_SCHEMA, TEST_CASE_TEMPLATE)
design_* (artifact context)	stratification_*	—
normalize_design, run_design, design_factors, render_design_catalog	normalize_stratification, run_stratification, stratification_dimensions, render_stratification_catalog	—
transcripts_path, TRANSCRIPTS_FILE	inference_set_path, INFERENCE_SET_FILE	—
render_policy_json, {{policy_json}} (template placeholder)	render_taxonomy_json, {{taxonomy_json}}	Judge prompt assembly — matches the policy.json → taxonomy.json artifact rename
run_systematization_to_policy	run_systematization_to_taxonomy	Stage entry point in p2m/stages/systematization_convert.py
SeedsStageTest (test class)	TestSetStageTest	tests/test_test_set_stage.py

Renamed files

• p2m/stages/rollout.py → p2m/stages/inference.py
• p2m/stages/design.py → p2m/stages/stratification.py
• p2m/analysis/design_metrics.py → p2m/analysis/stratification_metrics.py
• prompts/test_set_design.md → prompts/test_set_stratification.md
• tests/test_design_stage.py → tests/test_stratification_stage.py
• examples/concepts/ → examples/behavior_specs/

Removed

• Companion behavior .md files (6) — single-YAML authoring with free-form behavior.description is the canonical path now
• docs/writing-eval-specs.md rewritten to match the single-YAML model (spec goes in behavior.description; variations go in pipeline.test_set.stratify.dimensions, not top-level)
• Dead p2m analysis taxonomy-logs subcommand + its loader and DEFAULT_LOGS_DIR / DEFAULT_PLOTS_DIR constants (legacy .eval-archive analyzer; underlying module was already gone from main)
• No single_turn / multi_turn rename proposals anywhere in docs — prompt/scenario is final

Viewer UI labels

The viewer was relabeled in this PR to match the canonical artifact vocabulary. Inherited framing ("Conversations", "Transcripts") only fits chat-shaped targets; the inference set also holds agent action sequences, so the labels were generalized:

Surface	Before	After
Inference Preview section header	Available Conversations	Inference set
Audit / preview counters	{n} conversations	{n} results
Drawer header	Conversation · {n} turns	Result · {n} turns
Drawer narrative panel	Conversation summary	Result summary
Drawer loading / error toasts	Loading conversation, Could not load conversation, Fetching the transcript for {id}	Loading result, Could not load result, Fetching the result for {id}
Runtime mode badge	agentic transcript	agentic
Citation source fallback	transcript	message

The word "transcript" is preserved only where it literally means "the stream of messages within one result" (e.g., transcript highlights in evidence captions, No transcript available empty state, citation-resolution error messages). The internal TypeScript types (UnifiedTranscriptRow, etc.) are unchanged in this PR — type-name rename is a follow-up.

Validation

• uv run python -m compileall -q p2m — clean
• uv run pytest -q — 725 passed, 14 skipped, 13 subtests passed
• cd viewer; npm run check — 0 errors / 0 warnings
• CLI smoke: p2m --help, p2m run --help, all 10 example eval configs load via from p2m.config import load_config
• Stale-term grep across README.md, AGENTS.md, CONFIG_REFERENCE.md, docs/, examples/: clean for design.json, transcripts.jsonl, auditor, pipeline.rollout, examples/concepts, sub_risk, seeds.jsonl

Notes

• The docs-site skeleton (site/) was terminology-refactored locally but is kept out of this PR — it lands in a separate docs-site PR.

[Minsoo Thigpen (minthigpen)]

YAML Spec updates
"failure mode" -> rename to "behavior" (currently "concept")
Under pipeline steps:

• "policy" -> "systematize" (produces taxonomy.json)
• "seeds" -> "test set" (produces test_set.json)
• keep "prompt" and "scenario" under test set

[Minsoo Thigpen (minthigpen)]

taxonomy.jsonl["failure_modes"] -> taxonomy.jsonl["behavior_category"]

[Minsoo Thigpen (minthigpen)]

"factors" -> should be renamed to "test_set_dimensions"

[Minsoo Thigpen (minthigpen)]

rename p2m to assert in CLI flags

[Chang Liu (changliu2)]

remaining file terminology migration in scope:

sub_risks -> behavior_categories

systematization.json['behavior']

seeds/transcripts.jsonl["type"]: "prompt" | "scenario"

seeds.jsonl -> test_set.jsonl["dimensions"] and bring design and factors to the test_set generation section in the yaml (e.g., test_set.stratify.model and test_set.stratify.dimensions, instead of standalone design/factors)

test_set.dimensions

p2m run -> assert run

[Chang Liu (changliu2)]

To simplify user experience to only ask for authoring one config file (the .yaml), proposing to merge behavior_spec.md into behavior.name and behavior.description

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

[tangym]

Great PR. A few items from the rename mapping weren't carried through:

1. CLI: assert alias added but rename mapping overstates the change

   The rename mapping says "CLI: assert run (renamed from p2m run)" but the change just adds assert as an alias in pyproject.toml and p2m remains the primary name everywhere. If this is intentional (alias, not rename), suggest updating the mapping wording to avoid confusion.

   Side note: assert as a package name would shadow Python's assert keyword. So import assert is a SyntaxError. Fine as a CLI-only entry point, but worth being intentional about it.

2. SeedsStageTest class name not renamed

   • tests/test_test_set_stage.py:13: class is still SeedsStageTest, should be TestSetStageTest

3. policy_json internals --> taxonomy_json

   The stage output was renamed to taxonomy.json but internal references still use policy:

   • p2m/core/judge.py — render_policy_json() function name + policy_json variable
   • prompts/judge_system.md — {{policy_json}} template placeholder
   • Tests referencing policy_json / render_policy_json

   ~9 occurrences across these files.

4. run_systematization_to_policy --> run_systematization_to_taxonomy

   • systematization_convert.py — function name is still run_systematization_to_policy
   • Callers and tests that reference it

   ~10 occurrences.

Items 2–4 are internal-only (no customer-facing impact) but worth cleaning up for consistency with the rest of the rename.

[Chang Liu (changliu2)]

Thanks tangym — all four items addressed.

1. CLI mapping wording. Updated the PR body. The reality (and the intent) is "assert is added as an alias, p2m stays". Pyproject defines both entry points; p2m run keeps working. Your shadow-of-Python-keyword note is accurate and points to a bigger problem. The current alias solution is a temp workaround. Any ideas is welcome.

2. SeedsStageTest → TestSetStageTest. Renamed (tests/test_test_set_stage.py:13).

3. policy_json / render_policy_json → taxonomy_json / render_taxonomy_json. Renamed across:

• p2m/core/judge.py — __all__, function def, template placeholder call
• prompts/judge_system.md — {{policy_json}} → {{taxonomy_json}}
• tests/test_shared_infra_helpers.py — 4 template strings
• tests/test_exception_handling.py — test_corrupt_policy_json_raises_value_error → test_corrupt_taxonomy_json_raises_value_error

(Left policy_raw parameter alone — it's a separate concept from the rendered JSON placeholder and you didn't flag it. Happy to sweep it too if you'd prefer.)

4. run_systematization_to_policy → run_systematization_to_taxonomy. Renamed in p2m/stages/systematization_convert.py, plus all callers and tests (p2m/stages/systematize.py, tests/test_exception_handling.py, tests/test_stage_runner_smoke.py, tests/test_systematization_convert_stage.py).

Verification: grep across the repo shows 0 remaining occurrences of SeedsStageTest, policy_json, render_policy_json, or run_systematization_to_policy. Affected pytest suites: 69 passed.

Commit: 664dd20.

[Jake Present (jakepresent)]

Directionally this looks good, and the main CI checks are green now. I’m holding approval on two things that look like actual merge blockers:

1. p2m analysis taxonomy-logs ... crashes because p2m/cli.py still imports p2m.analysis.analyze_policies as analyze_taxonomies, but analyze_policies doesn’t exist on this branch. I reproduced locally.

2. docs/writing-eval-specs.md still describes an eval spec as “a short markdown file” and still shows top-level dimensions:. That contradicts the new single-YAML authoring model (behavior.description and pipeline.test_set.stratify.dimensions).

Nonblocking follow-ups: tool_source: per_seed is still user-visible, assert is added as an alias but help/version/examples still say p2m, and the viewer still has some seed-language labels/routes. I wouldn’t block the PR on all of those, but they’re worth deciding intentionally since this PR is meant to be the final terminology sweep.

[Chang Liu (changliu2)]

Thanks Jake Present (@jakepresent) — both blockers fixed, follow-ups addressed below. Commit: 445968e.

Blocker 1 — p2m analysis taxonomy-logs crash. Reproduced locally (same ModuleNotFoundError: p2m.analysis.analyze_policies). Root cause is that analyze_policies.py was deleted in 103edd5 ("Sync to main", Apr 2026) but the lazy import + subcommand body were left behind. The legacy .eval-archive format it consumed is no longer used anywhere, so the whole taxonomy-logs subcommand, its loader (_load_analyze_taxonomies), and the dead DEFAULT_LOGS_DIR / DEFAULT_PLOTS_DIR constants are removed. p2m analysis --help now lists only test-set-metrics.

Blocker 2 — docs/writing-eval-specs.md. Rewritten end-to-end:

• Removed the "short markdown file" framing — the spec is now free-form text inside behavior.description in the YAML, no separate .md file.
• Replaced the top-level dimensions: example with pipeline.test_set.stratify.dimensions:.
• Kept the example shape and the rubric guidance; added a pointer to CONFIG_REFERENCE.md for the full key set.

Follow-up 1 — tool_source: per_seed value lag. Real miss. Renamed the canonical value to per_test_case (matches the existing TOOL_SOURCE_PER_TEST_CASE symbol, which previously held the lagging string). Existing user configs keep working: per_seed is accepted as a deprecated alias and emits a DeprecationWarning. Updated CONFIG_REFERENCE.md, the one example YAML that used it (examples/pipes/health_assistant_generated_tools.yaml), all error messages, all test fixtures, and added focused coverage for both the alias and canonical paths.

Follow-up 2 — assert partial. Help text, --version, and most prose still say p2m — that's intentional for this PR. assert is added as an alias, not a rename. The full brand cutover (uppercase ASSERT, from assertion import ... as the Python-side surface, help-text rewrite, examples) needs a separate design decision because it touches naming for the OSS distribution and runs into the Python keyword collision. Tracking separately; not blocking this PR.

Follow-up 3 — viewer seed-language. Audited. The only user-visible holdouts were:

• Seed Generation stage label in the run page and the monitor page → Test Set Generation
• Prompt is missing a seed id. toast → Prompt is missing a test case id.

Both fixed. Remaining seed* strings in viewer/src/ are URL params (/runs/.../prompt/[seed]/) and internal TypeScript identifiers (PromptSeed, normalizePromptSeeds, etc.) — not user-visible. Type renames are deferred to a viewer-only follow-up so this PR stays scoped.

Validation: full suite pytest -q = 725 passed, 14 skipped, 13 subtests passed. Viewer npm run check = 0/0.

Also acking tangym's approval — thanks for the second pass.