Add ontology-backed semantic layer

AGENT.md

@@ -65,6 +65,10 @@ The data contract these steps assume is normative in [SPEC.md](SPEC.md).
      (`--table edges`): `{src,dst,kind}` by default; an edge may optionally be
      **cited** with a verbatim `quote` + `source_id`, and scrip mints+verifies its
      `anchor` just like a claim (an unverifiable quote fails the batch, exit 1).
+   - if `vault/ontology.yaml` exists, `scrip fact add` also validates entity
+     kinds, edge kinds, and claim predicates against it, and canonicalizes claim
+     predicate aliases before storage. Use `scrip ontology` to check the optional
+     vocabulary directly.
 2. `scrip stamp vault/facts/_meta.yaml` — every append (any table) drops the
    set's `input-hash`, so it deliberately shows STALE until you stamp it.
 3. `scrip verify` (anchors resolve) and `scrip query contradictions` (catch
@@ -92,10 +96,11 @@ reconciliations.
 
 `scrip-harness answer "<question>"` runs the safe form of this ladder: it refuses
 stale artifacts, broken anchors, and open contradictions by default; gathers
-facts/wiki evidence first; falls back to `scrip search` when compiled evidence is
-thin; and accepts the model answer only after every claim citation resolves via
-`scrip span` or every raw quote mints via `scrip anchor`. `--save` writes the
-verified answer to `wiki/explorations/`.
+facts/wiki evidence first; includes bounded graph context when relevant; falls
+back to `scrip search` when compiled evidence is thin; and accepts the model
+answer only after every claim citation resolves via `scrip span` or every raw
+quote mints via `scrip anchor`. Graph context and wiki pages are not citable by
+themselves. `--save` writes the verified answer to `wiki/explorations/`.
 
 ## PROMOTE — turn a good answer into a page
 
@@ -157,6 +162,7 @@ caveat is left to the read-only view layer rather than mutating a stamped page.
 | record provenance hashes after compiling | `scrip stamp [path…]` |
 | check every citation still resolves | `scrip verify` |
 | query the facts layer | `scrip query claims \| entities \| edges \| contradictions \| --sql "…"` |
+| validate optional semantic vocabulary | `scrip ontology` |
 | retrieve source blocks (rung 4) | `scrip search "<question>"` |
 | build the semantic index (optional) | `scrip index` *(needs the `[embeddings]` extra)* |
 | answer with verified citations | `scrip-harness answer "<question>" [--save]` |

CHANGELOG.md

@@ -7,6 +7,24 @@ reference CLI. The file **contract** is versioned separately in
 
 ## [Unreleased]
 
+### Added
+- **Optional ontology vocabulary.** `vault/ontology.yaml` can constrain claim
+  predicates, entity kinds, and edge kinds; `scrip ontology` validates and
+  summarizes it; and claim predicate aliases canonicalize before storage.
+- **External entity identity metadata.** Entity rows may carry optional `uri`,
+  `same_as`, and `external_ids` fields while keeping local `entity/<slug>` ids as
+  the stable contract identity.
+- **Ontology-aware harness drafting.** `scrip-harness extract` and
+  `scrip-harness graph` feed the active ontology summary into model prompts so
+  drafts prefer local vocabulary before `scrip fact add` validates them.
+- **Graph-guided answer context.** `scrip-harness answer` includes bounded,
+  ranked entity/edge context when relevant; this context is not citable, so final
+  citations remain verified claim ids or raw quotes.
+
+### Changed
+- **scrip-harness now requires `scriptoria>=0.9.0`.** The harness ontology
+  prompts rely on deterministic ontology enforcement in the keeper.
+
 ## [0.8.1] — 2026-07-07
 
 scriptoria moves to 0.8.1 for a lower-allocation NDJSON parser. (Released

README.md

@@ -69,6 +69,9 @@ scrip query claims --where "list_contains(tags, 'caching')"
 scrip query contradictions
 scrip query --sql "SELECT source_id, count(*) AS n FROM claims GROUP BY 1 ORDER BY n DESC"
 
+# optional semantic vocabulary: validates fact predicates/kinds when present
+scrip ontology
+
 # rung 4 — retrieve source blocks for an uncompiled question (grep by default)
 scrip search "what makes adding one document expensive?"
 
@@ -79,7 +82,8 @@ scrip similar --title "Compilation over retrieval" --from raw/karpathy-llm-wiki
 ### Optional: answer demo
 
 The optional harness can answer from a green vault with model output constrained
-to verified claim ids or raw quotes. A sanitized fixture is included for demos:
+to verified claim ids or raw quotes; graph neighborhoods may guide the answer
+but are not citable by themselves. A sanitized fixture is included for demos:
 
 ```sh
 scripts/demo_answer.sh --provider openai

SPEC.md

@@ -44,6 +44,7 @@ vault/
   raw/      immutable sources (+ .meta.yaml sidecars)   ← never edited after ingest
   facts/    structured extractions, queryable as data   ← derived from raw/
   wiki/     synthesized prose (concepts, entities)       ← derived from raw/
+  ontology.yaml   optional semantic vocabulary           ← validates facts if present
 ```
 
 ### 2.1 `raw/`
@@ -59,6 +60,9 @@ vault/
 ### 2.2 `facts/`
 Newline-delimited JSON (one record per line), queryable directly as data:
 - `facts/entities.ndjson` — entities (people, works, organizations, systems).
+  Entities have local `entity/<slug>` ids and may optionally carry external
+  identity metadata (`uri`, `same_as`, `external_ids`) without changing their
+  local contract identity.
 - `facts/claims.ndjson` — the claims table (§5).
 - `facts/graph.ndjson` — edges between entities/sources (citation/idea graph). An
   edge is `{src, dst, kind}`; it may optionally be **cited** — carrying a verbatim
@@ -80,6 +84,31 @@ appends without rewrites and diffs line-by-line in git.
 - `wiki/log.md` — an append-only journal of compiles, answers, reconciliations
   (not a tracked derived artifact).
 
+### 2.4 `ontology.yaml` (optional)
+
+`vault/ontology.yaml` is an optional semantic vocabulary. If absent, facts remain
+validated only by their base schema. If present, `scrip fact add` uses it to
+constrain selected free-text fields and canonicalize claim predicates via aliases:
+
+```yaml
+entity_kinds: [tool, concept, person, organization]
+edge_kinds: [depends-on, part-of, cites, alternative-to]
+claim_predicates: [caches, depends-on, contradicts]
+predicate_aliases:
+  stores: caches
+```
+
+- `entity_kinds` constrains `facts/entities.ndjson` `kind`.
+- `edge_kinds` constrains `facts/graph.ndjson` `kind`.
+- `claim_predicates` constrains `facts/claims.ndjson` `predicate`.
+- `predicate_aliases` maps proposed predicates to canonical predicates before
+  storage; alias targets must be listed in `claim_predicates` when that list is
+  present.
+
+The ontology is a checked source file, not a cache. It does not affect staleness
+hashes or provenance anchors; it only makes fact writes stricter and more
+canonical when operators opt in.
+
 ---
 
 ## 3. Identifiers
@@ -94,6 +123,11 @@ appends without rewrites and diffs line-by-line in git.
 A `raw/<slug>` id maps to the file `vault/raw/<slug>.md`. Block-scoped ids append
 `#<block_id>` (§7.2), e.g. `raw/friston-2010#b3f9a1c2d4e5`.
 
+Entity ids remain local, stable `entity/<slug>` identifiers even when an entity
+also carries an external `uri`, `same_as`, or `external_ids` mapping. External
+identity enriches integration; it does not replace the local id used by facts,
+wiki pages, and graph edges.
+
 ---
 
 ## 4. Frontmatter schema (derived artifacts)
@@ -257,7 +291,8 @@ source of truth; deleting `.kb/` and recomputing from `vault/` must be a no-op.
 Answering a question is a descent through rungs; stop at the first that applies:
 
 1. **Consult compiled** — look in `wiki/` and query `facts/` first
-   (index-before-search).
+   (index-before-search). Entity/edge neighborhoods may guide context, but they
+   are not citations by themselves.
 2. **Hit & fresh** (`scrip status` clean) → answer from the compiled layer and
    cite anchors. *Cheapest path; no re-derivation.*
 3. **Hit & stale** → recompile only the stale artifact from its sources
@@ -332,6 +367,12 @@ behaviours; their exit codes are part of the contract surface
   it (§6), so `scrip verify` covers it. Backward-compatible: bare `{src,dst,kind}`
   edges are unaffected, and the optional fields do not change block ids, so the
   manifest `version` stays `2`.
+- **Optional ontology vocabulary (additive).** `vault/ontology.yaml` may constrain
+  entity kinds, edge kinds, and claim predicates, and may canonicalize claim
+  predicate aliases before append. Instances without the file are unaffected.
+  Entity rows may also carry optional `uri`, `same_as`, and `external_ids`
+  metadata. These are backward-compatible schema additions and do not change
+  block ids, so the manifest `version` stays `2`.
 - **v1 → v2 (block ids).** Block ids became content-derived (a digest of the
   normalized block text) instead of positional `b0,b1,…`, making block-precise
   dependencies insertion-stable (§7.2). The id *form* is the only change: a

docs/roadmap.md

@@ -57,3 +57,33 @@ Keep comparisons honest:
 
 Future work should integrate with those layers where useful, rather than
 rebuilding their entire product surface.
+
+## Phase 5 — Semantic Layer
+
+Status: initial implementation started.
+
+The Knowledge Graph Guys' recurring lesson maps cleanly onto the next layer of
+scriptorium: a graph becomes more useful when its identifiers and relationships
+are constrained by explicit semantics, not just free-text labels.
+
+Implemented:
+
+- Optional `vault/ontology.yaml` vocabulary.
+- `scrip ontology` validates and summarizes that vocabulary.
+- `scrip fact add` validates claim predicates, entity kinds, and edge kinds when
+  the ontology exists.
+- Claim predicate aliases canonicalize before storage, improving deterministic
+  contradiction grouping without adding model judgment.
+- Entity rows may carry optional external identity metadata (`uri`, `same_as`,
+  `external_ids`) while retaining local `entity/<slug>` ids as the contract
+  identity.
+- Harness EXTRACT and GRAPH prompts include the active ontology summary, so model
+  drafts prefer local vocabulary before deterministic validation runs.
+- `scrip-harness answer` includes a bounded, ranked `graph_context` packet from
+  relevant entities and edges. This is context-only: final citations still have
+  to be claim ids or verified raw quotes.
+
+Next:
+
+- Add focused views over the graph rather than whole-graph visualization, avoiding
+  the usual unreadable graph hairball.

harness/README.md

@@ -76,8 +76,8 @@ So the model owns *what to say*; `scrip` owns *what is true on disk*.
    contradictions` must be clean by default. Stale artifacts, broken anchors, or
    open contradiction pairs stop the answer before any model call.
 2. **Gather** — the harness ranks claims from `facts/`, reads relevant compiled
-   wiki pages as context, and falls back to `scrip search` when compiled evidence
-   is thin.
+   wiki pages and graph neighborhoods as context, and falls back to `scrip search`
+   when compiled evidence is thin. Wiki/graph context cannot be cited directly.
 3. **Draft** — the selected provider answers from that bounded evidence packet and returns
    structured citation records: either an existing `claim_id` or a verbatim raw
    quote.

harness/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "scrip-harness"
-version = "0.10.2"
+version = "0.11.0"
 description = "Runnable AGENT.md compile loop for scriptorium: drives `scrip` via subprocess + a model provider. Not part of the deterministic scrip core."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -19,9 +19,9 @@ dependencies = [
     "pydantic>=2",
     # Version floor (not the dev path source below) is what ships in the wheel's
     # Requires-Dist, so an installed scrip-harness pulls scriptoria from PyPI.
-    # >=0.8.1: includes cited-edge support plus the LF-only NDJSON parser fix the
-    # harness shells through when it submits fact batches.
-    "scriptoria>=0.8.1",
+    # >=0.9.0: includes optional ontology vocabulary enforcement used by the
+    # harness prompt hints and the deterministic fact writer.
+    "scriptoria>=0.9.0",
 ]
 
 [project.optional-dependencies]

harness/src/scrip_harness/__init__.py

@@ -11,4 +11,4 @@
 way. The harness is optional and lives outside the deterministic core.
 """
 
-__version__ = "0.10.2"
+__version__ = "0.11.0"

harness/src/scrip_harness/answer.py

@@ -24,8 +24,8 @@
     "Avoid generic table headers; quote enough source words to be unique.\n"
     "- Keep the body free of footnote definitions; the harness mints and appends "
     "verified definitions after your draft.\n"
-    "- Do not cite wiki pages directly. They are context; final citations must be "
-    "claim ids or raw quotes."
+    "- Do not cite wiki pages or graph context directly. They are context; final "
+    "citations must be claim ids or raw quotes."
 )
 
 

harness/src/scrip_harness/cli.py

@@ -11,7 +11,7 @@
 import argparse
 import sys
 from pathlib import Path
-from typing import cast
+from typing import Any, cast
 
 
 def _resolve_root(root_arg: str | None) -> Path:
@@ -35,6 +35,22 @@ def _normalize_sources(raw: str) -> list[str]:
     ]
 
 
+def _ontology_for_model(root: Path) -> dict[str, Any] | None:
+    try:
+        import scrip.ontology as ontology_mod
+        from scrip.errors import DataError
+    except ImportError as e:
+        raise RuntimeError(
+            "ontology-aware harness commands require scriptoria>=0.9.0"
+        ) from e
+
+    try:
+        ont = ontology_mod.load(root)
+    except DataError as e:
+        raise RuntimeError(str(e)) from e
+    return ont.summary() if ont.active else None
+
+
 def _add_model_args(parser: argparse.ArgumentParser, *, include_provider: bool = True) -> None:
     if include_provider:
         parser.add_argument(
@@ -204,6 +220,14 @@ def main(argv: list[str] | None = None) -> int:
     chosen_model = args.model
     chosen_provider = cast(model_mod.Provider, getattr(args, "provider", "auto"))
     api_key_file = cast(str | None, getattr(args, "api_key_file", None))
+    needs_ontology = args.command in {"extract", "graph"} or (
+        args.command == "ingest" and args.through in {"extract", "graph"}
+    )
+    try:
+        local_ontology = _ontology_for_model(root) if needs_ontology else None
+    except RuntimeError as e:
+        print(f"scrip-harness: {e}", file=sys.stderr)
+        return 1
 
     if args.command == "ingest":
         def _compile_draft(text: str, *, source_id: str, failures=None):
@@ -216,12 +240,14 @@ def _extract_draft(text: str, *, source_id: str, failures=None):
             return model_mod.draft_extraction(
                 text, source_id=source_id, provider=chosen_provider,
                 model=chosen_model, failures=failures, api_key_file=api_key_file,
+                ontology=local_ontology,
             )
 
         def _graph_draft(text: str, *, source_id: str):
             return model_mod.draft_graph(
                 text, source_id=source_id, provider=chosen_provider,
                 model=chosen_model, api_key_file=api_key_file,
+                ontology=local_ontology,
             )
 
         clean_fn = None
@@ -365,6 +391,7 @@ def extract_draft_fn(text: str, *, source_id: str, failures=None):
                 model=chosen_model,
                 failures=failures,
                 api_key_file=api_key_file,
+                ontology=local_ontology,
             )
 
         extract_sources = None
@@ -404,6 +431,7 @@ def graph_draft_fn(text: str, *, source_id: str):
                 provider=chosen_provider,
                 model=chosen_model,
                 api_key_file=api_key_file,
+                ontology=local_ontology,
             )
 
         try:
@@ -419,7 +447,7 @@ def graph_draft_fn(text: str, *, source_id: str):
         )
         extras = []
         if result["dropped_edges"]:
-            extras.append(f"{len(result['dropped_edges'])} edge(s) dropped (unknown endpoint)")
+            extras.append(f"{len(result['dropped_edges'])} edge(s) dropped")
         if result["skipped_entities"]:
             extras.append(f"{len(result['skipped_entities'])} entity name(s) skipped (no slug)")
         if extras:

harness/src/scrip_harness/extract.py

@@ -5,7 +5,8 @@
 from __future__ import annotations
 
 import json
-from typing import Literal
+from collections.abc import Mapping
+from typing import Any, Literal
 
 from pydantic import BaseModel
 
@@ -53,15 +54,36 @@ class DraftExtraction(BaseModel):
     claims: list[DraftFact]
 
 
-def build_extract_prompt(source_text: str) -> str:
+def _ontology_guidance(ontology: Mapping[str, Any] | None) -> str:
+    if not ontology or not ontology.get("active"):
+        return ""
+    lines = ["\n\nLOCAL ONTOLOGY:"]
+    predicates = ontology.get("claim_predicates") or []
+    aliases = ontology.get("predicate_aliases") or {}
+    if predicates:
+        lines.append("- Use claim `predicate` values only from: " + ", ".join(predicates) + ".")
+    if aliases:
+        rendered = ", ".join(f"{alias} -> {target}" for alias, target in sorted(aliases.items()))
+        lines.append("- Prefer canonical predicates; accepted aliases map as: " + rendered + ".")
+    return "\n".join(lines)
+
+
+def build_extract_prompt(source_text: str, ontology: Mapping[str, Any] | None = None) -> str:
     return (
         "Extract the atomic factual claims from the source below as structured "
         "records. Each claim needs a verbatim `quote`, a subject/predicate/object "
-        "triple, and a polarity.\n\n----- SOURCE -----\n" + source_text
+        "triple, and a polarity."
+        + _ontology_guidance(ontology)
+        + "\n\n----- SOURCE -----\n"
+        + source_text
     )
 
 
-def build_extract_retry_prompt(source_text: str, failures: list[dict]) -> str:
+def build_extract_retry_prompt(
+    source_text: str,
+    failures: list[dict],
+    ontology: Mapping[str, Any] | None = None,
+) -> str:
     """Ask for a replacement for each failed quote, in the reported order. An
     empty replacement quote tells the runner to drop that claim."""
     listing = "\n".join(
@@ -77,7 +99,10 @@ def build_extract_retry_prompt(source_text: str, failures: list[dict]) -> str:
         "Return exactly one replacement claim per failed quote, in the same "
         "order, with the corrected verbatim `quote` and the claim's triple/"
         "polarity. If a claim cannot be supported by a verbatim quote, return it "
-        "with an empty `quote` to drop it.\n\n----- SOURCE -----\n" + source_text
+        "with an empty `quote` to drop it."
+        + _ontology_guidance(ontology)
+        + "\n\n----- SOURCE -----\n"
+        + source_text
     )