From b1836731ed1a0c812261f322dc848a76a909d6ec Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Apr 2026 15:48:10 +0000 Subject: [PATCH 1/4] =?UTF-8?q?docs(arch):=20LAB-ONLY=20vs=20canonical=20s?= =?UTF-8?q?plit=20=E2=80=94=20anti-hallucination=20guard?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cognitive-shader-driver IS the unified API. Its re-exported super DTO family (ShaderDispatch/ShaderHit/MetaWord/BindSpace/ShaderDriver plus UnifiedStep/OrchestrationBridge) is the ONLY consumer-facing surface. REST/gRPC servers, per-op Wire DTOs, codec_research, codec_bridge, planner_bridge are LAB-ONLY scaffolding for Claude Code backend testing — never shipped to consumers. - Cargo.toml: `lab` umbrella feature (serve+grpc+with-planner+ with-engine) and shader-lab binary. Canonical library surface available with zero features; REST+gRPC coexist behind siblings. - lib.rs: explicit "Canonical / LAB-ONLY" section header + STOP block pointing future sessions at the knowledge doc before they add REST endpoints. - wire / serve / grpc / codec_research / codec_bridge / planner_bridge: `//! **LAB-ONLY.**` banners naming the hallucination failure mode and the canonical alternative. Anti-hallucination docs: - .claude/knowledge/lab-vs-canonical-surface.md — Decision Procedure, Kahneman-Tversky / Dunning-Kruger traps named explicitly, 5-layer stack mapping (L0 ndarray → L4 Planner), list of every canonical re-export. - .claude/agents/BOOT.md — 3 new activation triggers (REST/gRPC/Wire DTO; OrchestrationBridge/UnifiedStep; codec research). - CLAUDE.md — P0 rule for the new knowledge doc. Grounded against THINKING_RECONCILIATION.md, INTEGRATION_PLAN_CS.md, BINDSPACE_MIGRATION_GAP.md, COGNITIVE_SHADER_HYDRATION.md, COGNITIVE_MERGE_MAP.md, SESSION_HANDOFF_PRIORITIES.md, CODEC_INVARIANTS_AND_EXPERIMENTS.md. Research is one consumer of the unified DTO via StepDomain::Ndarray — not the reason the DTO exists. Build: both `cargo check` (canonical, no features) and `cargo check --features lab` compile clean. `cargo test --lib --features lab` → 46 passing. Zero runtime cost on canonical path. https://claude.ai/code/session_01SbYsmmbPf9YQuYbHZN52Zh --- .claude/agents/BOOT.md | 3 + .claude/knowledge/lab-vs-canonical-surface.md | 193 ++++++++++++++++++ CLAUDE.md | 11 + crates/cognitive-shader-driver/Cargo.toml | 20 ++ .../src/codec_bridge.rs | 14 +- .../src/codec_research.rs | 12 +- crates/cognitive-shader-driver/src/grpc.rs | 11 +- crates/cognitive-shader-driver/src/lib.rs | 99 +++++++-- .../src/planner_bridge.rs | 18 +- crates/cognitive-shader-driver/src/serve.rs | 10 +- crates/cognitive-shader-driver/src/wire.rs | 11 +- 11 files changed, 366 insertions(+), 36 deletions(-) create mode 100644 .claude/knowledge/lab-vs-canonical-surface.md diff --git a/.claude/agents/BOOT.md b/.claude/agents/BOOT.md index c0e215d6..a4fef2dc 100644 --- a/.claude/agents/BOOT.md +++ b/.claude/agents/BOOT.md @@ -159,6 +159,9 @@ documents listed in its trigger row BEFORE producing output. | **Story arc / ONNX emergence** | ripple-architect + contradiction-cartographer | crystal-quantum-blueprints.md (§Quantum mode), endgame-holographic-agi.md | | **Argmax / codec / PolarQuant** | truth-architect + family-codec-smith | fractal-codec-argmax-regime.md (ORTHOGONAL to grammar work) | | **Cross-repo harvest** | savant-research | cross-repo-harvest-2026-04-19.md, linguistic-epiphanies-2026-04-19.md | +| **REST / gRPC / Wire DTO / endpoint / serve / shader-lab / external API** | integration-lead + truth-architect | **lab-vs-canonical-surface.md (MANDATORY — prevents System-1 "add another REST endpoint" hallucination)** | +| **OrchestrationBridge / UnifiedStep / StepDomain / BridgeSlot** | integration-lead | lab-vs-canonical-surface.md, cam-pq-unified-pipeline.md | +| **codec research / calibrate / probe / tensors op** | integration-lead + truth-architect | lab-vs-canonical-surface.md (research is just a consumer, not the canonical) | **The insight update cycle:** diff --git a/.claude/knowledge/lab-vs-canonical-surface.md b/.claude/knowledge/lab-vs-canonical-surface.md new file mode 100644 index 00000000..a2c4bf49 --- /dev/null +++ b/.claude/knowledge/lab-vs-canonical-surface.md @@ -0,0 +1,193 @@ +# LAB-ONLY vs Canonical Surface — Anti-Hallucination Guard + +> READ BY: all agents touching cognitive-shader-driver, REST/gRPC +> endpoints, Wire DTOs, OrchestrationBridge, UnifiedStep, codec research, +> planner integration, or anything that looks like "external API" + +## Status: FINDING (architectural invariant — do not drift) + +Date: 2026-04-20 +Related PRs: #220, #221 + +--- + +## The One-Line Rule + +**`cognitive-shader-driver` IS the unified API.** Its public surface — +the super DTO family `ShaderDispatch` / `ShaderHit` / `MetaWord` / +`BindSpace*` columns / `ShaderDriver` / `ShaderSink`, plus +`UnifiedStep` + `OrchestrationBridge` for cross-domain composition — +is the ONLY canonical consumer surface. REST, gRPC, per-op Wire DTOs, +the shader-lab binary, codec research endpoints, and every +`Wire*Request` type are LAB-ONLY scaffolding. Everything compiles into +one binary (`shader-lab`); consumers never see the scaffolding. + +## Two Canonical Layers (both live in `cognitive-shader-driver`'s re-exports) + +1. **Per-cycle super DTO** (the hot path every consumer walks): + `ShaderDispatch` → `ShaderDriver` → `ShaderHit` + `MetaWord` + emission through `ShaderSink`. This is the unified dispatch shape + for one cognitive cycle. `BindSpace` columns are the substrate; + `MetaFilter` + `StyleSelector` configure the sweep. + +2. **Cross-domain composition** (when one cycle is part of a larger + orchestration): `UnifiedStep` routed through + `OrchestrationBridge` by `StepDomain` (Crew / Ladybug / N8n / + LanceGraph / Ndarray / Thinking / Query / Semantic / Persistence / + Inference / Learning). Every domain implements the same trait; + research is just one domain. + +Both layers are zero-dep (defined in `lance-graph-contract`, +re-exported by `cognitive-shader-driver`). Both are always compiled. +Both are consumer-facing. + +## What "LAB-ONLY" Means Concretely + +These modules are gated behind `--features lab` (or its components: +`serve` / `grpc` / `with-planner` / `with-engine`) and are NOT part of +the canonical library surface: + +| Module | Purpose | LAB-only reason | +|---|---|---| +| `cognitive-shader-driver/src/wire.rs` | Per-op REST/proto DTOs (WireDispatch, WireCalibrate, WireProbe, WireTensors, WirePlan, WireRunbook) | Test-transport convenience; canonical DTO is `UnifiedStep` | +| `cognitive-shader-driver/src/serve.rs` | Axum REST server with `/v1/shader/*` handlers | HTTP transport for Claude Code backend testing only | +| `cognitive-shader-driver/src/grpc.rs` | tonic gRPC service | Same as REST, different wire format | +| `cognitive-shader-driver/src/codec_research.rs` | Backing logic for calibrate/probe/tensors ops | Research is one consumer, not the canonical architecture | +| `cognitive-shader-driver/src/codec_bridge.rs` | `CodecResearchBridge: OrchestrationBridge` for StepDomain::Ndarray | The research consumer's bridge impl — canonical trait, lab-only consumer | +| `cognitive-shader-driver/src/planner_bridge.rs` | Per-op WirePlan shortcut → PlannerAwareness | `PlannerAwareness` already impls OrchestrationBridge directly; this is a lab test adapter only | + +The canonical surface — all of it re-exported from +`cognitive-shader-driver` so consumers depend on one crate: + +| Surface (via `cognitive-shader-driver` re-export) | Role | +|---|---| +| `ShaderDispatch`, `ShaderHit`, `ShaderBus`, `ShaderSink`, `NullSink`, `ShaderCrystal`, `ShaderResonance`, `MetaWord`, `MetaFilter`, `MetaSummary`, `ColumnWindow`, `EmitMode`, `RungLevel`, `StyleSelector`, `CognitiveShaderDriver` | Per-cycle super DTO family (from `contract::cognitive_shader`) | +| `BindSpace`, `BindSpaceBuilder`, `EdgeColumn`, `FingerprintColumns`, `MetaColumn`, `QualiaColumn` | L1 BindSpace columns | +| `CognitiveShaderBuilder`, `ShaderDriver` | L2 driver (the shader IS the driver) | +| `engine_bridge::{UnifiedStyle, UNIFIED_STYLES, unified_style, ingest_codebook_indices, dispatch_from_top_k, write_qualia_17d, read_qualia_17d, persist_cycle, EngineBusBridge}` | Engine bridge utilities | +| `GateDecision`, `MergeMode` (from `contract::collapse_gate`) | L3 CollapseGate protocol | +| `OrchestrationBridge`, `UnifiedStep`, `StepDomain`, `BridgeSlot` (from `contract::orchestration`) | Cross-domain composition | +| `Blackboard`, `BlackboardEntry`, `ExpertCapability` (from `contract::a2a_blackboard`) | Layer-1 runtime A2A bus | + +## 5-Layer Stack Mapping (per `docs/INTEGRATION_PLAN_CS.md`) + +``` +L4 Planner strategies → lance-graph-planner (canonical crate) +L3 CollapseGate → contract::collapse_gate (canonical DTO) +L2 CognitiveShader → contract::cognitive_shader + driver::{driver,engine_bridge} +L1 BindSpace → driver::bindspace (struct-of-arrays columns) +L0 ndarray SIMD → ndarray crate (F32x16 / U8x64 / F16x32 / F64x8) +``` + +Every layer has its canonical surface in a zero-dep or feature-free +module. The lab transport (REST/gRPC) is **orthogonal to the stack** — +it's a debug probe, not a layer. + +## The Failure Mode This Doc Prevents + +Future sessions reading this crate see `/v1/shader/dispatch`, +`/v1/shader/calibrate`, `/v1/shader/probe`, `/v1/shader/plan` and think: + +> "There's a REST API. I'll add `/v1/shader/` for this new +> feature." + +**That is the hallucination.** The REST endpoints are not the canonical +API. Adding more of them does not extend the consumer surface — it only +bloats the lab transport. + +### The cognitive traps (name them to dodge them) + +- **Kahneman-Tversky System-1 easy route.** Pattern-matching on "REST + server exists → add REST endpoint" is fast and feels productive. The + System-2 correct move is slower: read this doc, extend + `OrchestrationBridge` / add a new `StepDomain` variant in the + contract, implement the trait in the relevant consumer crate, and + let the lab transport dispatch the new `step_type` automatically + with zero new endpoint code. +- **Dunning-Kruger confidence.** A session that sees axum + tonic + + `Wire*Request` types assumes it understands the architecture. It + does not. The architecture is `OrchestrationBridge`. The HTTP layer + is a test harness. Confidence about "the API" without reading this + doc is the confidence zone where hallucinations ship. +- **Availability bias.** Because REST examples in training data are + everywhere, the easy path is "write more REST." Our stack is + deliberately different: one trait, one `UnifiedStep`, N bridges. + +## The Decision Procedure (paste this into the session's thinking) + +Before adding ANY new endpoint, handler, or Wire DTO, answer in order: + +1. **Does the canonical bridge already handle this?** Check + `StepDomain` variants (Crew / Ladybug / N8n / LanceGraph / Ndarray / + future) and existing `OrchestrationBridge` impls. If an existing + bridge can dispatch the new operation via a new `step_type` string, + **stop — no new endpoint is needed**. The lab transport already + routes `UnifiedStep` through whichever bridge claims the domain. + +2. **Is a new StepDomain needed?** If yes, add the variant in + `lance-graph-contract::orchestration::StepDomain`. Implement + `OrchestrationBridge` for the consumer in that consumer's crate + (e.g. the way `PlannerAwareness` does it in + `lance-graph-planner/src/orchestration_impl.rs`). **Do not add a + new `/v1/` endpoint.** + +3. **Is this purely a lab test convenience?** If and only if the + answer is unambiguously yes (e.g., "I want a curl-friendly + per-op shortcut for debugging"), add it under `src/wire.rs` + + `src/serve.rs` or `src/grpc.rs` and document that it routes + through the canonical bridge internally. Mark it `LAB-ONLY` in + the module doc. + +4. **Never**: + - Publish a per-op endpoint as "the API" in PR descriptions or + knowledge docs. Canonical is `UnifiedStep` via the bridge. + - Add a per-op DTO outside `wire.rs`. + - Introduce a new REST handler that has its own dispatch logic. + Handlers must construct a `UnifiedStep` (or delegate to an + existing bridge) and return the bridge's result. + - Expose HTTP/gRPC types (axum / tonic / prost) from any + crate-root re-export without the feature gate. + +## Feature Gate Reference + +```toml +# crates/cognitive-shader-driver/Cargo.toml +[features] +default = [] # canonical library only +with-engine = ["dep:thinking-engine"] # lab: engine consumer +with-planner = ["dep:lance-graph-planner"] # lab: planner consumer +serve = ["dep:serde", "dep:serde_json", + "dep:axum", "dep:tokio"] # lab: REST transport +grpc = ["dep:prost", "dep:tonic", + "dep:tonic-build", "dep:tokio"] # lab: gRPC transport +lab = ["serve", "grpc", "with-engine", + "with-planner"] # umbrella: the shader-lab binary +``` + +All the lab features coexist — REST and gRPC are siblings, both under +the lab umbrella. Neither is a subset of the other. + +## Related Invariants + +- The whole lab surface compiles into ONE binary (`shader-lab`). There + is no service split; consumers embed the library. +- The canonical bridge has no runtime cost when lab features are off — + `OrchestrationBridge` is a plain trait; `UnifiedStep` is a plain + struct; `StepDomain` is a plain enum. +- `CodecResearchBridge` is a consumer like any other — it happens to + need the lab features because it only exists to back the research + Wire DTOs, but the trait it implements is canonical. + +## Where to Read Next + +- `crates/lance-graph-contract/src/orchestration.rs` — the canonical trait + DTO. +- `crates/lance-graph-planner/src/orchestration_impl.rs` — the reference consumer impl. +- `crates/cognitive-shader-driver/src/codec_bridge.rs` — the lab-only consumer impl for research. +- `.claude/knowledge/cam-pq-unified-pipeline.md` — the pipeline doc that produced this invariant. +- `.claude/knowledge/cognitive-shader-architecture.md` — the broader shader-as-driver architecture. + +--- + +**If you are a future session and this doc is loaded, you are on the +correct path. If you are about to add a REST handler without having +read this section, stop and re-read the Decision Procedure above.** diff --git a/CLAUDE.md b/CLAUDE.md index e97a2080..d32661da 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -535,6 +535,7 @@ SIBLING REPOS: .claude/knowledge/two-basin-routing.md — Two-basin doctrine, representation routing, pairwise rule, attribution .claude/knowledge/encoding-ecosystem.md — MANDATORY: full encoding map, synergies, read-before-write checklist .claude/knowledge/frankenstein-checklist.md — Composition failure modes (VibeTensor §7), boundary test matrix +.claude/knowledge/lab-vs-canonical-surface.md — MANDATORY before touching REST/gRPC/Wire DTO/endpoint/shader-lab (prevents "add another REST endpoint" hallucination) .claude/CALIBRATION_STATUS_GROUND_TRUTH.md — OVERRIDE: read BEFORE any SESSION_*.md .claude/PLAN_BF16_DISTANCE_TABLES.md — 5-phase plan for BF16 distance tables .claude/TECHNICAL_DEBT_SIGNED_SESSION.md — 56% useful, 44% bypass (honest review) @@ -548,6 +549,16 @@ codec, encoding, distance, compression, or representation work.** This is the map of all 8+ encoding representations, their crate locations, their invariants, their synergies, and their FINDING/CONJECTURE status. Never guess architecture. +**P0 Rule: `.claude/knowledge/lab-vs-canonical-surface.md` must be read BEFORE +any work that mentions REST, gRPC, Wire DTOs, `/v1/shader/*` endpoints, the +shader-lab binary, `OrchestrationBridge`, `UnifiedStep`, codec research ops, +or "external API".** The canonical consumer surface is `UnifiedStep` via +`OrchestrationBridge` — the REST/gRPC server and per-op Wire DTOs are +LAB-ONLY scaffolding. Adding another `/v1/` endpoint is the +Kahneman-Tversky System-1 easy path and is nearly always wrong; extending +the canonical bridge is the System-2 correct move. See the Decision +Procedure in that doc before writing a single new handler. + Every `.claude/knowledge/` document has a `READ BY:` header listing which agents MUST load it before producing output in that domain. When a knowledge trigger fires (see `.claude/agents/BOOT.md § Knowledge Activation Protocol`), the relevant diff --git a/crates/cognitive-shader-driver/Cargo.toml b/crates/cognitive-shader-driver/Cargo.toml index e2d62262..1c6e395b 100644 --- a/crates/cognitive-shader-driver/Cargo.toml +++ b/crates/cognitive-shader-driver/Cargo.toml @@ -23,6 +23,15 @@ name = "shader-grpc" path = "src/bin/grpc.rs" required-features = ["grpc"] +# shader-lab — the ONE lab binary. Compiles every endpoint behind the +# unified DTO (UnifiedStep via OrchestrationBridge). Per-op endpoints +# inside are lab-test shortcuts that dispatch through the same bridge +# and are NEVER the canonical consumer surface. +[[bin]] +name = "shader-lab" +path = "src/bin/serve.rs" +required-features = ["lab"] + [dependencies] lance-graph-contract = { path = "../lance-graph-contract" } p64-bridge = { path = "../p64-bridge" } @@ -58,4 +67,15 @@ with-planner = ["dep:lance-graph-planner"] serve = ["dep:serde", "dep:serde_json", "dep:axum", "dep:tokio"] grpc = ["dep:prost", "dep:tonic", "dep:tonic-build", "dep:tokio"] +# `lab` — umbrella switch for the single shader-lab binary. Enables every +# endpoint (REST + gRPC), the planner bridge, the thinking-engine bridge, +# and the research consumer (codec_research + CodecResearchBridge). +# +# Architecture note: the canonical surface is `UnifiedStep` via +# `OrchestrationBridge` from lance-graph-contract. The `lab` feature only +# adds test-convenience wrappers (per-op Wire DTOs, HTTP/gRPC transports) +# on top of that canonical trait. Research is just one consumer of the +# unified DTO, not the reason the DTO exists. +lab = ["serve", "grpc", "with-engine", "with-planner"] + [dev-dependencies] diff --git a/crates/cognitive-shader-driver/src/codec_bridge.rs b/crates/cognitive-shader-driver/src/codec_bridge.rs index 22ad6008..be43ccca 100644 --- a/crates/cognitive-shader-driver/src/codec_bridge.rs +++ b/crates/cognitive-shader-driver/src/codec_bridge.rs @@ -1,8 +1,16 @@ -//! OrchestrationBridge impl for codec research — owns `StepDomain::Ndarray`. +//! **LAB-ONLY consumer.** `OrchestrationBridge` impl for codec research — +//! owns `StepDomain::Ndarray`. +//! +//! The canonical architecture is `OrchestrationBridge` + `UnifiedStep` in +//! the contract. Research is just one consumer plugged into that trait; +//! production consumers plug in the same way under different `StepDomain` +//! values (Crew / Ladybug / N8n / LanceGraph / Thinking / Query / Semantic +//! / Persistence / Inference / Learning). The architecture does not +//! revolve around this consumer. //! //! Dispatches `nd.calibrate`, `nd.probe`, `nd.tensors` step-types through -//! the codec_research module. This complements the LanceGraph bridge on -//! the planner — together they cover `lg.*` + `nd.*` domains. +//! the codec_research module. Combined with the LanceGraph bridge on the +//! planner, together they cover `lg.*` + `nd.*` domains. //! //! Consumers combine bridges: `Vec>` and route //! each step to whichever bridge reports `domain_available() = true`. diff --git a/crates/cognitive-shader-driver/src/codec_research.rs b/crates/cognitive-shader-driver/src/codec_research.rs index eaf6a037..f474ed3a 100644 --- a/crates/cognitive-shader-driver/src/codec_research.rs +++ b/crates/cognitive-shader-driver/src/codec_research.rs @@ -1,13 +1,17 @@ -//! Codec research — backing logic for the `/v1/shader/{tensors,calibrate,probe}` -//! DTO operations on the unified shader-driver API. +//! **LAB-ONLY.** Codec research — one consumer of the unified DTO. +//! +//! Research is not the canonical architecture; it's a consumer like any +//! other domain. It plugs into `OrchestrationBridge` via +//! `CodecResearchBridge` (see `codec_bridge.rs`) under `StepDomain::Ndarray`. +//! The functions below are the lab-convenience backing logic for the per-op +//! Wire DTOs; the canonical dispatch path is `UnifiedStep` through the +//! bridge. //! //! Reuses: //! - `ndarray::hpc::cam_pq::{train_geometric, CamCodebook}` — production codec //! - `ndarray::hpc::safetensors::read_safetensors_header` — tensor directory //! - `ndarray::hpc::gguf::read_tensor_f32` — BF16/F16/F32 dequant //! - `lance_graph_contract::cam::route_tensor` — CamPq / Passthrough / Skip -//! -//! Zero new feature gates — rides on the existing `serve` / `grpc`. use std::fs::File; use std::io::BufReader; diff --git a/crates/cognitive-shader-driver/src/grpc.rs b/crates/cognitive-shader-driver/src/grpc.rs index 4f1f2bd7..f1219eba 100644 --- a/crates/cognitive-shader-driver/src/grpc.rs +++ b/crates/cognitive-shader-driver/src/grpc.rs @@ -1,7 +1,12 @@ -//! gRPC service — protobuf interface for live shader testing. +//! **LAB-ONLY.** gRPC service — protobuf interface for live shader testing +//! in the Claude Code backend. Not shipped to consumers. //! -//! Behind `--features grpc`. Uses tonic + prost. Same quarantine as REST: -//! debug-only, never in production binary. +//! Same story as `serve.rs`: the canonical consumer surface is +//! `UnifiedStep` + `OrchestrationBridge` in the library; this module just +//! exposes that bridge over tonic/prost for test convenience. +//! +//! Behind `--features grpc` (or the umbrella `--features lab`). Uses +//! tonic + prost. Never in any production binary. //! //! ```bash //! # Start gRPC server: diff --git a/crates/cognitive-shader-driver/src/lib.rs b/crates/cognitive-shader-driver/src/lib.rs index fa374b00..1e4ff3db 100644 --- a/crates/cognitive-shader-driver/src/lib.rs +++ b/crates/cognitive-shader-driver/src/lib.rs @@ -34,46 +34,111 @@ //! - **Commit sinks** (`ShaderSink` trait; `NullSink` as zero-cost default) //! - **Feature gates** (`with-engine` pulls thinking-engine; core stays lean) //! - **No forward pass at runtime** — bgz17 distance is precomputed +//! +//! --- +//! +//! ## STOP — Canonical vs LAB-ONLY (read before touching this crate) +//! +//! **Do NOT add new REST endpoints, new per-op Wire DTOs, or new +//! HTTP/gRPC handlers without reading +//! `.claude/knowledge/lab-vs-canonical-surface.md` first.** +//! +//! **`cognitive-shader-driver` IS the unified API.** Its public +//! re-exports form the super DTO family that every consumer speaks: +//! +//! - **Per-cycle hot path:** `ShaderDispatch` → `ShaderDriver` → +//! `ShaderHit` + `MetaWord` via `ShaderSink`. `BindSpace*` columns +//! are the substrate. +//! - **Cross-domain composition:** `UnifiedStep` routed via +//! `OrchestrationBridge` by `StepDomain`. Every consumer +//! (thinking-engine, planner, codec research) plugs in through this +//! trait. +//! +//! Both layers are zero-dep, always compiled, and consumer-facing. +//! That is the stable downstream-visible API. +//! +//! Everything else in this crate — `wire.rs`, `serve.rs`, `grpc.rs`, +//! `codec_research.rs`, `codec_bridge.rs`, `planner_bridge.rs`, and the +//! per-op `Wire*` DTOs — is **LAB-ONLY** scaffolding. It exists so the +//! Claude Code backend can test the canonical bridge over HTTP/gRPC. +//! It is **not** part of the consumer API. Future sessions must not +//! "extend the REST API" with new per-op endpoints; they must extend the +//! canonical bridge and let the lab transport follow automatically. +//! +//! Feature layout: +//! +//! ```text +//! default = canonical library surface only (no HTTP, no gRPC, no wire DTOs) +//! serve = lab REST transport +//! grpc = lab gRPC transport +//! with-engine = thinking-engine consumer wired in +//! with-planner = planner consumer wired in +//! lab = umbrella: all of the above → the shader-lab binary +//! ``` +//! +//! Research (codec_research, codec_bridge) is gated under serve/grpc +//! because it only exists to be tested through the lab transport. It is +//! one consumer of the unified DTO — not the reason the DTO exists. #![warn(rust_2018_idioms)] +// ────────────────────────────────────────────────────────────────────── +// Canonical surface — always compiled. Consumers depend on exactly this. +// ────────────────────────────────────────────────────────────────────── +// +// The canonical consumer-facing DTO is `UnifiedStep` + `OrchestrationBridge` +// re-exported from `lance-graph-contract` (see the `pub use` block below). +// Everything below dispatches through that trait; research, planner, and +// engine are just consumers that plug in via the bridge. pub mod bindspace; pub mod driver; pub mod auto_style; pub mod engine_bridge; pub mod sigma_rosetta; -// Debug-only: REST server + wire types. NEVER in production binary. -// serde is a debugging tool, not a runtime dependency. +// ────────────────────────────────────────────────────────────────────── +// LAB-ONLY modules — compiled only into the shader-lab binary. Never +// shipped to downstream consumers; never part of the canonical API. +// ────────────────────────────────────────────────────────────────────── +// +// Umbrella switch: `--features lab` turns on everything here at once. +// Individual features (`serve`, `grpc`, `with-planner`, `with-engine`) +// remain selectable for narrower lab builds. +// +// Architecture rule: these modules expose per-op Wire DTOs and REST/gRPC +// transports for test convenience in the Claude Code backend. They all +// ultimately route through `OrchestrationBridge` + `UnifiedStep` — the +// canonical surface. No business logic lives here that isn't also +// reachable through the canonical bridge. + +// Per-op Wire DTOs (REST + protobuf). LAB-ONLY. #[cfg(feature = "serve")] pub mod wire; + +// Axum REST server. LAB-ONLY. #[cfg(feature = "serve")] pub mod serve; -// gRPC: protobuf + tonic. Also debug-only. +// tonic gRPC server. LAB-ONLY. #[cfg(feature = "grpc")] pub mod grpc; -// Codec research (calibrate / tensors / probe) — same debug quarantine -// as serve. Rides on the unified shader-driver API surface; no new feature -// gate. Runs the production CAM-PQ codec from ndarray against safetensors -// tensors selected via route_tensor from lance-graph-contract. +// Codec research consumer — backing logic for the research Wire DTOs. +// LAB-ONLY. Research is one consumer of the unified DTO, not its reason. #[cfg(any(feature = "serve", feature = "grpc"))] pub mod codec_research; -// Planner bridge — delegates WirePlan DTOs to lance-graph-planner's -// PlannerAwareness. Optional, gated on `with-planner`. Same EmbedAnything -// pattern as `with-engine`. Without this feature the shader-driver REST -// server still works; /v1/shader/plan returns 503. -#[cfg(feature = "with-planner")] -pub mod planner_bridge; - -// OrchestrationBridge impl for codec research — owns StepDomain::Ndarray. -// Dispatches nd.calibrate / nd.probe / nd.tensors through codec_research. -// Complement to planner's LanceGraph bridge. +// OrchestrationBridge impl for the codec research consumer — owns +// StepDomain::Ndarray. LAB-ONLY (sits next to its backing logic). #[cfg(any(feature = "serve", feature = "grpc"))] pub mod codec_bridge; +// Planner bridge — lab test-shortcut for the per-op WirePlan DTOs. +// PlannerAwareness implements OrchestrationBridge directly in the +// planner crate; that's the canonical path. LAB-ONLY. +#[cfg(feature = "with-planner")] +pub mod planner_bridge; + pub use lance_graph_contract::cognitive_shader::{ CognitiveShaderDriver, ColumnWindow, EmitMode, MetaFilter, MetaSummary, MetaWord, NullSink, RungLevel, ShaderBus, ShaderCrystal, ShaderDispatch, ShaderHit, diff --git a/crates/cognitive-shader-driver/src/planner_bridge.rs b/crates/cognitive-shader-driver/src/planner_bridge.rs index ce8a2936..fdf0051a 100644 --- a/crates/cognitive-shader-driver/src/planner_bridge.rs +++ b/crates/cognitive-shader-driver/src/planner_bridge.rs @@ -1,12 +1,16 @@ -//! Planner bridge — delegates `WirePlan*` DTOs to `lance-graph-planner`. -//! -//! Behind `--features with-planner`. Same pattern as `with-engine`: -//! heavy dep is optional; absence returns a clear runtime error. +//! **LAB-ONLY adapter.** Planner bridge — delegates `WirePlan*` DTOs to +//! `lance-graph-planner`. //! //! Per INTEGRATION_PLAN_CS.md §5-layer-stack, Layer 4 is the planner. -//! The shader-driver (Layer 2) exposes plan operations through the same -//! unified REST/gRPC endpoint and delegates execution here. Nothing -//! below Layer 4 reimplements planning. +//! `PlannerAwareness` already implements `OrchestrationBridge` directly +//! (see `lance-graph-planner/src/orchestration_impl.rs`); that's the +//! canonical consumer surface. This module only exists as a test shortcut +//! that converts the per-op `WirePlan*` DTOs into direct calls for the +//! lab REST/gRPC transports. +//! +//! Behind `--features with-planner` (or the umbrella `--features lab`). +//! Same pattern as `with-engine`: heavy dep is optional; absence returns +//! a clear runtime error. use std::time::Instant; diff --git a/crates/cognitive-shader-driver/src/serve.rs b/crates/cognitive-shader-driver/src/serve.rs index bfa07937..10969bd0 100644 --- a/crates/cognitive-shader-driver/src/serve.rs +++ b/crates/cognitive-shader-driver/src/serve.rs @@ -1,4 +1,12 @@ -//! Axum REST server — external API for live testing cognitive shader cycles. +//! **LAB-ONLY.** Axum REST server for live testing cognitive shader cycles +//! in the Claude Code backend. Not shipped to consumers. +//! +//! Canonical consumer surface is `UnifiedStep` + `OrchestrationBridge` in +//! the library. This server just exposes that bridge over HTTP for test +//! convenience. Per-op endpoints (`/v1/shader/{calibrate,probe,tensors, +//! dispatch,plan}`) are thin adapters that build a `UnifiedStep` and +//! dispatch through the same bridge; the canonical endpoint is +//! `/v1/shader/route`. //! //! ```bash //! cargo run -p cognitive-shader-driver --features serve --bin shader-serve diff --git a/crates/cognitive-shader-driver/src/wire.rs b/crates/cognitive-shader-driver/src/wire.rs index 9a521c9b..e3bc7c31 100644 --- a/crates/cognitive-shader-driver/src/wire.rs +++ b/crates/cognitive-shader-driver/src/wire.rs @@ -1,4 +1,13 @@ -//! Wire types for the external REST + protobuf API. +//! **LAB-ONLY.** Wire types for the REST + protobuf transports used by the +//! shader-lab binary. Not part of the canonical consumer surface. +//! +//! The canonical consumer surface is `UnifiedStep` + `OrchestrationBridge` +//! from `lance-graph-contract`. Everything in this file — per-op DTOs +//! (`WireDispatch`, `WireCalibrateRequest`, `WireProbeRequest`, +//! `WireTensorsRequest`, `WirePlanRequest`, `WireRunbookRequest`) — is +//! test-convenience scaffolding that ultimately dispatches through the +//! same bridge. Consumers (including the research consumer) speak +//! `UnifiedStep`, not these per-op shortcuts. //! //! These are the EXTERNAL types — serde + prost, owned strings, no lifetimes. //! Internal types in lance-graph-contract stay zero-dep and zero-copy. From 62530f4a4cbadf5900a39d830cbbe0adc58471ad Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Apr 2026 16:18:10 +0000 Subject: [PATCH 2/4] docs(knowledge): AGI-as-SoA invariant + 10 architecture rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extends lab-vs-canonical-surface.md with: - AGI IS the struct-of-arrays section (Era 8 CognitiveRecord → 4-axis BindSpace columns: topic/angle/thinking/planner) - Per-cycle contract: dispatch/branch/merge/dream lifecycle - Migration status note (4-column SoA distills 11-field Era 8) - 10 non-negotiable architecture invariants citing docs/: 1. BindSpace read-only + CollapseGate airgap protocol 2. ndarray::simd::* canonical import rule 3. Layer temporal scopes (L0 sub-ns → L4 ms) 4. Temperature hierarchy (Hot/Warm/Cold/Frozen) 5. Thinking = AdjacencyStore (one engine, two graphs) 6. Weights are seeds (GGUF → hydrate → BindSpace, no matmul) 7. Per-cycle cascade budget (~2.3ms/1M, monotone narrowing) 8. 4096 address surface (16 prefix × 256 slots) 9. Three DTO families (Stream/Resonance/Bus) — doctrinal 10. HEEL/HIP/BRANCH/TWIG/LEAF progressive precision hierarchy Grounded in: INTEGRATION_PLAN_CS.md, COGNITIVE_SHADER_HYDRATION.md, SEMIRING_ALGEBRA_SURFACE.md, METADATA_SCHEMA_INVENTORY.md, THINKING_PIPELINE.md, integrated-architecture-map.md, ZECKF64_VSA_SYNERGY_MAP.md, HISTORICAL_CONTEXT.md. https://claude.ai/code/session_01SbYsmmbPf9YQuYbHZN52Zh --- .claude/knowledge/lab-vs-canonical-surface.md | 229 ++++++++++++++++++ 1 file changed, 229 insertions(+) diff --git a/.claude/knowledge/lab-vs-canonical-surface.md b/.claude/knowledge/lab-vs-canonical-surface.md index a2c4bf49..65ae7bfc 100644 --- a/.claude/knowledge/lab-vs-canonical-surface.md +++ b/.claude/knowledge/lab-vs-canonical-surface.md @@ -83,6 +83,88 @@ Every layer has its canonical surface in a zero-dep or feature-free module. The lab transport (REST/gRPC) is **orthogonal to the stack** — it's a debug probe, not a layer. +## AGI IS the struct-of-arrays (per `docs/HISTORICAL_CONTEXT.md` Era 8) + +**The structural identity** — commit this to muscle memory: + +``` +struct-of-arrays ⇄ cognitive-shader-driver ⇄ UnifiedStep + OrchestrationBridge + (BindSpace) (per-cycle dispatch) (cross-cycle composition) + +AGI = (topic, angle, thinking, planner) + = struct-of-arrays consuming cognitive-shader-driver +``` + +AGI is **not a new crate**, not a new `struct Agi`, not a new API. AGI +is the semantic interpretation of the already-shipped `BindSpace` +columns, driven by the already-shipped `ShaderDriver`, composed across +cycles by the already-shipped `OrchestrationBridge`. + +### The four AGI axes = four SoA columns + +Era 8's `CognitiveRecord` (HISTORICAL_CONTEXT.md lines 176-198) lists +11 fields in 4 semantic groups. The four AGI-perspective axes are the +consumer-visible distillation: + +| AGI axis | BindSpace column (shipped) | Era 8 `CognitiveRecord` field(s) | 5D-stream dimension | +|---|---|---|---| +| **topic** — "what about" | `fingerprints: FingerprintColumns` | `topic: Fingerprint<256>` (+ `fingerprint` for content identity) | D2: Context | +| **angle** — "from whose view" | `qualia: QualiaColumn` (18×f32) | `angle: Fingerprint<256>` + `qualia: [f32; 18]` | D3: Perspective | +| **thinking** — dispatch + modulation | `meta: MetaColumn` (packed u32 MetaWord) | `shader_mask: u8` (active layers) + style bits | dispatch gate | +| **planner** — "why/how", causal composition | `edges: EdgeColumn` (CausalEdge64) | `edge: CausalEdge64` + `rung: u8` | D4: Causality | + +D1 (Content = raw Fingerprint) and D5 (Time = temporal index) are +carried inside the `meta` column's packed bits and the per-row +position index respectively — so the 5D stream still walks through +the same 4 SoA columns. + +### The per-cycle contract (what ShaderDriver actually does) + +`ShaderDriver::dispatch(ShaderDispatch)` reads the row-window from +`BindSpace`, processes the 5D stream, and emits `ShaderHit` + `MetaWord` +through the sink: + +``` +per cycle: CognitiveShader processes 5D stream over a row window +per branch: independent causal trajectory (CausalEdge64 on edges column) +per merge: NARS evidence accumulation via CausalEdge64 revision +per dream: offline consolidation (prune / merge / permute-XOR-bind) +``` + +### Migration status (per `docs/BINDSPACE_MIGRATION_GAP.md`) + +The shipped BindSpace is a 4-column SoA that **compresses** the Era 8 +11-field CognitiveRecord into the 4 AGI axes. Era 8's distinct +`topic` + `angle` + `fingerprint` (three Fingerprint<256> columns) are +currently folded into a single `fingerprints` column + the 18D qualia +projection. This is a deliberate distillation, not a loss: + +- `fingerprints` carries row identity (content = topic-fingerprint). +- `qualia` carries the perspective (angle = 18D phenomenal coords). +- The full Era-8 schema lands if/when the migration in + `BINDSPACE_MIGRATION_GAP.md` ships (Container/CogRecord canonicalisation + in `lance-graph-contract`). That extension **adds columns to the same + SoA**; it does not add a new crate or a new dispatch API. + +### Consequences for future sessions + +- **Never** write a parallel `struct Agi { topic, angle, thinking, planner }`. + Those ARE the BindSpace columns. Wrapping them in a new struct + breaks the SoA layout and the SIMD sweep. +- **Never** introduce an `agi` or `cognitive_record` crate that re- + exports the axes. `cognitive-shader-driver` already IS the unified + API; extend it by adding columns, not by wrapping it. +- **Extend by column, not by layer.** New AGI capability = new + BindSpace column (with its SoA allocation + meta-prefilter + sink + emission) consumed by the existing `ShaderDriver`. Not a new trait, + not a new endpoint, not a new DTO family. +- **Era 8's "weights are seeds" doctrine**: weights don't compress + into parameters; they seed the holographic BindSpace. If a future + session proposes "train an AGI model," the correct framing is "hydrate + new BindSpace rows from seed weights and let ShaderDriver dispatch + cycles against them." No gradient-descent training loop lives on + the canonical surface. + ## The Failure Mode This Doc Prevents Future sessions reading this crate see `/v1/shader/dispatch`, @@ -188,6 +270,153 @@ the lab umbrella. Neither is a subset of the other. --- +## Architecture Invariants (from `docs/` — non-negotiable) + +The AGI-as-SoA identity above does NOT stand alone. It composes with +these load-bearing invariants. Violating any one of them breaks the +contract. + +### 1. BindSpace is read-only; writes go through CollapseGate (airgap) + +Per `docs/INTEGRATION_PLAN_CS.md`: + +- Columns are `Arc<[u64; 256 * N]>` — shared, read-only, mmap-friendly. +- Writers hold **owned `Copy` microcopies** (CausalEdge64, Band, + TruthValue, ThinkingStyle — all ≤16 bytes, stack-only). +- Deltas cross the airgap via `GateDecision` + `MergeMode`: + - `Xor` — single-writer commit (self-inverse, rollback is free). + - `Bundle` — multi-writer majority vote. + - Superposition — ambiguity preserved for next cycle. +- **No locks. No `&mut` during compute.** XOR is commutative + + associative → ordering irrelevant. + +If proposing "mutate column in place" → stop. Use +`gate.submit(delta)` → `gate.commit()` → next-generation column. + +### 2. Canonical import surface: `ndarray::simd::*` only + +Per `docs/INTEGRATION_PLAN_CS.md`: + +```rust +// Correct (stable public surface): +use ndarray::simd::{F32x16, U8x64, F16x32, Fingerprint, MultiLaneColumn, array_window}; + +// Wrong (reaches into private internals): +use ndarray::hpc::fingerprint::Fingerprint; +``` + +If a type isn't in `ndarray::simd::*`, it's internal. lance-graph +consumers must not use it. + +### 3. Layer temporal scopes (budget discipline) + +| Layer | Scope | Budget | +|---|---|---| +| L4 Planner | per query | milliseconds | +| L3 CollapseGate | per commit cycle | microseconds | +| L2 CognitiveShader | per step | nanoseconds | +| L1 BindSpace | per lane read | nanoseconds (zero-copy) | +| L0 ndarray SIMD | per instruction | sub-nanosecond | + +Pushing L2/L1 work up to L3 or L4 breaks the budget. L0/L1 kernels +never allocate. + +### 4. Temperature hierarchy — cold path numbs thinking + +Per `docs/SEMIRING_ALGEBRA_SURFACE.md` §3: + +| Path | Semiring | Purpose | +|---|---|---| +| **Hot** — BindSpace sweep | XorBundle / HammingMin / Resonance | full 16Kbit HDR, SIMD popcount | +| **Warm** — CAM-PQ cascade | CamPqAdc | 6-byte codes, 500M candidates/s | +| **Cold** — DataFusion joins | Boolean | Arrow RecordBatch, standard SQL | +| **Frozen** — metadata skeleton | none | Pure CRUD, no semiring | + +**Rule:** cold joins narrow on scalars FIRST; semiring algebra fires +only on narrow survivors. Never run a full HDR semiring over all rows. + +### 5. Thinking is an `AdjacencyStore`, not a separate system + +Per `docs/THINKING_PIPELINE.md`: + +- 36 styles = nodes in a real `AdjacencyStore` (CSR/CSC + NARS edges). +- Cognitive verbs call the **same** `batch_adjacent()` as data queries. +- **One engine, two graphs.** Styles live at τ-prefix `0x0D` in the + `Addr(u16)` space (per `METADATA_SCHEMA_INVENTORY.md` §1E). +- "Add a thinking engine outside the planner" is wrong. Extend the + topology, extend the cognitive verbs — same substrate. + +### 6. Weights are seeds — not runtime parameters + +Per `docs/COGNITIVE_SHADER_HYDRATION.md`: + +GGUF weights are hydrated once into BindSpace as holographic memory: +1. 256 archetypes (palette) + Fingerprint<256> per archetype. +2. 256×256 FisherZTable (64 KB, L1-resident). +3. Holographic residual (phase + magnitude slots). +4. CausalEdge64 wiring (S(row) × P(role) × O(col)). + +After bake: inference = Hamming cascade + palette lookup + XOR +compose. No `matmul`, no FP inner loop. "Training" = hydrating new +BindSpace rows from seed weights, not gradient descent. + +### 7. Per-cycle cascade ∼2.3 ms / 1M rows (monotone narrowing) + +Per `docs/COGNITIVE_SHADER_HYDRATION.md`: + +``` +sweep topic[] → 50K survivors (~2 ms Hamming) +sweep angle[] → 5K survivors (~0.2 ms Hamming) +sweep causality[] → 500 survivors (~0.05 ms CausalEdge64 filter) +sweep qualia[] → 50 survivors (scalar 18D range) +exact on 50 → palette lookup → CausalEdge64 output +``` + +Intersect is bitmap AND across per-column hit masks. Monotone +narrowing must hold (each stage shrinks the set by ≥10×). Breaking +the order (e.g. qualia before Hamming) loses the 99% reject gate. + +### 8. 4096 address surface = 16 prefix × 256 slots + +Per `docs/METADATA_SCHEMA_INVENTORY.md`: + +- `Addr(u16)` = 8-bit prefix : 8-bit slot. +- Prefix 0x0D = thinking style templates (τ addresses). +- Each slot holds one `[u64; 256]` = 16,384-bit fingerprint. +- This IS the BindSpace substrate before column extension. + +### 9. Three DTO families above SoA (doctrinal, not yet shipped) + +Per `docs/integrated-architecture-map.md` §5: + +| DTO | Holds | Stage | +|---|---|---| +| `StreamDto` | arrival-order inputs (text, SPO seeds, anchor hints) | pre-parse | +| `ResonanceDto` | superposition field (topic/angle/hypothesis, pre-collapse) | active sweep | +| `BusDto` | explicit thought (topic anchor + angle mask + edges + style ordinal) | post-collapse → CognitiveShader | + +**Field ≠ sweep ≠ bus.** The field is searchable terrain (10K [i8/i16]); +the sweep is the cheap XOR/bundle collapse into a lookup vector; the +bus is explicit execution through p64 / CognitiveShader. Do not +conflate them. + +### 10. HEEL/HIP/BRANCH/TWIG/LEAF = progressive precision hierarchy + +Per `docs/integrated-architecture-map.md` §3: + +| Level | Role | Maps to | +|---|---|---| +| HEEL | coarse basin routing | bgz17, Base17 | +| HIP | family sharpening | 16384 palette, i16 | +| BRANCH | contradiction / signed split | polarity residuals | +| TWIG | local prototype neighborhood | CLAM ripple | +| LEAF | exact member | full fingerprint | + +Each level narrows; never skip levels. bgz17 IS HEEL — do not +ask it to also be LEAF identity. + +--- + **If you are a future session and this doc is loaded, you are on the correct path. If you are about to add a REST handler without having read this section, stop and re-read the Decision Procedure above.** From 59adfb973b9a47762df5b78e30fec0dcca33de19 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Apr 2026 16:23:14 +0000 Subject: [PATCH 3/4] fix(knowledge): deduplicate architecture invariants section Remove duplicate "Architecture Invariants (from docs/)" that was positioned after the doc's closing sentinel. The canonical version (I1-I10) sits inside the doc body at its correct structural position. https://claude.ai/code/session_01SbYsmmbPf9YQuYbHZN52Zh --- .claude/knowledge/lab-vs-canonical-surface.md | 311 +++++++++--------- 1 file changed, 159 insertions(+), 152 deletions(-) diff --git a/.claude/knowledge/lab-vs-canonical-surface.md b/.claude/knowledge/lab-vs-canonical-surface.md index 65ae7bfc..e004f34a 100644 --- a/.claude/knowledge/lab-vs-canonical-surface.md +++ b/.claude/knowledge/lab-vs-canonical-surface.md @@ -165,6 +165,165 @@ projection. This is a deliberate distillation, not a loss: cycles against them." No gradient-descent training loop lives on the canonical surface. +## Architecture Invariants (cross-cutting; cite these, do not drift) + +The AGI-as-SoA identity above doesn't stand alone. A small set of +load-bearing invariants from the wider architecture docs compose +with it. Treat these as non-negotiable; cite the ref when extending. + +### I1. BindSpace is read-only; writes cross the `CollapseGate` airgap + +Ref: `docs/INTEGRATION_PLAN_CS.md`. + +- BindSpace columns are `Arc<[u64; 256 * N]>` — shared, read-only, + mmap-friendly. Shader kernels never hold `&mut` into a column. +- Writers hold owned `Copy` microcopies (`CausalEdge64`, `Band`, + `TruthValue`, `ThinkingStyle` — all ≤ 16 B, stack-only). +- Deltas cross the airgap through `contract::collapse_gate::{GateDecision, + MergeMode}`. `MergeMode::Xor` = single-writer (XOR self-inverse); + `MergeMode::Bundle` = multi-writer majority; superposition = + preserve ambiguity for next cycle. +- No locks, no `&mut` during compute. Ordering doesn't matter (XOR + commutative + associative); rollback is free. + +If a session proposes "let the shader mutate a column in place" or +"let the REST handler write directly," stop — handlers construct a +`UnifiedStep`, the bridge yields a delta, the gate commits. + +### I2. Canonical SIMD import surface: `ndarray::simd::*` + +Ref: `docs/INTEGRATION_PLAN_CS.md` §"Architecture separation". + +```rust +// Correct — stable public surface: +use ndarray::simd::{F32x16, U8x64, F16x32, Fingerprint, MultiLaneColumn, array_window}; + +// Wrong — reaches into private implementation: +use ndarray::hpc::fingerprint::Fingerprint; +use ndarray::hpc::simd_avx512::F32x16; +``` + +Anything not re-exported from `ndarray::simd::*` is internal. The +`hpc::*` path is free to refactor; consumers never reach it. + +### I3. Layer temporal budgets + +Ref: `docs/INTEGRATION_PLAN_CS.md` §"5-Layer Stack". + +| Layer | Scope | Budget | +|---|---|---| +| L4 Planner | per query | milliseconds | +| L3 CollapseGate | per commit cycle | microseconds | +| L2 CognitiveShader | per step | nanoseconds | +| L1 BindSpace | per lane read | nanoseconds, zero-copy | +| L0 ndarray SIMD | per instruction | sub-nanosecond | + +L0/L1 kernels never allocate. "Push fingerprint similarity into the +planner" violates the budget and is rejected. + +### I4. Temperature hierarchy — cold narrows, then algebra fires + +Ref: `docs/SEMIRING_ALGEBRA_SURFACE.md` §3 "Cold Path Numbing Effect". + +| Path | Substrate | Semiring | Role | +|---|---|---|---| +| Hot | BindSpace HDR sweep | XorBundle / HammingMin / Resonance | full 16 Kbit, SIMD popcount | +| Warm | CAM-PQ cascade | CamPqAdc | 6-byte codes, 500 M candidates/s | +| Cold | DataFusion columnar joins | Boolean | scalar columns only — no fingerprint data | +| Frozen | `metadata.rs` | none | pure CRUD skeleton | + +Cold joins narrow the candidate set first on scalar columns; HDR +semirings fire only on the narrow survivors. A full HDR semiring over +all rows is what the cold/warm pre-filters exist to prevent. + +### I5. Thinking IS an `AdjacencyStore` — one engine, two graphs + +Ref: `docs/THINKING_PIPELINE.md` §"Layer 3: ThinkingGraph". + +- 36 thinking styles are nodes in a real `AdjacencyStore` (CSR/CSC + + NARS-weighted edges). Styles live at τ-prefix `0x0D` in the + `Addr(u16)` space (`docs/METADATA_SCHEMA_INVENTORY.md` §1E). +- Cognitive verbs (`EXPLORE` / `FANOUT` / `HYPOTHESIS` / …) call the + same `batch_adjacent()` / `adjacent_truth_propagate()` / + `adjacent_incoming()` the data queries use. +- **One engine, two graphs** — thinking is another client of the + planner's adjacency substrate, not a meta-layer above it. + +"Add a thinking engine that lives outside the planner" is wrong; +extend the topology, the verbs, or the NARS edge set. + +### I6. Weights are seeds — hydrate-then-cascade, not matmul + +Ref: `docs/COGNITIVE_SHADER_HYDRATION.md`, Era 8. + +Each GGUF weight matrix hydrates (build-time) into: +1. 256 archetypes (bgz17 palette) + `Fingerprint<256>` per archetype. +2. 256×256 FisherZTable (64 KB, L1-resident). +3. Holographic residual (phase + magnitude slots). +4. `CausalEdge64` wiring (`S(row) × P(role) × O(col)`). + +After bake, inference = Hamming cascade + palette lookup + XOR +compose. No matmul, no FP inner loop. "Training a model" on the +canonical surface means hydrating new BindSpace rows; no +gradient-descent loop lives on the canonical surface. + +### I7. Per-cycle cascade budget (monotone narrowing) + +Ref: `docs/COGNITIVE_SHADER_HYDRATION.md` §"Why struct-of-arrays". + +``` +sweep topic[] → 50 000 survivors (~2 ms Hamming) +sweep angle[] → 5 000 survivors (~0.2 ms Hamming) +sweep causality[] → 500 survivors (~0.05 ms CausalEdge64 filter) +sweep qualia[] → 50 survivors (scalar, 18-D range) +exact on 50 → palette lookup → CausalEdge64 output +``` + +Intersect = bitmap AND over per-column hit masks. A proposed dispatch +order that breaks monotone narrowing (scoring qualia before Hamming, +for instance) loses the 99 % reject gate — reject the proposal. + +### I8. 4096 address surface = 16 prefix × 256 slots + +Ref: `docs/METADATA_SCHEMA_INVENTORY.md` §1A. + +- `Addr(u16)` = 8-bit prefix : 8-bit slot. +- Prefix `0x0D` = thinking style templates (τ addresses). +- Each slot holds one `[u64; 256]` = 16 384-bit fingerprint. +- This IS the BindSpace substrate before column extension. + +### I9. Three DTO families above SoA (doctrinal — not yet shipped) + +Ref: `docs/integrated-architecture-map.md` §5. + +| DTO | Holds | Stage | +|---|---|---| +| `StreamDto` | arrival-order inputs (text, SPO seeds, anchors) | pre-parse | +| `ResonanceDto` | superposition field (topic / angle / hypothesis) | active sweep | +| `BusDto` | explicit thought (anchor + mask + edges + style) | post-collapse → CognitiveShader | + +**Field ≠ sweep ≠ bus.** The field is searchable terrain (10 K +`[i8/i16]`); the sweep is the cheap XOR/bundle collapse into a +lookup vector; the bus is explicit execution through p64 / +CognitiveShader. Do not conflate them. + +### I10. HEEL / HIP / BRANCH / TWIG / LEAF — progressive precision + +Ref: `docs/integrated-architecture-map.md` §3. + +| Level | Role | Maps to | +|---|---|---| +| HEEL | coarse basin routing | bgz17, Base17 | +| HIP | family sharpening | 16 384 palette, i16 | +| BRANCH | contradiction / signed split | polarity residuals | +| TWIG | local prototype neighborhood | CLAM ripple | +| LEAF | exact member | full fingerprint | + +Each level narrows; never skip levels. bgz17 IS HEEL — do not ask +it to also be LEAF identity. + +--- + ## The Failure Mode This Doc Prevents Future sessions reading this crate see `/v1/shader/dispatch`, @@ -268,155 +427,3 @@ the lab umbrella. Neither is a subset of the other. - `.claude/knowledge/cam-pq-unified-pipeline.md` — the pipeline doc that produced this invariant. - `.claude/knowledge/cognitive-shader-architecture.md` — the broader shader-as-driver architecture. ---- - -## Architecture Invariants (from `docs/` — non-negotiable) - -The AGI-as-SoA identity above does NOT stand alone. It composes with -these load-bearing invariants. Violating any one of them breaks the -contract. - -### 1. BindSpace is read-only; writes go through CollapseGate (airgap) - -Per `docs/INTEGRATION_PLAN_CS.md`: - -- Columns are `Arc<[u64; 256 * N]>` — shared, read-only, mmap-friendly. -- Writers hold **owned `Copy` microcopies** (CausalEdge64, Band, - TruthValue, ThinkingStyle — all ≤16 bytes, stack-only). -- Deltas cross the airgap via `GateDecision` + `MergeMode`: - - `Xor` — single-writer commit (self-inverse, rollback is free). - - `Bundle` — multi-writer majority vote. - - Superposition — ambiguity preserved for next cycle. -- **No locks. No `&mut` during compute.** XOR is commutative + - associative → ordering irrelevant. - -If proposing "mutate column in place" → stop. Use -`gate.submit(delta)` → `gate.commit()` → next-generation column. - -### 2. Canonical import surface: `ndarray::simd::*` only - -Per `docs/INTEGRATION_PLAN_CS.md`: - -```rust -// Correct (stable public surface): -use ndarray::simd::{F32x16, U8x64, F16x32, Fingerprint, MultiLaneColumn, array_window}; - -// Wrong (reaches into private internals): -use ndarray::hpc::fingerprint::Fingerprint; -``` - -If a type isn't in `ndarray::simd::*`, it's internal. lance-graph -consumers must not use it. - -### 3. Layer temporal scopes (budget discipline) - -| Layer | Scope | Budget | -|---|---|---| -| L4 Planner | per query | milliseconds | -| L3 CollapseGate | per commit cycle | microseconds | -| L2 CognitiveShader | per step | nanoseconds | -| L1 BindSpace | per lane read | nanoseconds (zero-copy) | -| L0 ndarray SIMD | per instruction | sub-nanosecond | - -Pushing L2/L1 work up to L3 or L4 breaks the budget. L0/L1 kernels -never allocate. - -### 4. Temperature hierarchy — cold path numbs thinking - -Per `docs/SEMIRING_ALGEBRA_SURFACE.md` §3: - -| Path | Semiring | Purpose | -|---|---|---| -| **Hot** — BindSpace sweep | XorBundle / HammingMin / Resonance | full 16Kbit HDR, SIMD popcount | -| **Warm** — CAM-PQ cascade | CamPqAdc | 6-byte codes, 500M candidates/s | -| **Cold** — DataFusion joins | Boolean | Arrow RecordBatch, standard SQL | -| **Frozen** — metadata skeleton | none | Pure CRUD, no semiring | - -**Rule:** cold joins narrow on scalars FIRST; semiring algebra fires -only on narrow survivors. Never run a full HDR semiring over all rows. - -### 5. Thinking is an `AdjacencyStore`, not a separate system - -Per `docs/THINKING_PIPELINE.md`: - -- 36 styles = nodes in a real `AdjacencyStore` (CSR/CSC + NARS edges). -- Cognitive verbs call the **same** `batch_adjacent()` as data queries. -- **One engine, two graphs.** Styles live at τ-prefix `0x0D` in the - `Addr(u16)` space (per `METADATA_SCHEMA_INVENTORY.md` §1E). -- "Add a thinking engine outside the planner" is wrong. Extend the - topology, extend the cognitive verbs — same substrate. - -### 6. Weights are seeds — not runtime parameters - -Per `docs/COGNITIVE_SHADER_HYDRATION.md`: - -GGUF weights are hydrated once into BindSpace as holographic memory: -1. 256 archetypes (palette) + Fingerprint<256> per archetype. -2. 256×256 FisherZTable (64 KB, L1-resident). -3. Holographic residual (phase + magnitude slots). -4. CausalEdge64 wiring (S(row) × P(role) × O(col)). - -After bake: inference = Hamming cascade + palette lookup + XOR -compose. No `matmul`, no FP inner loop. "Training" = hydrating new -BindSpace rows from seed weights, not gradient descent. - -### 7. Per-cycle cascade ∼2.3 ms / 1M rows (monotone narrowing) - -Per `docs/COGNITIVE_SHADER_HYDRATION.md`: - -``` -sweep topic[] → 50K survivors (~2 ms Hamming) -sweep angle[] → 5K survivors (~0.2 ms Hamming) -sweep causality[] → 500 survivors (~0.05 ms CausalEdge64 filter) -sweep qualia[] → 50 survivors (scalar 18D range) -exact on 50 → palette lookup → CausalEdge64 output -``` - -Intersect is bitmap AND across per-column hit masks. Monotone -narrowing must hold (each stage shrinks the set by ≥10×). Breaking -the order (e.g. qualia before Hamming) loses the 99% reject gate. - -### 8. 4096 address surface = 16 prefix × 256 slots - -Per `docs/METADATA_SCHEMA_INVENTORY.md`: - -- `Addr(u16)` = 8-bit prefix : 8-bit slot. -- Prefix 0x0D = thinking style templates (τ addresses). -- Each slot holds one `[u64; 256]` = 16,384-bit fingerprint. -- This IS the BindSpace substrate before column extension. - -### 9. Three DTO families above SoA (doctrinal, not yet shipped) - -Per `docs/integrated-architecture-map.md` §5: - -| DTO | Holds | Stage | -|---|---|---| -| `StreamDto` | arrival-order inputs (text, SPO seeds, anchor hints) | pre-parse | -| `ResonanceDto` | superposition field (topic/angle/hypothesis, pre-collapse) | active sweep | -| `BusDto` | explicit thought (topic anchor + angle mask + edges + style ordinal) | post-collapse → CognitiveShader | - -**Field ≠ sweep ≠ bus.** The field is searchable terrain (10K [i8/i16]); -the sweep is the cheap XOR/bundle collapse into a lookup vector; the -bus is explicit execution through p64 / CognitiveShader. Do not -conflate them. - -### 10. HEEL/HIP/BRANCH/TWIG/LEAF = progressive precision hierarchy - -Per `docs/integrated-architecture-map.md` §3: - -| Level | Role | Maps to | -|---|---|---| -| HEEL | coarse basin routing | bgz17, Base17 | -| HIP | family sharpening | 16384 palette, i16 | -| BRANCH | contradiction / signed split | polarity residuals | -| TWIG | local prototype neighborhood | CLAM ripple | -| LEAF | exact member | full fingerprint | - -Each level narrows; never skip levels. bgz17 IS HEEL — do not -ask it to also be LEAF identity. - ---- - -**If you are a future session and this doc is loaded, you are on the -correct path. If you are about to add a REST handler without having -read this section, stop and re-read the Decision Procedure above.** From 6faaf20030a734ea5baa3ec2f10ce107d2934c30 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Apr 2026 16:43:06 +0000 Subject: [PATCH 4/4] docs(knowledge): lab surface positive purpose (iteration velocity) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The LAB-ONLY surface isn't just quarantine scaffolding — it's the codec-research iteration testbed. Its reason for existing is the cost of the alternative: every codec candidate re-measured through a cargo build cycle burns minutes per iteration. With the lab REST/gRPC + wire DTOs, a single binary serves dozens of candidates against the same safetensors in seconds per call. PR #220 falsified PR #219's ICC-0.9998 claim via exactly this path: the calibration CLI + /v1/shader/calibrate endpoint surfaced mean ICC 0.195 / 0/234 pass rate on full Qwen3-TTS-0.6B tensors before any production consumer linked the codec. Two purposes now named explicitly in the doc: 1. Iteration velocity (positive) — lab surface = curl-friendly research loop, no rebuild per candidate. 2. Canonical firewall (guard) — consumers still walk UnifiedStep via OrchestrationBridge; they never see Wire* per-op DTOs. Changes: - New subsection "Why the Lab Surface Exists (positive purpose — not just quarantine)" with the #219 → #220 worked-example table. - Decision Procedure item 3 reframed: research ops and curl-friendly debug shortcuts are a legitimate use of the lab surface, with a graduation rule (full-size validation → new StepDomain variant; lab endpoint stays for continued iteration, production moves to bridge). https://claude.ai/code/session_01SbYsmmbPf9YQuYbHZN52Zh --- .claude/knowledge/lab-vs-canonical-surface.md | 79 +++++++++++++++++-- 1 file changed, 73 insertions(+), 6 deletions(-) diff --git a/.claude/knowledge/lab-vs-canonical-surface.md b/.claude/knowledge/lab-vs-canonical-surface.md index e004f34a..bf7fcba0 100644 --- a/.claude/knowledge/lab-vs-canonical-surface.md +++ b/.claude/knowledge/lab-vs-canonical-surface.md @@ -56,6 +56,66 @@ the canonical library surface: | `cognitive-shader-driver/src/codec_bridge.rs` | `CodecResearchBridge: OrchestrationBridge` for StepDomain::Ndarray | The research consumer's bridge impl — canonical trait, lab-only consumer | | `cognitive-shader-driver/src/planner_bridge.rs` | Per-op WirePlan shortcut → PlannerAwareness | `PlannerAwareness` already impls OrchestrationBridge directly; this is a lab test adapter only | +### Why the Lab Surface Exists (positive purpose — not just quarantine) + +The lab API is not tech debt tolerated until it can be deleted. It is +the **codec-research iteration testbed**. Its reason for existing is +the cost of the alternative: every codec candidate re-measured through +a `cargo build` cycle burns minutes per iteration. Over dozens of +candidates (Fractal, Zipper, 5^5, 7^7, I8-μ-law, CamPqRaw, CamPqPhase, +residual PQ, Hadamard pre-rotation, OPQ…) that compounds into hours of +waiting between measurements — which is what kills research velocity. + +With the lab surface: + +``` +HTTP POST /v1/shader/calibrate { tensor_path, codec, params } + ↓ (no recompile) +codec_research::calibrate_tensor(...) + ↓ +ICC / reconstruction / SHA256 in manifest.json +``` + +Same binary serves dozens of candidates against the same safetensors, +in seconds per call. This is how PR #220 falsified PR #219's ICC-0.9998 +claim: the calibration CLI + REST endpoint exercised CAM-PQ on full +Qwen3-TTS-0.6B tensors and surfaced mean ICC 0.195 / 0/234 pass rate +before any production consumer linked the codec. + +**Two purposes held together — neither dominates the other:** + +1. **Iteration velocity** (positive). The lab surface lets research + measure N codec candidates against real tensors without + recompiling. `/v1/shader/{dispatch,calibrate,probe,tensors,plan}` + are the curl-friendly entry points for that loop. +2. **Canonical firewall** (guard). The lab surface MUST NOT become + what production consumes. Consumers walk `UnifiedStep` via + `OrchestrationBridge`; they never see `WireCalibrate` / + `WireProbe` / `WireTensors`. + +**Worked example — the #219 → #220 falsification arc:** + +| PR | Role | Velocity enabled by | +|---|---|---| +| #219 | Claimed ICC 0.9998 at 6 B/row on 128 training rows; lab-gated CAM-PQ candidates behind `--features lab` | Research iteration inside the lab surface; no production wiring yet | +| #220 | D5 full-size validation via `cam_pq_calibrate` CLI on real safetensors; mean ICC **0.195**, 0/234 tensors ≥ 0.99; EPIPHANIES marks prior entry SUPERSEDED | Same lab surface exercised on full tensors — negative result surfaced **before** the codec linked into any canonical consumer | + +The gate did its job twice: (a) the iteration loop was fast enough to +falsify the claim in a day, (b) nothing production-facing depended on +the falsified codec when the result came back. That is exactly what +the LAB / canonical split exists to buy. + +**Rule of thumb for a new codec / research op:** + +- Add it under `codec_research.rs` + a `Wire*` DTO + a handler under + `/v1/shader/`. Measure against real tensors via curl / CLI. +- Land findings as `.claude/knowledge/codec-findings-*.md` with + measured ICC / error / SHA256 + EPIPHANIES entry. +- Only after full-size validation passes the gate (e.g. ≥ 0.99 ICC + on held-out rows for argmax-regime tensors) does the op graduate + — and even then it graduates as a new `StepDomain` variant behind + `OrchestrationBridge`, not as a permanent `/v1/` endpoint. + The canonical surface — all of it re-exported from `cognitive-shader-driver` so consumers depend on one crate: @@ -372,12 +432,19 @@ Before adding ANY new endpoint, handler, or Wire DTO, answer in order: `lance-graph-planner/src/orchestration_impl.rs`). **Do not add a new `/v1/` endpoint.** -3. **Is this purely a lab test convenience?** If and only if the - answer is unambiguously yes (e.g., "I want a curl-friendly - per-op shortcut for debugging"), add it under `src/wire.rs` + - `src/serve.rs` or `src/grpc.rs` and document that it routes - through the canonical bridge internally. Mark it `LAB-ONLY` in - the module doc. +3. **Is this codec-research iteration or lab debugging?** This is a + legitimate positive use of the lab surface — it's *why* the lab + surface exists. Research ops (calibrate / probe / tensors / + bench) and curl-friendly debugging shortcuts belong under + `src/wire.rs` + `src/serve.rs` or `src/grpc.rs`. Mark the module + `LAB-ONLY` in its doc and ensure the handler either (a) stays + inside `codec_research.rs` for research, or (b) constructs a + `UnifiedStep` and delegates to the canonical bridge. When a + research op graduates (passes full-size validation — e.g. ICC + ≥ 0.99 on held-out rows), it migrates to a new or existing + `StepDomain` variant behind `OrchestrationBridge`; the lab + endpoint stays for continued iteration but is no longer the + path production consumes. 4. **Never**: - Publish a per-op endpoint as "the API" in PR descriptions or