Unified API: codec research + runbook + planner-via-DTO + OrchestrationBridge

[AdaWorldAPI]

Summary

Follow-up to merged PR #220. Lands the unified-API wiring the review kept correcting toward:

1. Codec research on the canonical endpoint — /v1/shader/{tensors,calibrate,probe} REST + gRPC, same Wire DTOs, no parallel server on the planner.
2. WireRunbook — scheduled test injection: POST one DTO, server executes a labeled sequence of steps (Tensors/Calibrate/Probe/Dispatch/Ingest/Plan).
3. Planner-via-DTO — /v1/shader/plan behind --features with-planner, Plan step in the runbook.
4. impl OrchestrationBridge for PlannerAwareness — the canonical dedup trait finally has an implementation; consumers can route lg.* step-types without coupling to planner internals.
5. cam-pq-unified-pipeline.md — knowledge doc mapping the existing lance-graph/src/cam_pq/* surface (storage/udf/ivf/jitson_kernel) and 5 integration gaps found during this PR.

Architectural corrections captured

• cognitive-shader-driver IS the unified REST + gRPC API — not lance-graph-planner.
• cam_pq/storage.rs already is the D3 Lance schema; cam_pq_calibrate (from D1+D2+D5: CAM-PQ calibration pipeline — honest negative result #220) writes a parallel raw format — migration queued.
• register_cam_udfs exists but isn't called from datafusion_planner — 1-line fix queued.
• OrchestrationBridge was documented in the contract since before this session but had no impl — now filled for StepDomain::LanceGraph.
• planner_bridge.rs (introduced in this PR) duplicates OrchestrationBridge. Retirement queued.

New endpoints (cognitive-shader-driver)

Route	Feature	Purpose
POST /v1/shader/tensors	serve	List tensors with CamPq/Passthrough/Skip routes
POST /v1/shader/calibrate	serve	Train CAM-PQ, measure ICC + reconstruction
POST /v1/shader/probe	serve	ICC vs row-count degradation curve
POST /v1/shader/runbook	serve	Scheduled sequence of labeled steps
POST /v1/shader/plan	serve + with-planner	Plan via PlannerAwareness
rpc Tensors/Calibrate/Probe	grpc	gRPC mirror of REST research ops

DTO parameters (num_subspaces, num_centroids, kmeans_iterations, max_rows, icc_samples) drive codec research without recompilation.

New trait impl (lance-graph-planner)

impl OrchestrationBridge for PlannerAwareness {
    fn route(&self, step: &mut UnifiedStep) -> Result<(), OrchestrationError>;
    fn resolve_thinking(&self, style, inference_type) -> ThinkingContext;
    fn domain_available(&self, domain: StepDomain) -> bool;
}

Step-type routing: lg.plan_auto / lg.orchestrate / lg.health. Other domains return DomainUnavailable — consumers combine bridges via Vec<Box<dyn OrchestrationBridge>>.

Tests

Suite	Count	Status
lance-graph-contract	133	✅ all pass
lance-graph-planner lib (incl. 5 new orchestration_impl)	167	✅ all pass
cognitive-shader-driver lib with serve grpc with-planner	46	✅ all pass

New wire tests: wire_plan_request_defaults, wire_plan_request_full_mode, wire_runbook_accepts_plan_step, wire_runbook_parses_mixed_steps.

Queued follow-ups (per cam-pq-unified-pipeline.md)

1. Wire register_cam_udfs in datafusion_planner/mod.rs
2. Migrate cam_pq_calibrate output to Lance schema via build_codebook_batch
3. impl OrchestrationBridge for codec research (nd.* step-types)
4. Retire planner_bridge.rs → shader-driver holds Box<dyn OrchestrationBridge>
5. Collapse per-op Wire DTOs → UnifiedStep { step_type, args } + BridgeSlot

~1-2 days total, each step additive and independently verifiable.

Test plan

• cargo test -p lance-graph-contract — 133/133
• cargo test -p lance-graph-planner --lib — 167/167 (was 162, +5 new)
• cargo test --manifest-path crates/cognitive-shader-driver/Cargo.toml --features "serve grpc with-planner" --lib — 46/46

https://claude.ai/code/session_01SbYsmmbPf9YQuYbHZN52Zh