feat(draft): archon-draft crate + FCDP gate pipeline (sandbox-validated)#51
Open
ProfessorVR wants to merge 61 commits into
Open
feat(draft): archon-draft crate + FCDP gate pipeline (sandbox-validated)#51ProfessorVR wants to merge 61 commits into
ProfessorVR wants to merge 61 commits into
Conversation
…le train) Port ste-bah#2 — structured PDF ingestion wired into run_pdf_ingest_pipeline: - archon-ingest-ext: Marker JSON -> Block parser; token-aware, bbox-carrying chunker (chars/4 code-point parity, pairwise undersized-merge); is_real_table gate + HTML->grid; Bekker/page-number locator strip+capture. - archon-docs: additive doc_chunk_spatial / doc_chunk_hashes / doc_locators satellites; block_chunking (ChunkOut->ChunkArtifact, id format preserved, bbox capture); provenance_chunks (commit_hash -> chunks_root -> extract_text_spatial record on ALL ingest, fills provenance_record_id; verify_chunks_root tamper check); marker_source (device-agnostic subprocess/http/pre-extracted); reprocess clear-path covers new satellites. - archon-policy: PdfPolicy.chunker (default token_aware) + marker_sidecar/device. - scripts: device-agnostic archon_marker_sidecar.py (auto cuda->mps->cpu, with --selftest) + chunk_parity_check.py. token_aware is the default (page_anchor kept as fallback); integrity runs on all ingest; Marker is used when a sidecar is configured, else a flat-text fallback. Intentional, documented deviations from the Python reference: corrected page_end (chunk spans only pages it actually contains); Bekker regex broadened to catch 4-digit Aristotle numbers (e.g. 1147a). Reviewed via a 5-lens adversarial pass (7 findings fixed: byte->code-point tokens, chained->pairwise merge, reprocess satellite cleanup, restored Princeton/Bollingen title markers, Marker-path artifact hash, zero-chunk guard, source-file provenance hash). Tests: archon-ingest-ext 29, archon-docs 189, archon-policy 5; cargo build --bin archon clean. SITREP: plans/ARCHON-PORT2-COMPLETE-2026-06-25.md (in the dissertation repo). Also includes port #1 from the prior session: `archon style train` first-class subcommand (new archon-lanham crate + cli_args/command/dispatch wiring). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… embeddings
B2 — Marker as a local, device-agnostic sidecar (standalone, no WRAITH):
- PdfPolicy.marker_python runs the sidecar via an isolated venv interpreter
(system python may be too new for torch/marker). marker 1.10.2 JSON matches the
Rust parser directly (bbox + /page/N/). Verified on Mac: ocr_engine=marker,
coord_space=marker real bboxes in doc_chunk_spatial.
I2 — cross-modal CLIP image embeddings (text<->image), local/standalone:
- embed_fastembed: CLIP ViT-B/32 vision (embed_image via embed_bytes) + text
(embed_image_query) encoders (fastembed 4.9.1, shared 512-dim), lazy-loaded;
image_dimension().
- ensure_vec_schema(dim, image_dim): vec_page_images is sized to the IMAGE
dimension (512 CLIP), independent of the text dim (fixes a hardcode that broke
the multimodal test); all call sites updated.
- ingest: image OCR is NON-FATAL — a failed OCR (e.g. text-less game frames) no
longer aborts the document; the image still gets CLIP-embedded.
- pdf_image_enrichment: PDF embedded figures are CLIP-embedded too (per-figure
key {page_id}-imgN) so they're visually searchable alongside standalone images.
- retrieval_image::search_images + `archon docs search-images <text>`: text->image
cross-modal search over vec_page_images (CLIP-text query); resolve_page strips
the figure suffix.
- ocr/rapid: ARCHON_RAPIDOCR_PYTHON auto-detects the ~/.archon-marker-venv helper
venv (which bundles rapidocr) so figure OCR works standalone, no env needed.
Verified live on the Mac: standalone frame -> "Image embeddings: 1" (CLIP vision
model downloaded); cross-modal search ranks frames by text query. Tests 189/29/5
green single-threaded (the parallel suite has pre-existing global-provider
test-isolation flakiness, unrelated to this change — both flaky tests pass isolated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…leak, OCR status, search guard) Found by an adversarial 5-lens wiring/bug audit of 594ecb5 (each finding independently verified against the code; the HIGH one empirically reproduced). - schema: vec_page_images dim-mismatch MIGRATION (HIGH). A DB created by a pre-CLIP build sized vec_page_images to the TEXT dim (768); `:create` is a no-op when the relation exists, so 512-dim CLIP image vectors were rejected forever (silent — only a warning, image embeddings never persisted). ensure_vec_page_images now reads the existing dim via `::columns` and, on a mismatch, drops the HNSW index then the relation and recreates it at the image dim. Regression test added (reproduces the old failure, asserts the fix). - reprocess: clear vec_page_images (MEDIUM). remove_generated_rows never deleted image/figure embeddings → orphaned/stale vectors that `search-images` still returned after reprocess. Now `:rm`'s all "page-{document_id}-" keyed rows (page-level + per-figure {page_id}-imgN). - ingest: keep the OCR-run status Failed on the non-fatal image path (LOW). The failed run was being relabeled Completed by the shared completion update; now gated by image_ocr_failed so doc_ocr_runs provenance stays accurate (the document still proceeds to CLIP embedding). - retrieval_image: guard search_images for a missing vec_page_images relation (LOW). `search-images` on a fresh/never-indexed DB now returns empty (friendly message) instead of a hard "relation not found" error, mirroring the text path. Tests: archon-docs 190 green single-threaded (+1 migration regression test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t planner New leaf crate `archon-accel` (Port ste-bah#2 device layer). RUNTIME (not compile-feature- gated) detection of CUDA / Apple-Metal-unified / CPU plus *free* memory, and a pure, deterministic placement planner that fits the Marker sidecar to free VRAM. - detect.rs: degrade-safe probe (nvidia-smi free VRAM, WSL-aware; Apple unified via sysinfo; CPU fallback). Never panics; always yields a valid report. - report.rs: AcceleratorReport + best_gpu() (selects by FREE, not card size). - placement.rs: plan_placement/plan_marker_ingest. Load-bearing rule `free < footprint -> CPU`; GPU placements ALWAYS carry per-doc OOM->CPU; force_marker_device + memory_budget_mb overrides; surya batch-cap env mapping; dormant multi-consumer seam (whisper/frame-VLM) for the video round. - 13 tests incl. the verified co-tenancy case (32607 MiB total / 139 MiB free -> CPU); examples/probe.rs prints the live report + plan for the host. Additive: nothing consumes it yet (PR-C wires it into ingest). Gates green: build, clippy -D warnings, fmt, file-size (max 379 <= 500). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…ity mode
Lock the 6 intentional Rust-vs-Python ingestion divergences (structured TableGrid vs
Python flatten; chunks_root/commit_hash Rust-only; cleaning_version none vs clean_v1;
anchored running-head vs unanchored inline locators; page_end last-page-contained;
broadened Bekker regex) + the 4 must-match chunk constants as EXPECTED, not drift.
Every anchor source-verified.
Correction: the Bekker-broadening gain is vs layout_analyzer.py:54
`_BEKKER_RE=^\s*\d{2,3}[a-b]?\d{0,2}\s*$` (caps at 3 digits, misses 1147a), NOT
markdown_chunker.py:303 (\d{2,4}, already reaches 4) — register documents both.
scripts/chunk_parity_check.py: add an additive `--marker-json <path>` mode to print the
Python reference chunk table from a real Marker JSON dump (for PR-D corpus diffing). The
no-arg invocation is byte-identical to baseline (sha256 match, exit 0).
.gitignore: re-include /docs/ingestion/ (matches the existing committed-subtree pattern).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
… OOM→CPU retry Wire archon-accel's free-VRAM placement planner into the live PDF ingest path. The Marker sidecar's device + surya batch caps are now resolved from the host's FREE VRAM (not a static string): marker_device None/"auto" -> planner-chosen, explicit cuda|mps|cpu forces it. The load-bearing correctness guarantee is the per-doc GPU-OOM -> CPU retry. - archon-docs/marker_source.rs: from_policy() resolves placement via archon_accel::detect() + plan_marker_ingest(); MarkerSource::Subprocess now carries the resolved env (TORCH_DEVICE, surya batch caps, PYTORCH_CUDA_ALLOC_CONF); run_sidecar() classifies a torch-OOM (sidecar exit 42 or stderr signature) and retries the document on CPU. + overrides_from_policy + 2 host-agnostic tests. (archon-docs lib: 192 tests green.) - archon-accel: marker_cpu_fallback_env() (CPU env for the OOM retry) + re-export. - archon-policy: PdfPolicy.marker_memory_budget_mb override (clamps usable free VRAM). - sidecar archon_marker_sidecar.py: --device "auto" now triggers real auto-detect (was forwarded verbatim → an invalid torch device); torch-OOM is caught and exits 42 so the Rust caller can retry on CPU. ingest_pdf.rs unchanged (from_policy is the chokepoint). Build + tests green; the touched code is clippy-clean (pre-existing >7-arg lints in untouched files are CI-advisory). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
scripts/bbox_jitter_diff.py — run the SAME PDF's Marker JSON on multiple devices (5090 / 5070 / Mac-MPS / CPU), align blocks by id, and report the per-coordinate delta distribution plus the per-bucket quantization unify-rate. This is the measurement that decides whether rounding bbox coords before hashing can make spatial_hash device-independent (so re-ingestion is portable) or whether geometry stays verify-by-recompute-only. Stdlib only; handles bbox or polygon. Smoke-tested on the selftest fixture (identical dumps → 0px delta, 100% unify). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
The lanham (port #1) and ingestion-port (port ste-bah#2) feature crates, plus this session's device-adaptive additions, were committed without running `cargo fmt`, leaving the branch dirty under `cargo fmt --all -- --check` — the CI fmt gate (dtolnay/rust-toolchain@stable, which runs on PRs to main). Normalize the whole branch to canonical rustfmt (verified: local rustfmt 1.9.0 / rustc 1.96.0 is the same stable line CI installs, and the existing `style: cargo fmt` commits stay clean under it). Pure formatting; no semantic change. 19 files: archon-docs (9), archon-ingest-ext (4), archon-lanham (3), src/command (3). Verified post-fmt: `cargo fmt --all -- --check` exit 0; `cargo build --bin archon` clean; archon-ingest-ext 29 (parity golden gate) + archon-accel 13 + archon-policy 5 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…(PR-D) crates/archon-ingest-ext/examples/chunk_dump.rs — reads a Marker JSON dump, runs parse_marker_str + chunk_blocks_default, and prints the Rust chunk table (idx, page_start, page_end, text_len, text_head). Mirrors scripts/chunk_parity_check.py --marker-json so the Rust port diffs directly against the Python reference on real corpus PDFs. No native deps; builds on WSL + Mac. Validated on a real Mac (marker-pdf 1.10.2, MPS) Marker JSON of a 13pp article: Rust and Python produce 10 identical chunks (every text_len byte-exact); the sole diff is the documented page_end correction (divergence ste-bah#5). Confirms the port is faithful on real hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…4 MiB) Real GPU memory measured on the RTX 5070 Laptop (torch 2.11.0+cu128, surya-ocr 0.17.1, 13pp doc): torch peak reserved ~5956 MiB (peak allocated ~5194, + ~0.5 GiB CUDA context). The prior 5120 estimate was nearly exact on *allocated* but undercounted the reserved pool + context. Bump ModelFootprintTable.marker_mb 5120 -> 6144 so the fit decision reflects real VRAM: on an idle 8 GB card (7822 free) Marker still fits Generous (need 6144+1536=7680, and it ran there), while a busier 8 GB card now correctly drops to Reduced/CPU instead of risking OOM. Adjusted 3 test free-memory thresholds to the new footprint. surya peak also scales with the configured batch caps (measured at surya defaults); MPS peak still pending (Mac was asleep). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…both platforms PR-D Mac measurement: torch MPS driver-allocated ~6089 MiB for the same 13pp doc — essentially matches the RTX 5070 CUDA reserved peak (~5956 MiB). Marker's footprint is ~6 GiB on both CUDA and MPS, so the 6144 marker_mb tuned in c36edef covers either platform; no logic change. Doc comment updated (removes the "MPS pending" caveat). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Placement adapted DOWN (fit smaller cards / CPU) but "Generous" was a fixed batch-cap ceiling — a 16 GB or 32 GB card ran the same surya batches as a barely-fitting 8 GB card, leaving VRAM and throughput on the table. Make it adapt UP too, safely: - SuryaTier gains Ample + Max; decide_one picks the tier by FREE VRAM bands (>=need -> Generous, >=12 GiB -> Ample, >=20 GiB -> Max), so a bigger card runs bigger batches. - marker_env_for(device, tier) centralizes the device+tier -> env mapping. Generous stays the measured ~6 GiB config; Ample/Max are PROVISIONAL upscales. - marker_env_ladder() builds the per-doc OOM->retry-smaller sequence: start at the chosen tier, step DOWN GPU tiers on torch-OOM, CPU only as the last resort. marker_source::fetch_json now iterates this ladder (advancing only on OOM), replacing the single OOM->CPU jump — so an over-eager upscale on a hard doc drops to a smaller batch tier instead of all the way to CPU. archon-accel 17 tests (4 new: tier-scaling, batch monotonicity, ladder step-down, cpu-only); archon-docs 192 lib tests green; clippy -D warnings + fmt clean. The Ample/Max caps + AMPLE/MAX VRAM thresholds are provisional — calibrate with mem_probe.py; the ladder makes over-estimates safe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…alibration) PR-D calibration on the 5090 (marker-pdf 1.10.2 / surya 0.17.1) disproved the premise of the batch-size tiers: Marker's VRAM is INERT to batch size (batch 1..256 identical) and instead tracks DOCUMENT SIZE — surya's OCR encoder processes the whole document's line-crops at once. Three points (13pp->5956, 129pp->9424, 578pp->8470 MiB reserved) show a bounded, SATURATING footprint, not page-linear-unbounded (578pp used less than 129pp; page resolution drives the top). - Revert 3b0e490's Ample/Max upward tiers + retry-smaller-batch ladder (they can't work: bigger batch != more VRAM; smaller batch != OOM relief). SuryaTier -> {Gpu,Cpu,None}; GPU leaves batch to surya defaults, CPU caps it to bound RAM. marker_env_ladder = [GPU, CPU]. - Add marker_footprint_mb(pages) = min(6000 + 30*pages, 10240): fits all 3 points, conservative cap, OOM->CPU backstop. Thread the already-computed extract_result.page_count through from_policy -> marker_env_ladder -> the placement footprint. decide_one: GPU iff free >= need. - headroom 1536 -> 512 (footprint now includes context + is conservative). - scripts/mem_probe.py: the VRAM-peak calibration tool that produced the model. Net: placement now depends on BOTH free VRAM AND document size. A small doc runs on an 8 GB card's GPU; a 300pp doc routes to CPU there (no wasted GPU-OOM cycle) and to GPU on a 16 GB+ card. archon-accel 14 tests, archon-docs 192 lib tests, clippy -D warnings + fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Batch size is VRAM-inert (PR-D), so the only lever to fit a document larger than a card's free
VRAM onto its GPU is to process it in page-range slices. Verified on the 5090: marker-pdf's
--page-range restricts the run AND keeps ABSOLUTE page ids (chunk 0-6 -> pages 0..6, chunk 7-12
-> pages 7..12), so slice block-streams concatenate in page order with no re-offset.
archon-accel:
- marker_chunk_pages(usable_mb): inverse of the footprint model (shared FLOOR/PER_PAGE/CAP consts)
-> the largest page-range whose footprint fits free VRAM.
- MarkerChunk { page_range, attempts } + marker_ingest_plan(report, overrides, pages): a whole-doc
chunk when it fits (or forced / no-GPU / CPU), else contiguous GPU chunks sized to free VRAM;
each chunk keeps its own [GPU, CPU] OOM ladder. +7 host-agnostic tests (8GB splits 300pp into 7,
16GB runs whole, Apple splits on mps, tiny-free / forced-cpu stay whole-doc CPU).
archon-docs (marker_source):
- Subprocess now carries Vec<MarkerChunk>; blocks_for runs each chunk (--page-range when set)
through run_chunk's ladder and concatenates the parsed blocks; from_policy uses marker_ingest_plan.
sidecar:
- --page-range START-END (0-indexed inclusive) -> PdfConverter page_range config.
archon-accel 20 tests, archon-docs 192 tests, clippy(touched files)+fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
`cargo run -p archon-docs --example chunk_ingest -- <pdf> <sidecar> <python> <budget_mb> <pages>` forces a small VRAM budget so marker_ingest_plan splits the doc into page-range GPU chunks, runs the real MarkerSource::blocks_for over them, and checks the concatenated blocks span every page. Hardware-gate tool for the small-card / Apple-MPS chunking path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…shareable template Ingestion runs Marker (device-adaptive placement + page-range chunking + real bboxes) only when [policy.docs.pdf].marker_sidecar is set; otherwise it silently falls back to pdftotext text-only chunks. This turns it on: - .archon/policy.toml (now gitignored): active marker_sidecar + marker_python for THIS machine — real absolute paths, kept out of git so they never upload. - .archon/policy.example.toml (tracked): the shareable GitHub template — same config with the Marker keys as commented placeholders + instructions, header marked "EXAMPLE TEMPLATE". Others: cp it to policy.toml and set their paths. - .gitignore: ignore .archon/policy.toml (machine-specific), track the template. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…ml enables Marker `cargo run -p archon-policy --example effective_policy` loads the effective policy for the cwd (system -> user -> workspace .archon/policy.toml) and prints the resolved PDF/Marker settings, so you can confirm device-adaptive Marker ingestion is the DEFAULT on a given machine (config file honored). Verified on WSL (CUDA) + Mac (MPS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Adds docs/ingestion/hardware-vram-guide.md — device-adaptive placement, MEASURED footprints (Marker 6-10 GB page-scaled; qwen2.5vl:7b 14-22 GB by context), a what-runs-on-what-VRAM table, the sequential-pipeline + VLM-residency notes, constrained-host tuning (num_ctx / keep_alive / CPU offload), and per-tier configs — including the locked 8 GB laptop policy (Marker on GPU + VLM on CPU). policy.example.toml: recommend LOCAL qwen2.5vl (cross-version; llama3.2-vision's mllama arch is dropped in current Ollama) with a pointer to the guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…_gpu
Adds three optional knobs to [policy.docs.vlm.ollama] so constrained hosts can keep
the VLM from starving Marker of VRAM:
- num_ctx (u32) — smaller context window = much less KV-cache VRAM (default 32K
is wasteful for image description)
- keep_alive (str) — "0" unloads the model immediately after a doc's images, freeing
VRAM for the next doc's Marker; "5m" is Ollama's default
- num_gpu (u32) — 0 forces CPU (Marker owns the GPU on <=8 GB hosts)
Wired through models.rs (OllamaVlmPolicy + Default) -> loader_docs.rs (Raw + merge) ->
vlm/ollama.rs (provider fields, from_policy, request body). num_ctx/num_gpu nest under
"options"; keep_alive is top-level. All three are Option and OMITTED from the request
body when unset, so behavior is byte-identical to today (the pre-existing exact-match
wiremock test still passes; two new tests cover the set/omit paths). 0 is meaningful
(force CPU) and distinct from unset — hence Option, not a u32 sentinel.
Also: repoint repository_policy_template_parses_all_vlm_provider_fields at the committed
.archon/policy.example.toml (the real policy.toml is now gitignored) + document the knobs.
archon-docs vlm::ollama 8 tests, archon-policy 15 tests green; build + fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…ntegrity) Image-OCR (pdf-image-ocr-*) and VLM-description chunks were persisted AFTER the document's chunks_root Merkle root was sealed, so image/figure-derived text was excluded from the tamper-evidence root (and silently from verify_chunks_root). Now they are re-folded in: enrich_pdf_images returns the image ChunkArtifacts and the pipeline re-runs persist_chunk_integrity over the text+image UNION after enrichment. - enrich_pdf_images / persist_image_result / persist_ocr_result / persist_vlm_result thread a &mut Vec<ChunkArtifact> accumulator; persist_image_ocr_chunks and persist_vlm_description now return their chunks. - ingest_pdf keeps the early text-only seal (so a fatal VLM error still leaves a valid text root), captures it in TextSeal, then re-seals over the union — but ONLY when the doc had a text seal AND produced image chunks. chunks_root sorts commits, so a no-image ingest is byte-identical to before; the second seal is an idempotent upsert of the same record. - Test: seal text-only, re-fold over text+image, assert the root changes, verifies over the superset, and tampering an image-OCR chunk now flips verify (was silently excluded before). Follow-up (noted in code): image-only/scanned PDFs (empty full_text) still get no root — needs a synthetic artifact/record. archon-docs 195 lib tests, clippy (touched files) + fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Self-scanned book PDFs store one full-page raster per page + an OCR text layer. The text layer triggers Marker (which OCRs the pages WITH bboxes = the real content), but the image-enrichment path then treated each full-page scan as a "figure" — re-OCRing it (duplicate chunks, now folded into chunks_root) and VLM-ing it (useless "a page of text"). Skip enrichment for these. Detection (is_scanned_page_images): >=70% of pages have exactly one LARGE (min side >=1000px) embedded image. Distribution is the signal, not raw count: born-digital docs cluster figures (King&Salvo = 17 imgs across 8/17 pages, ~24%) while a scanned Uexkull has one per page (100%). enrich_pdf_images returns early (page metadata still marked) when detected; non-scanned docs (born-digital figures) enrich exactly as before. Follow-up: figure-region VLM (describe Marker-detected figure bboxes) for figures baked INTO scanned pages — a separate feature. +4 unit tests; archon-docs 199 lib tests, clippy(touched files)+fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…ement
Enabling figure VLM needs BOTH [policy.workers] vlm = "allow-local" (the coarse master
gate) AND [policy.docs.vlm] enabled=true — a "deny" on the worker gate silently overrides
the detailed config ("denied by policy.workers.vlm" in ingest logs). Surfaced by the King
pipeline test. Also note `tesseract-ocr` is needed for image/figure OCR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Refine the full-page-scan detector: an embedded image counts as a page-scan only if it is large AND page-shaped (long/short side ratio in ~1.2-1.6 — the Letter/A4/book range, orientation-agnostic). The prior size-only check false-positived on a large square diagram or wide chart appearing on most pages, wrongly flagging the doc as scanned and skipping real figures. New is_page_scale() + a per-page "exactly one page-scan and nothing else" rule feed the same 70% doc-level ratio. (ste-bah#3 is already satisfied by the doc-level binary: a born-digital doc enriches ALL its images — including any full-page infographic — because the skip only fires for scanned_book docs.) +3 unit tests (large square / wide chart per page are NOT scans; page-shaped sizes ARE). Follow-up remains true page-dimension coverage % via the page MediaBox. clippy+fmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
The 1.6 upper bound (calibrated on Letter 1.29 / A4 1.41) was too tight for taller book formats: Uexkull's real scans measure ~1.58 with crops up to ~1.61, so a chunk of its pages slipped the gate and would have been wrongly enriched. Widen to [1.2, 1.7] (covers Letter, A4, US Legal 1.65, and book formats; a >1.7 portrait image is a tall figure, not a page). Verified against the real Uexkull image dims + a 20-page book-scan detection test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Before a single-PDF ingest, classify how the image-enrichment path will treat the doc (SCANNED BOOK -> skip N page-scans, or BORN-DIGITAL -> enrich M figures) and print a LOUD report with a warning, so a misclassification (like the Uexkull aspect-gate edge) is caught BEFORE any OCR/VLM. When interactive (stdin is a TTY) and not --yes, prompt to proceed; non-interactive/batch/-y auto-proceeds (the report is still logged). - pdf::classify_pdf_enrichment(path): lightweight -- pdfimages -list (dims) + pdfinfo (pages), no byte extraction, no Marker; reuses the pipeline's own is_scanned_page_images detector so the report matches what actually happens. - `docs ingest` gains -y/--yes; the confirm follows the existing auth.rs prompt pattern. Verified: King -> BORN-DIGITAL (17 figures), Uexkull -> SCANNED BOOK (281 scans skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
… vs aspect) Add a coverage-based scanned-book detector — image drawn-size / page MediaBox — run A/B against the shipped pixel-dims aspect heuristic. A `scan_detector` policy knob (default "aspect") selects the active detector; both always run for the pre-ingest report and a loud divergence log, so coverage can be validated on the corpus before any default flip. - pdf_scan.rs (new): ScanDetector, CoverageVerdict, classify_by_coverage (per-page coverage summed and capped at 1.0; unusable ppi / missing page dims defer to the aspect page-scale test and flag low-confidence), classify_scan A/B wrapper, and page_dimensions/get_media_box via lopdf (inheritance walk + depth-10 cycle guard + array guard + Integer-or-Real numeric parsing). - pdf.rs: keep x-ppi/y-ppi/size from `pdfimages -list`; classify_pdf_enrichment returns both detector verdicts. - enrich_pdf_images gains scanned_override so coverage mode drives the skip decision; the aspect default path stays byte-identical. - archon-policy: PdfPolicy.scan_detector + loader wiring (the field was dropped by the Raw-struct layering) with known-value validation. - docs.rs: pre-ingest banner shows both verdicts, peak coverage, and a divergence warning. - scripts/coverage_oracle.py: pypdfium2 dry-run oracle validating the ppi-proxy against true placement rects. lopdf 0.36 added. archon-docs coverage/lopdf tests + archon-policy loader tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Adversarial review found the coverage/aspect pre-ingest report classified on the raw `pdfimages -list`, while the ingest pipeline filters and gates embedded images — so the report could promise "SCANNED BOOK, enrichment skipped" while the pipeline enriched a different (filtered) image set. Apply the pipeline's gate inside the classifier: honor extract_embedded_images and filter by min_image_dimension + min_image_bytes (via the size-column proxy for the extracted-PNG size — a conservative approximation that only ever drops small images, never real page-scans, so it cannot wrongly flip a scanned book to born-digital). No object dedup in the report path: `pdfimages` already lists a shared XObject once per page, which is the per-page granularity the coverage sum needs. Coverage mode now drives the pipeline from classify_scan's selected verdict — including its aspect fallback when page dims are unreadable — so the report and the pipeline never disagree in coverage mode. The aspect default path is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
Add a "union" scan_detector: a page is a scan if the aspect page-scale test OR the coverage >= 0.80 test flags it. The corpus dry-run (161 PDFs) showed neither base detector alone is complete — aspect misses low-DPI scans (below its 1000px pixel floor; 29 corpus docs) and coverage misses margin-cropped scans (text-block-only images that fill ~0.73 of the page; Konstan). Their union classifies every divergent corpus doc correctly: Konstan -> SCANNED (via aspect), the 29 low-DPI books -> SCANNED (via coverage), and every agreeing doc unchanged. - classify_by_union + UnionVerdict; factor coverage_per_page out of classify_by_coverage so both share the per-page coverage map, and add aspect_scan_page_set mirroring is_scanned_page_images' per-page rule. - ScanDetector::Union + parse/validation; resolve() selects it. - ingest_pdf: any non-aspect detector resolves the verdict from the path (coverage + union); aspect stays on the in-memory default, unchanged. - policy.example.toml documents "union" as the recommended setting for the corpus flip. Default remains "aspect" (opt-in). Validated end-to-end: Konstan (438pp, coverage peak 67%) now classifies SCANNED BOOK under union. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
The corpus dry-run (161 PDFs) validated union as strictly better than either base detector — it catches the 29 low-DPI scans aspect misses AND the margin-cropped scans coverage misses — so promote it from opt-in to the shipped default. - PdfPolicy::default().scan_detector = "union"; the loader keeps the default on an unknown value. - ScanDetector::parse falls back to Union (not Aspect), so a typo degrades to the best detector rather than a weaker one. - policy.example.toml ships scan_detector = "union". Every PDF ingest now resolves the scanned-book verdict from the file (pdfimages + lopdf page dims); when page dims are unreadable it falls back to the shipped aspect heuristic, so the decision degrades gracefully. "aspect" / "coverage" remain selectable for A/B. 228 archon-docs lib + archon-policy tests green; validated end-to-end (no config → union active). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
…(PR-E/C3) An image-only PDF (page scans, no text layer) was classified SCANNED and its enrichment skipped — but with no text layer there is nothing for Marker to OCR, so under the union default the doc got NO content at all (a regression of the old "chunks but no root" gap). Even when image-OCR did run, the chunks had no integrity root. - Gate the scanned-book skip on whether a text seal was produced: scanned + text layer -> skip (Marker/text owns the pages; unchanged); scanned + NO text layer -> OCR the page scans (the only content) with VLM forced off (a full-page scan is a page reproduction, not a discrete figure, and VLM over 100+ pages is wasteful). - Widen the content-extraction gate so Marker runs on image-only pages when configured — its surya OCR reads scanned pages into real bbox text + a normal chunks_root. - Synthetic chunks_root: image-only docs with no text seal now seal their image-OCR chunks under a synthetic OCR artifact, for the same tamper-evidence a text doc gets. - The pre-ingest banner distinguishes "SCANNED BOOK (text layer present) -> skipped" from "IMAGE-ONLY SCAN (no text layer) -> WILL be OCR'd" via a lightweight pdftotext probe. Validated: Frede (9pp image-only) now yields 55 OCR chunks + a root (was 0). New integration test (+ a pdfinfo mock hook in the test guard). 229 archon-docs lib tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PyE618vmwGS6YnBkVEoEqg
`docs verify-integrity [--doc <id>] [--json]` recomputes each document's
chunks_root (Merkle-style root over the per-chunk commit hashes) and compares
it to the sealed extract_text_spatial provenance record — surfacing the V-1
tamper-evidence that previously had no CLI. Per doc it reports INTACT / MISMATCH
/ NO-RECORD plus the covered chunk count and the record id; --json emits
{all_pass, documents[]} for scripting. A doc may carry more than one ocr_text
artifact (text + image union under C3), so it passes if ANY sealed root matches
the current chunk set.
scripts/spotcheck_corpus.sh is a self-driving post-flip verification harness: it
runs verify-integrity over all docs, samples per-doc inspect (chunks/OCR/images/
provenance), probes hybrid retrieval, and auto-derives a real verbatim phrase
from a search hit to exercise verify-quote (locate + fuzzy + absent), confirming
marker bboxes / coord_space on born-digital docs.
Validated on the live King + Uexküll docs: both chunks_root INTACT, quote-verify
returns a marker bbox on an exact-ish locate and FUZZY 93% on a perturbed phrase.
verify_chunks_root's mismatch branch is already covered by unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8sSHgWg4nHyfwYgXt1RRV
…+ integrity guards
Wire a persistent Marker server (surya models loaded once) through the existing
MarkerSource::Http transport, add VRAM-adaptive VLM-enrichment concurrency, and
harden the ingest integrity path. Built to re-ingest the dissertation corpus
with real bboxes without paying the per-document surya-model-reload cost.
Persistent Marker (no per-doc reload):
- scripts/archon_marker_server.py (NEW): FastAPI/uvicorn server, loads surya once
at startup, serves POST /convert with output byte-identical to the subprocess
sidecar; GET /health {status,models_loaded}; CUDA-OOM -> empty_cache -> CPU
retry ladder; convert_lock serializes model access.
- scripts/archon_marker_core.py (NEW): shared resolve_device + run_marker core
imported by BOTH the sidecar and the server, so the two transports cannot drift.
- archon_marker_sidecar.py: refactored to import the core (behavior-preserving;
--selftest stays torch-free).
- policy.docs.pdf.marker_url selects the Http transport (overrides marker_sidecar);
when unset, the subprocess path is byte-for-byte unchanged.
Adaptive parallel VLM enrichment:
- auto_image_workers(): derive worker count from FREE VRAM (reserve model weights
+ headroom, serial floor of 1, unified-memory cap of 2).
- --jobs <auto|N> flag; interactive prompt when auto; explicit N rejected if out
of 1..=16 rather than silently clamped. DB writes stay serial (single writer);
only the stateless OCR+VLM work fans out.
Integrity guards (so GPU starvation degrades VISIBLY, not silently):
- Http transport: a Marker error / empty result / timeout is a HARD failure
(document Failed -> sources_failed), never a silent bbox-less COORD_NONE that
still counts as "Ingested".
- /health preflight before an ingest run when marker_url is set (bounded retry
for the model-load window; hard-error if never ready).
- 900s per-convert request timeout; end-of-run COORD_MARKER vs COORD_NONE tally.
Verified: cargo build clean, 251 tests pass; two adversarial-review passes;
byte-identical subprocess/server parity; corpus re-ingest of 126 PDFs -> 124 docs
(123/124 with real marker bboxes; content-corruption audit 124/124 clean;
quote-verify EXACT with a marker bbox confirmed end-to-end).
Known follow-ups (see task notes): the VLM VRAM reserve is a hardcoded constant
that under-estimates ollama's actual allocation (~22.8GB observed vs 6.5 assumed)
and the 8GB-laptop/Mac paths are only unit-tested, not run on real hardware;
the ingestion path still lacks a hang-watchdog, and there is a tokio
runtime-drop-on-exit hang on index/model-status/reprocess.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D8sSHgWg4nHyfwYgXt1RRV
The Apple detector reported the Metal accelerator's free memory as the instantaneous free-RAM snapshot minus an OS reserve (`ram_free - 6144`). On a 24GB Mac with ~11GB free at launch that is ~5112 MB — below the planner's 6000 MB surya floor (`marker_chunk_pages` returns None) — so Marker was demoted to CPU despite 24GB of unified memory. The instantaneous snapshot is the wrong budget model for unified memory, where the GPU grows into the whole shared pool. `apple_unified_gpu_budget_mb` now takes max(instantaneous-free-minus-OS, total-pool-minus-OS-minus-coresident-VLM). The VLM term (6800 MB = measured ollama qwen2.5vl:7b Metal footprint) keeps surya and the resident VLM from fighting over the shared pool. On the 24GB Mac the budget rises 5112 -> 11632 MB, above the floor, so Marker places on MPS; 16GB/8GB machines still resolve below the floor and correctly stay on CPU. The budget is only a fit-GATE, not an allocation — surya still allocates its real ~6GB. Validated on the target 24GB Mac (native arm64): Marker launched --device mps, surya completed, 1 COORD_MARKER / 0 COORD_NONE, verify-integrity intact, no OOM. Adversarially reviewed (clean, no blockers). Non-macos/CUDA detect path is untouched. 4 new unit tests cover the 24/16/8GB + plenty-free cases. Known follow-up (see doc comment): the free-independent `unified` term can over-report on a large machine under heavy non-VLM pressure, and a jetsam SIGKILL is not ladder-catchable — bound the term to a fraction of free and classify signal-kills as CPU-retryable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ls on CPU Two hardening fixes from the ultracode review of the marker-MPS budget (follow-ups tracked on task ste-bah#12). Apple unified-memory budget (crates/archon-accel/src/detect.rs): - apple_unified_gpu_budget_mb's `unified` term was free-INDEPENDENT (total - OS - VLM), so on a large machine under heavy non-VLM pressure it over-reported wildly (64GB total / 3GB free -> ~51GB "MPS-viable"). Bound it to at most 1.5x actual free RAM (APPLE_UNIFIED_OVERCOMMIT 3/2). The validated 24GB Mac case is unchanged (still 11632 -> MPS); the 64GB/3GB case now yields 4608 (< the 6000 surya floor -> CPU). Two new tests cover the bound. Marker signal-kill -> CPU retry (crates/archon-docs/src/marker_source.rs): - A marker sidecar terminated by a signal (out.status.code() == None, e.g. a jetsam/OOM-killer SIGKILL that leaves no exit-42 and no OOM stderr) was classified SidecarError::Other, so run_chunk returned Err WITHOUT trying the CPU rung. Add SidecarError::Killed; run_sidecar returns it when code() is None; run_chunk treats it like Oom and advances the GPU->CPU ladder. Two new subprocess tests (a fake sidecar that SIGKILLs on GPU, succeeds on CPU) prove the ladder advances and that a last-rung kill surfaces a clear error. cargo build clean; archon-accel 20 tests + archon-docs marker_source 10 tests pass. Adversarially reviewed (no blockers on this fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…outs Resolve the observed parallel-VLM-enrichment HANG (archon at 0 CPU jiffies during image enrichment): a wedged external call (OCR subprocess or VLM) left a JoinSet worker blocked forever, so docs.join_next().await never returned and the whole ingest stalled with no recovery (task ste-bah#10). Per-image wall-clock backstop (crates/archon-docs/src/pdf_image_enrichment.rs): - Wrap process_image (OCR + VLM for one image) in tokio::time::timeout in BOTH the serial and JoinSet paths (env ARCHON_PDF_IMAGE_TIMEOUT_SECS, default 600s -- a LOOSE backstop well above legitimate VLM time, fires only on a true hang). A timed-out image is recorded as a per-image SKIP (record_image_timeout: progress line + failure counter + warning), never fatal to the document. The JoinSet task hands its ImageWork back on timeout (JoinSet gives no task identity at join time). Defensive: break the drain loop when no work remains instead of spinning. OCR subprocess timeouts (crates/archon-docs/src/ocr/{provider,rapid,local}.rs): - Every previously un-timed `Command...output().await` now uses kill_on_drop(true) + tokio::time::timeout so a wedged binary errors out instead of hanging: RapidOCR, tesseract, and pdftotext use ocr_timeout() (per-image/text, env ARCHON_OCR_TIMEOUT_SECS default 120s); the WHOLE-document pdftoppm render uses a separate, generous pdf_render_timeout() (env ARCHON_PDF_RENDER_TIMEOUT_SECS default 1800s) -- it rasterizes every page in one invocation, so a 300-600pp scan must not be clamped by the 120s per-page bound (that regression was caught in review before commit). cargo build clean; archon-docs full suite 254 tests pass (incl. new OCR-timeout tests). Adversarially reviewed; the review's one blocker (pdftoppm mis-bounded by the per-page timeout) is fixed here via the separate render timeout. Known follow-up: the marker CONVERSION subprocess (run_sidecar) still has no timeout/kill_on_drop, so a wedged marker (MPS deadlock / surya spin) can hang a worker; run_chunk should treat an elapse like the Killed rung and advance to CPU. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…recovers The Subprocess marker path (run_sidecar) called cmd.output().await with no timeout and no kill_on_drop, so a marker that HANGS mid-conversion (MPS driver deadlock / surya spin / GPU contention — an observed Mac corpus-flip failure mode) never returned and the ingest worker blocked forever with no CPU fallback. The A2 signal-kill/OOM ladder only handles sidecars that EXIT; a hang never reaches it. The Http transport already bounds converts (HTTP_CONVERT_TIMEOUT_SECS=900) — this brings the Subprocess path to parity and completes the "fully debug the ingestion function" goal (with tasks ste-bah#10/ste-bah#12). - SidecarError::TimedOut { device }, alongside Oom/Killed. - marker_timeout(device): per-attempt wall clock. GPU marker is fast, so a long GPU run is a hang → tight default 900s (ARCHON_MARKER_GPU_TIMEOUT_SECS) catches it and the ladder falls to CPU. CPU marker of a large page-range chunk is legitimately slow → generous default 3600s (ARCHON_MARKER_CPU_TIMEOUT_SECS); it is the last rung, so an elapse there hard-fails the chunk exactly as a true hang would. - run_sidecar: cmd.kill_on_drop(true) + tokio::time::timeout(marker_timeout, output()); on elapse the child is killed (and reaped by tokio's process driver) and TimedOut is returned. Success path unchanged. - run_chunk: a TimedOut rung records `last` and advances the GPU->CPU ladder, mirroring Oom/Killed. Two real-subprocess tests (a bash sidecar that sleeps 30s on GPU, succeeds on CPU): a 1s GPU budget advances to CPU in ~1s (not 30s), and a last-rung 1s CPU timeout surfaces a "timed out … device=cpu" error. Both wrap the call in a 10s guard so removing the timeout fails the test rather than hanging it. cargo build clean; archon-docs full suite 256 tests pass (12 marker_source incl. the 2 new). Adversarially reviewed (no blockers). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…etup guide Document the corpus-ingestion + prose-style-analytics additions this fork makes over upstream archon (3 new crates: archon-accel, archon-ingest-ext, archon-lanham; archon-docs heavily extended). - docs/dissertation-port/README.md — improvements vs original archon (both subsystems), the `archon docs` / `archon style` CLIs, an at-a-glance comparison table, and an honest inventory of experimental/incomplete parts. - docs/dissertation-port/platform-agnostic-design.md — one Marker core on MPS/CUDA/CPU, archon-accel device abstraction, free-VRAM placement + page-range chunking + OOM→CPU ladder + Apple unified budget, byte-identical cross-platform chunk parity, verify-by-recompute, and the device-agnostic hang-hardening timeouts; validated-fleet table. - docs/dissertation-port/dependencies-and-setup.md — full dependency list (Rust + LIBCLANG, embedded Cozo/RocksDB/fastembed, the Marker Python venv per platform, Ollama VLM, poppler/rapidocr/tesseract), a minimal policy.toml, a step-by-step runbook, and a tunable-env-var table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… not pending work
The multi-consumer GPU arbiter the ConsumerKind::{Whisper,FrameVlm} seam would
feed is intentionally unbuilt: media-path GPU consumers run sequentially (Marker
frees VRAM before the VLM; video ASR -> frame-VLM would too), so single-consumer
free-VRAM placement already covers the real cases. Reword so the limitations
inventory reads as a design decision, not unfinished work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidates the prime directive, engineering rules, pipeline integrity, memory, and CI-gate sections into a shorter reference doc.
default_memory_data_dir/default_config_path resolve through dirs::data_dir()/dirs::config_dir(), which derive from $HOME on every platform and only consult $XDG_DATA_HOME/$XDG_CONFIG_HOME on Linux. On macOS the test's XDG overrides were a no-op, so the smoke test read the machine's real durable memory graph (accumulated counts) instead of a blank slate, failing the 0/30, 0/20, 0/3 gate assertions. Overriding $HOME isolates the CLI on both platforms. Found via a clean-clone build+test run of this branch on a Mac with real archon usage history — verified passing there and on WSL.
Trackpad/mouse-wheel scroll should work out of the box regardless of WSL detection. Users who need native terminal text selection can opt out with ARCHON_TUI_MOUSE_CAPTURE=0. Also gitignore /parity-ref/, a local portability copy of the god-agent markdown_chunker.py used by scripts/chunk_parity_check.py on machines that can't reach the WSL reference tree directly.
This reverts the mouse-capture-default change from 0477386. Keeping WSL-only default mouse capture as originally implemented; not wanted.
Adds the FCDP (Fable Console Drafting Protocol) integration, promoted from a standalone sandbox that validated every component against a live dissertation section (ALL GATES GREEN; hash-chained provenance verified): - crates/archon-draft: typed context-pack schema + G-P validator, «Qnn» quote-ID substitution (quote text never model-generated), stylometric measurement over archon-lanham, two-tier variance-derived G-A style gate (Tier-1 per-section hard bands; Tier-2 + T2-derived labels at chapter scale) - scripts/fcdp: stage orchestration (D1/D1.5/D2), judge gates G-E/G-G as fresh lean-context calls with randomized binary batteries and free-prose verdict extraction, mechanical gauntlet (G-B/C/D/F), hash-chained provenance + generated AI-use declaration, bounded R-loop drivers - docs/fcdp: locked G-A gate config (MA-applications register), band derivation artifacts, exemplar pool, README Follow-ups tracked in docs/fcdp/README.md: archon draft subcommand surface, Rust port of orchestration, archon-provenance CozoDB promotion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Satisfies the CI rustfmt gate (hard-fail check) ahead of review. No behavior change — build, tests, and gate verdicts unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All four orchestration scripts (m6_run, rcycle, slice_run, final_cycle) took the output-directory CLI arg as a relative Path while subprocess calls run with cwd=<script directory>. This was latent in the original sandbox (script and workdir were siblings under one root, so it coincidentally worked) and surfaced as soon as the scripts were promoted into scripts/fcdp/ and invoked from the repo root — every relative workdir path resolved against scripts/fcdp/ instead of the caller's cwd, causing fcdp substitute to write nowhere and panic, and every downstream JSON-reading step to crash on empty stdout. Found by a live end-to-end run on the Mac dev box. Fix: .resolve() the workdir the same way pack_path already is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Compiled bytecode cache files got committed alongside the Python orchestration scripts. Untrack them and ignore the directory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…theorists
The gendered-pronoun-bound-to-player check flagged 'Identification, on his
account, holds the player ...' as a violation, but 'his' refers to Burke (a
named theorist introduced in the prior sentence), not the player — exactly
the kind of legitimate use the terminology lock already permits (gendered
pronouns for named individuals). Found live during a Mac E2E run.
Exclude pronoun-then-comma appositive constructions ('on his account,',
'in her view,') from the match, which targets this failure mode without
weakening the true-positive case (verified: 'the player finds that his
world... he chooses it' still flags correctly; full seeded-defect
regression suite still 7/7).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d_request_body supports_output_effort was a hardcoded false stub that silently dropped the effort knob for every model; enable it for opus-4/sonnet-5/fable-5 (needed by the FCDP drafting pipeline, which relies on effort:medium to cap runaway thinking). No other caller sets request.effort today, so the blast radius is limited to opt-in requests. Also make build_request_body pub so downstream crates can assert wire-body parity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moves scripts/fcdp/*.py into the crate as byte-faithful Rust: provenance.rs (hash chain + Python-json canonicalization), gauntlet.rs (G-B/C/D/F + exemplar leak, fancy-regex for the G-D pronoun lookahead), fable.rs (model call via archon-llm AnthropicClient -> subscription OAuth + API key; effort:medium + adaptive thinking; resolve_model seam, default claude-opus-4-8), judge.rs (battery + LCG shuffle + fail-closed extraction), orchestrator.rs (full D1->D1.5->D2->gauntlet->R-loop<=3 with an injectable model-call seam + RESUME). Binary gains `archon-fcdp run`. Validated by golden differential tests against the Python reference (gauntlet, judge extraction, provenance) + orchestrator control-flow replay. Fixtures derived from unpublished dissertation content are gitignored (m6/); those tests skip when absent. Live E2E on both fable-5 and opus-4-8 passed all gates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ProfessorVR
force-pushed
the
feat/fcdp-draft
branch
from
July 8, 2026 22:46
19dd4b2 to
8bfe441
Compare
…on-draft) The full pipeline now lives in the archon-draft crate (provenance/gauntlet/fable/ judge/orchestrator + `archon-fcdp run`), validated byte-for-byte against these scripts and live on both fable-5 and opus-4-8. The Python remains in git history and the tests/fixtures/judge/judge_extract_ref.py golden harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surfaces the archon-draft orchestrator as a first-class command of the main binary: `archon draft <pack> <workdir> [--model] [--gate-config]`. Model resolves via --model then the configured Anthropic Opus then the built-in default; auth (subscription OAuth or API key) via archon-llm. The sync orchestrator runs on a spawn_blocking thread since main is async (block_on within the runtime would panic). Validated: full-workspace check green, `archon draft --help` correct, and a live end-to-end run through the subcommand reached ALL GATES GREEN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…chon draft` After a run, `archon draft` promotes the JSONL chain into the shared archon-provenance CozoDB store (best-effort — a store failure never fails a completed draft): each FCDP record maps to a ProvenanceRecord + DerivedFrom edge, using archon-provenance chain hashes so `archon prov trace|export|verify <artifact-id>` work on FCDP artifacts. The JSONL chain stays as the self-contained per-run record (drives the disclosure declaration). Cross-linking drafts to the quotes' ingestion provenance is deferred. Validated by a round-trip test: synthetic chain -> import -> trace/export/verify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…source) The draft-import step now resolves each surviving quotation back to the corpus and records it as provenance, so `archon prov verify <draft>` reaches an ingested source_document and reports Valid: true (was false: the chain rooted at the generated movement plan with no link to sources). For every «Qnn» quote that survives verbatim into the final draft, resolve it via docs::quote_verify::locate_quote (same matcher as `docs verify-quote`) to its document_id + chunk(s) + page(s), gated at similarity >= 0.85, and insert Cites edges from the assembled-draft node to both the source document and the exact chunk(s). The chunk edges ride the ingestion pipeline's existing chunk -> page -> document edges, so the trace reconstructs the full passage- level lineage (chunk + page + bbox), not just book level. Content-verified (only quotes actually present in the finished draft are linked), best-effort (a missing/cold corpus yields no edge and never fails a completed import), and deterministic edge ids so re-import upserts. Validated on the Mac against the live corpus store, Fable + Opus: quotes resolve exact (1.00) to Calleja In-Game pp.168-170 and Burke A Rhetoric of Motives pp.35-36; prov verify -> Valid: true, Reaches source: true. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqFDFXun5AjrYXuKwuuYmg
Reformat the three fmt-unclean files so a whole-tree `cargo fmt --check` CI gate passes: - crates/archon-docs/src/marker_source.rs — pre-existing, untouched by our work - src/command/docs.rs — pre-existing, untouched by our work - src/command/draft.rs — our own (86400ed added a few unwrapped lines) rustfmt-only: no logic changes. Build + draft tests green; `cargo fmt --all --check` now reports zero diffs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqFDFXun5AjrYXuKwuuYmg
Run the FCDP drafting protocol from inside the interactive TUI instead of
only the one-shot `archon draft` CLI. `/draft <pack> <workdir> [--model
<name>] [--gate-config <path>]` drafts with the session's LIVE model — the
same value `/model` sets — overridable per-call with `--model`.
Wiring (mirrors the /diff effect-slot precedent, extended for a long-
running command):
- DraftHandler (src/command/draft.rs): parses args, reads the current model
from ctx.model_snapshot, stashes CommandEffect::RunDraft.
- CommandEffect::RunDraft (registry/effect.rs): owned {pack, workdir, model,
gate_config, cwd}.
- apply_effect (context.rs): hands off to spawn_draft_command_tui.
- spawn_draft_command_tui (slash.rs): spawns the same `archon` binary's
`draft` subcommand out-of-process (current_exe) with piped stdout/stderr
and a DETACHED task that streams each line to the TUI as TextDelta, then
emits a completion/error event. Detached because a draft takes minutes and
must not block the inline apply_effect await; out-of-process so the CLI
handler's println!/eprintln! never corrupts the TUI terminal.
- build_command_context: /draft shares /model's model_snapshot gate.
- registered in registry/default.rs; command-count constants bumped 84->85.
Model comes from the session (working dir set as the subprocess cwd, so
relative paths + the provenance store resolve from the project root).
Tests: DraftHandler arg-parse/model-resolution/effect-stash + usage;
stream_child_to_tui verified live against real subprocesses (stdout, stderr,
success completion, non-zero-exit error). Full bin suite 958 green; fmt +
clippy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqFDFXun5AjrYXuKwuuYmg
…flake Adaptive thinking could occasionally consume a stage's whole max_tokens and return NO visible text (stop_reason=max_tokens) → FableError::Empty → the orchestrator aborted the entire run (the Opus flake). Same root cause could also truncate a judge verdict mid-block → phantom "no clean verdict extracted — fail-closed" defects. fable::call now wraps a single attempt (call_once) in a token-budget retry: whenever a response stops at max_tokens — Empty (no answer) OR Ok-but-truncated (answer cut off) — it re-issues the SAME call with a doubled budget (escalate_budget, capped at MAX_TOKEN_CEILING=32k) until the answer completes or the ceiling is hit. The request SHAPE is unchanged (adaptive thinking + effort:medium — the validated V0 contract); only the budget grows, giving thinking room to leave a complete answer. This pipeline's outputs are short (≈450-650-word sections, terse verdicts), so a max_tokens stop is a reliable over-thinking signal, not a legitimately long completion. A genuine empty (non-max_tokens stop, or empty at the ceiling) still surfaces. Validated live on Opus (WSL, subscription), m6 pack, 4 runs: ZERO empty-output aborts (previously ~40% aborted mid-pipeline); 2 ALL-GATES-GREEN, and the escalation demonstrably cleared truncation-caused judge fail-closed (run 1 all GG fail-closed → runs 2/3 clean). Byte-differential parity preserved (first attempt unchanged; golden tests use the injected mock, not the network). Residual: an intermittent judge fail-closed remains when the verdict is present but phrased outside the `VERDICT: YES/NO` extractor pattern (format drift, not truncation) — a distinct verdict-extraction-robustness issue, tracked separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqFDFXun5AjrYXuKwuuYmg
…eiling
Follow-up to the empty-output retry fix. The escalation is silent, so a stage
that exhausts MAX_TOKEN_CEILING (32k) and still can't finish surfaces only as a
generic defect/abort — indistinguishable from a genuine content flaw. Now
fable::call emits operator diagnostics to stderr (streamed into both the CLI
output and the TUI /draft view):
- each escalation step ("output truncated at N tokens — retrying at 2N"), and
- a prominent CEILING HIT warning when 32k is exhausted, in two flavours:
* empty → "thinking exhausted the budget with no answer … Aborting."
* truncated → "may be a BUDGET WALL, not a content flaw. Returning truncated."
So when a hard passage genuinely needs more than the ceiling, the author sees
it was a token wall (and can raise the const or split the section) rather than
chasing a phantom content defect. No behavior change — the ceiling stays fixed
(no auto-raise, no cross-run state); this only makes a hit visible.
`ceiling_hit_diagnostic(empty)` is a pure fn (wording unit-pinned); the trigger
(escalate_budget → None at the ceiling) is already tested. archon-draft suite
green; fmt clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqFDFXun5AjrYXuKwuuYmg
The G-E/G-G judges intermittently reported "no clean verdict extracted —
fail-closed" on all gates even when the answer was complete: Opus's formatting
drifts run-to-run, and the strict parser (plain "N." headers + literal
"VERDICT: YES/NO") missed valid answers phrased slightly differently → phantom
defects that bounced good drafts through needless revision cycles. Distinct
from the empty/truncation flake (that was cut-off output; this is complete but
differently-formatted output), so the token-retry couldn't touch it.
Two-sided fix, biased toward the safe side:
- Prompt (judge.rs build_prompt): add a worked example and explicit rules
(plain "N." labels, no bold/headings/bullets; end with "VERDICT: YES|NO"),
so the model reliably emits the parseable shape.
- Parser (judge.rs extract): relax the VERDICT separator (accept a dash/en/em
dash, not only a colon), broaden item headers (leading markdown bullet/
heading marker, bold around the number, ":" terminator), and add a
conservative bare-terminal-YES/NO fallback (terminal_verdict).
Fail-closed safety preserved: every change only ever RESCUES an unambiguous
verdict — a lone terminal YES/NO, a dash where a colon was expected — while a
genuinely murky answer ("…unclear.", "…so no.") still fail-closes. The golden
extraction fixtures (which include real fail-closed items) parse byte-identically.
Validated live on Opus (WSL subscription), m6, 5 runs: 0/5 phantom fail-closed
(was ~50%), 0 budget escalations; 3 ALL-GREEN, 2 STOP-AND-SURFACE for REAL
defects (content / a sentence-length metric) — real defects still surfaced, so
no fail-open. Golden + new drift/fallback unit tests green; fmt clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqFDFXun5AjrYXuKwuuYmg
ProfessorVR
force-pushed
the
feat/fcdp-draft
branch
from
July 9, 2026 23:24
cdc4bab to
595037a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(draft): archon-draft crate + FCDP gate pipeline (sandbox-validated)
What
Integrates the FCDP (Fable Console Drafting Protocol) into archon as a new
workspace crate + orchestration scripts + gate data:
crates/archon-draft— typed context-pack schema with G-P validation;«Qnn» quote-ID substitution (quoted text is never model-generated — it enters
only by mechanical substitution from a PDF-verified bank); stylometric
measurement over
archon-lanham; a two-tier, variance-derived G-A style gate.Binary
archon-fcdp:measure | gp-validate | substitute | ga-gate.scripts/fcdp/— stage orchestration (D1 movement plan → D1.5 discourseskeleton → movement-by-movement D2 drafting), judge gates G-E/G-G as fresh
lean-context model calls (randomized binary batteries, rationale-before-verdict,
free-prose extraction, fail-closed), mechanical gauntlet G-B/C/D/F, hash-chained
provenance with generated AI-use declaration, bounded R-loop drivers.
crates/archon-draft/data/— locked G-A gate config (MA-applicationsregister; Tier-1 pooled P10–P90 per-section, Tier-2 2500w P5–P95 chapter-scale,
labels inherit their source metric's tier), derivation artifacts, exemplar pool.
Validation
Developed and validated in an isolated sandbox against archon crates as
read-only path deps (archon untouched), then promoted:
leave-one-document-out cross-validation; empirical feature tiering.
literal-quote violations); 3/3 pack defects caught by G-P; judge gates caught
semantic defects invisible to greps (ungraded assertions, memory loci,
quote misuse, reviser-invented content).
evidence items) ran pack → D1 → D1.5 → D2 → full gauntlet → R-loop to
ALL GATES GREEN; provenance chain verified; declaration generated.
identical G-A verdicts).
Design provenance
Research-hardened plan:
plans/fcdp-archon-port-plan-v2-2026-07-07.md(claudeflow-testing repo) — evidence-graded moves (variance-derived bands,
scale-matched gating, exemplar conditioning, judge hardening without same-model
panels, free-prose judge output, provenance-backed disclosure).
Follow-ups (tracked in
crates/archon-draft/data/README.md)archon draftsubcommand surface (CLI wiring)archon-provenance(CozoDB)🤖 Generated with Claude Code