feat(memory): embedding foundation — ONNX runtime, provisioning, embed-on-write, backfill (memory-core-redesign slice 2) - #1577
Merged
Aaronontheweb merged 10 commits intoJul 5, 2026
Conversation
Adds the src/Netclaw.Embeddings project (Microsoft.ML.OnnxRuntime CPU EP, FastBertTokenizer, System.Numerics.Tensors), referenced by nothing yet — daemon/CLI wiring is Stage B. - OnnxMemoryEmbedder: single InferenceSession (IntraOpNumThreads=4), BoundedConcurrencyGate (default max 2 concurrent inferences, peak concurrency observable for tests), FastBertTokenizer WordPiece tokenization truncated to 512 tokens. Feeds only the input names the loaded ONNX graph declares rather than hardcoding the 3-input BERT signature. CLS-token pooling (last_hidden_state[:, 0, :]) + L2 normalization — verified against both allowlisted models' model cards (snowflake-arctic-embed-m: "use the CLS token"; mxbai-embed-large-v1: "works really well with cls pooling (default)"). - EmbeddingModelProvisioner: pinned in-code allowlist (model id -> URL, SHA-256, byte size, dimensions) injected as a required dependency (not a hardcoded internal) so tests can supply a localhost-pointed allowlist instead of ever reaching the real HuggingFace URLs. Atomic temp-file-then-rename download, byte-size + SHA-256 verification before the destination file is ever created, unknown-id rejection listing the allowlist. Allowlist entries (URLs pinned to a specific upstream commit, not `main`): snowflake-arctic-embed-m (768 dims, plain fp32 model.onnx, ~416 MB) and mxbai-embed-large-v1 fallback (1024 dims, ~1.27 GB fp32). Tests (Netclaw.Embeddings.Tests, no network): - Tiny fixture ONNX graph + WordPiece vocab (generated by Fixtures/generate_fixture_model.py, committed with a header comment explaining the graph shape and regeneration steps) exercise OnnxMemoryEmbedder end-to-end: deterministic output, L2-normalized, content-sensitive (attention-masked mean pooling reported at the CLS position so a content-blind bug would be caught), batch order preservation. - BoundedConcurrencyGate tested in isolation against a controlled fake delayed workload (Task.Delay lives in the fake, not in test orchestration) proving the concurrency bound is actually enforced. - EmbeddingModelProvisioner tested against a local HttpListener fixture: hash-mismatch and byte-size-mismatch rejection with no leftover temp files, unknown-id rejection listing the allowlist, successful provision leaves exactly the two expected files. opsx: memory-core-redesign slice 2
…eddings schema - IMemoryEmbedder (+ UnavailableMemoryEmbedder degraded stub) in Netclaw.Actors/Memory so actor code carries no OnnxRuntime dependency; Netclaw.Embeddings implements the interface, never the reverse. UnavailableMemoryEmbedder throws InvalidOperationException with remediation text on Embed*Async rather than returning a garbage vector. - MemoryContentHasher: SHA-256 over normalized title+body, reusing CurationRulesEvaluator.NormalizeForContainment (promoted from private to internal) rather than a second hand-rolled normalizer, so curation's destructive-update guard and the embedding re-embed skip can't quietly disagree about what counts as changed content. - MemoryVectorIndex: per-model flat float[] + parallel id/kind arrays bundled into an immutable snapshot (no torn reads), TopK via System.Numerics.Tensors.TensorPrimitives.CosineSimilarity with a minCosine floor, reload gated on SQLiteMemoryStore.EmbeddingDataVersion so unchanged turns pay no reload cost. - SQLiteMemoryStore: memory_embeddings(item_id, item_kind, model_id, content_hash, dims, vector BLOB, created_at) DDL in the existing idempotent InitializeAsync; UpsertEmbeddingAsync (hash-skip: no write and no EmbeddingDataVersion bump when the content hash is unchanged, float32 LE blob); GetEmbeddingsForModelAsync (thin query for the vector index to consume — the design's FindNearestByEmbeddingAsync, renamed per plan); GetEmbeddingCoverageAsync (total recallable docs, embedded-current-hash count, other-model count) for the coverage diagnostics spec requirement; TombstoneDocumentAsync extended to delete the tombstoned document's embedding rows in the same transaction and bump the version counter, since vectors are derived data that must not keep surfacing a dead document as a kNN neighbor. No production code path calls any of this yet (embed-on-write, the vector index's runtime wiring, and the doctor/status degradation surfaces are Stage B) — this slice writes vectors, nothing reads them, zero behavior risk per the design's migration plan. opsx: memory-core-redesign slice 2
… slice 2) Task 2.12 (tests) stays unchecked — its warmup/gap-repair/doctor-facing scenarios land with Stage B daemon wiring; the store/index/hasher/ provisioner/embedder subset testable at this layer is covered.
…seams, config (opsx: memory-core-redesign slice 2, tasks 2.7/2.8/2.11)
- MemoryEmbedderHolder: mutable holder the warmup hosted service populates
(hosted-service startup order vs construction-time DI documented on the type)
- MemoryEmbedOnWriteCoordinator: single embed-on-write hook both curation
pipelines call post-commit; embedding failures never fail the memory write
(vectors are derived data, D3)
- SQLiteMemoryStore: ApplyInlineCurationBatchAsync/ApplyCurationBatchAsync now
return the written document rows (the post-commit ids+content the coordinator
needs); GetDocumentsNeedingEmbeddingAsync derives gap-repair/backfill state
(never a progress table); UpsertEmbeddingAsync reports wrote-vs-skipped
- MemoryCurationActor + MemoryCurationWorkerService callers embed after commit
- Memory.Embeddings config { Enabled=false (deliberate staging, flipped in
Slice 3/4), ModelId=snowflake-arctic-embed-m, AutoDownload=true } + schema
sync with defaults; NetclawPaths.ModelsDirectory + EmbeddingModelDirectory
- DaemonRuntimeStatus.Embeddings wire type (ok/degraded/disabled)
- EmbeddingModelProvisioner: skip-if-valid local copy (no network on restart)
+ TryLoadVerifiedAsync for AutoDownload=false paths
…urface (opsx: memory-core-redesign slice 2, tasks 2.7/2.8/2.10) - EmbeddingWarmupHostedService: provision-or-degrade at startup (AutoDownload gates the network path entirely — even to repair a corrupt local copy), one warm-up inference, then a batched (16, yielding) gap-repair sweep over documents missing a current-model/current-hash embedding - ANY failure => UnavailableMemoryEmbedder + error-level memory_embedding_unavailable log; daemon NEVER fails startup on embeddings - DI: holder starts as Unavailable stub; warmup populates it; SessionMemoryServices carries it to the inline curation actor - MemoryCurationWorkerService embeds written docs post-commit (task 2.8's second pipeline call site) - DaemonRuntimeStatusService reports embeddings: ok/degraded/disabled with modelId under the Memory status block
…k (opsx: memory-core-redesign slice 2, tasks 2.9/2.10) - New 'netclaw memory' command group (offline, direct SQLite/model-file access) with backfill-embeddings [--force]: provisions if needed (clear error when AutoDownload=false and model missing), embeds in batches of 16 with progress output, final embedded/skipped-hash-unchanged/failed summary; safe against a live daemon (WAL + per-item upserts whose hash check re-queries at call time) - MemoryEmbeddingDoctorCheck: Error when Enabled but model missing/hash-invalid; Warning on missing current-model embeddings (count) or mixed-model corpus (recommends --force backfill); Pass with coverage summary; Pass when disabled - Allowlist is an injected dependency on both (same seam as the provisioner) so tests use the tiny fixture model — no network in tests - Schema round-trip tests for the Memory.Embeddings config section
…2.12 complete (memory-core-redesign slice 2) design.md D2 said 'snowflake-arctic-embed 137M int8' — Stage A shipped the ~110M-param arctic-embed-m fp32 ONNX artifact pinned by hash; int8 is noted as a future optimization, not what the allowlist points at.
…(opsx: memory-core-redesign slice 2)
…sx) design.md Add tools/embed-latency-bench (standalone console, kept out of Netclaw.slnx): loads the production OnnxMemoryEmbedder path (hash-verified snowflake-arctic-embed-m, same pooling/threading/concurrency-gate config the daemon uses) and times batch=1 EmbedAsync calls across short-query/medium/doc-length corpora (20 warmup + 200 timed iterations each), plus cold-load and a concurrency=2 pass. Measured on the i9-9900K reference box (8 logical cores, contended: load avg 2.0-3.6, ~11/15 GiB RAM in use, live daemon running): short-query p50 281ms / p95 315ms - statistically indistinguishable from doc-length (p50 275ms / p95 294ms) because OnnxMemoryEmbedder pads every input to a fixed 512 tokens regardless of actual length, so the fixed-size fp32 forward pass dominates latency, not tokenization. Resolves memory-core-redesign task 2.13 and its design.md open question: the 150ms query-embedding sub-budget does NOT hold on this hardware (p95 ~2.1x over budget, margin ~-165ms). Updates D6, the Risks/Trade-offs entry, and the Open Questions table with the measured numbers and verdict; corrects stale "ONNX int8" wording to match the D2-shipped fp32 reality (int8 remains a deferred optimization). Highest-leverage unexplored mitigation: a query-specific max-length well below 512, not int8 quantization.
Extends tools/embed-latency-bench with a bench-only parallel code path (OnnxMemoryEmbedder production code untouched) that: - inspects InferenceSession.InputMetadata to confirm the ONNX graph's sequence axis is symbolic (dynamic), not fixed - runs the same short/medium/doc corpora padded to actual tokenized length (bucket-of-8 rounding) instead of fixed 512, same 20 warmup / 200 timed / batch=1 / Release protocol - cross-checks correctness: cosine similarity between fixed-512 and dynamic-length embeddings for 10 fixed sentences - records load average before/after for honest contention context Measured on the reference box: short-query p50 19.0ms / p95 20.9ms (vs 281.9ms / 310.5ms fixed-512) with 1.000000 cosine parity across all 10 sentences. Well under the 150ms Slice 4 sub-budget. Updates openspec/changes/memory-core-redesign/design.md (D6, Risks, Open Questions) with the measured numbers and the decision to adopt dynamic sequence length as the Slice 4 mitigation.
Merged
Aaronontheweb
merged commit Jul 5, 2026
08a7bb5
into
netclaw-dev:feature/memory-embeddings
14 of 15 checks passed
2 tasks
Aaronontheweb
added a commit
that referenced
this pull request
Jul 8, 2026
#1603) The memory-core-redesign slice 2 (#1577) added the `netclaw memory` CLI command but never updated the approved `help` screenshot baseline, so Screenshot Regression (Linux) has been red on every push to feature/memory-embeddings since (verified via gh run list against feature/memory-embeddings: 3/3 recent pushes failed this check). Reviewed the captured diff locally: the only change is the new memory Manage cross-session memory (embeddings backfill, offline) line shifting everything below it down one row. No other screenshot frame changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Slice 2 of the memory-core-redesign OpenSpec change (tasks 2.1–2.15, all checked): the complete embedding foundation. Ships dark by design —
Memory.Embeddings.Enableddefaults tofalse; nothing reads vectors until slices 3 (write-side nominator) and 4 (hybrid recall), so this slice is zero-behavior-risk per the migration plan. The enable-flip happens consciously in those slices.What's inside
Library (
src/Netclaw.Embeddings, new)OnnxMemoryEmbedder: in-process CPU inference (Microsoft.ML.OnnxRuntime + FastBertTokenizer), CLS pooling verified against both model cards and end-to-end against the real model, bounded concurrency.EmbeddingModelProvisioner: pinned in-code allowlist —snowflake-arctic-embed-m(primary) andmxbai-embed-large-v1(fallback), URLs pinned to upstream commit shas, SHA-256s verified against HuggingFace LFS object ids by actually downloading the artifacts. Atomic download, hash-verify-before-load, unknown-id rejection. Arbitrary URLs impossible.IMemoryEmbedderseam lives in Netclaw.Actors (consumer-defined; Actors does NOT reference Embeddings or OnnxRuntime) +UnavailableMemoryEmbedderdegraded stub that throws with remediation rather than returning garbage.Store + index (Netclaw.Actors)
memory_embeddingstable keyed(item_id, model_id)+ content hash: model swap = new rows + forced backfill, zero rewrite of the documents table; backfill state is derived, never a progress table; tombstones delete embedding rows.MemoryVectorIndex: brute-force TensorPrimitives cosine over a flat matrix, version-invalidated (~1.8MB at current corpus scale).MemoryContentHasher: normalized title+body SHA-256; hash-unchanged re-embeds are free.Daemon + CLI integration
EmbeddingWarmupHostedService: provision-or-degrade at startup, warm-up inference, gap-repair sweep (crash between write and embed self-heals). The daemon NEVER fails startup over embeddings — degraded is loud, not fatal: error-levelmemory_embedding_unavailable, daemon statusMemory.Embeddings.Status: degraded, doctor Error.MemoryEmbedOnWriteCoordinatorwith two thin call sites (there is no single physical commit point — the two store batch-apply methods assign document ids inside their transactions, so both now return write results and feed the one coordinator). Embedding failure never fails a memory write: vectors are derived data.netclaw memory backfill-embeddings [--force](new command group), live-daemon-safe.MemoryEmbeddingDoctorCheck: missing/hash-invalid model → Error; incomplete coverage or mixed-model corpus → Warning (mixed models invalidate similarity thresholds); coverage summary otherwise.Memory.Embeddings { Enabled=false, ModelId, AutoDownload }+ schema sync; skills synced (netclaw-memory 1.8.0, netclaw-operations 2.25.0); ARM64 release leg verifies published binaries are genuinely aarch64 (cross-compiled; natives are bundled single-file so ELF-arch check is the maximal cheap verification).Short-query embedding via the production path on the reference i9-9900K (contended, live daemon):
The Slice-4 150ms query-embedding sub-budget does not hold (p95 2.1× over). Root cause is diagnostic: the embedder pads every input to a fixed 512 tokens, so the forward pass is length-invariant — a 14-token query costs the same as a 450-token document. Mitigations ranked in design.md (now updated with the measured table): dynamic sequence length (highest leverage, O(n²) attention), int8 quantization, budget adjustment with the already-designed loud lexical fallback. A dynamic-length experiment is running before Slice 4's design is finalized. Cold load: 1.07s (LoadAsync+first embed) — the warmup service exists for exactly this.
Validation
memory_identity_preference_routing5/5 now that fix(evals): transcript assertions must grep with -a (invalid UTF-8 silently defeats matching) #1576's assertion fix is in.