Describe the bug
Semantic/vector search silently omits notes from results when an entity row and a relation (or observation) row in search_index share the same numeric id. Because each row type has its own auto-increment sequence, collisions are essentially guaranteed in any young database (entity ids 1..N overlap relation ids 1..M), and the affected notes are simply never returned by vector or hybrid search — no error, no log.
Root cause
In src/basic_memory/repository/search_repository_base.py:
_search_vector_only parses each vector hit's chunk_key (e.g. entity:4:0) but discards the type, keeping only the bare id:
_, si_id = self._parse_chunk_key(chunk_key)
_fetch_search_index_rows_by_ids then fetches WHERE id IN (...) with no type filter and builds a dict keyed by bare id:
result[row.id] = SearchIndexRow(...)
row is None and is dropped.
Note the FTS-filter branch a few lines above already guards against exactly this with (id, type) tuple keys ("Use (id, type) tuples to avoid collisions between different search_index row types that share the same auto-increment id") — but the primary lookup path doesn't.
Concrete example (real database, basic-memory 0.21.6, SQLite backend)
search_index contained entity rows with ids 1–4 and relation rows with ids 1, 2, 4, 5:
- entity 1 ("Learner Profile") — clobbered by relation 1 → never returned
- entity 2 — its row happened to be inserted after relation 2 → returned
- entity 3 — no colliding relation id 3 → returned
- entity 4 (a concept note) — clobbered by relation 4 → never returned
A direct KNN query against search_vector_embeddings ranked entity 4's chunk #1 for the test query (distance 0.64), confirming the embeddings are fine — the result is lost purely in the id→row hydration step. Searching with min_similarity=0 and verbatim text from the affected notes still returns nothing for them.
Expected behavior
Every vector hit should hydrate against the search_index row of the same type. Keying similarity_by_si_id / chunks_by_si_id / _fetch_search_index_rows_by_ids by (type, id) tuples fixes it (verified locally on 0.21.6 — the dropped notes immediately return at the expected ranks).
Environment
- basic-memory 0.21.6 (bug also present in v0.22.0 source and current
main)
- SQLite backend, semantic search enabled (fastembed, bge-small-en-v1.5)
- Windows 11
Describe the bug
Semantic/vector search silently omits notes from results when an
entityrow and arelation(orobservation) row insearch_indexshare the same numericid. Because each row type has its own auto-increment sequence, collisions are essentially guaranteed in any young database (entity ids 1..N overlap relation ids 1..M), and the affected notes are simply never returned by vector or hybrid search — no error, no log.Root cause
In
src/basic_memory/repository/search_repository_base.py:_search_vector_onlyparses each vector hit'schunk_key(e.g.entity:4:0) but discards the type, keeping only the bare id:_fetch_search_index_rows_by_idsthen fetchesWHERE id IN (...)with no type filter and builds a dict keyed by bare id:row is Noneand is dropped.Note the FTS-filter branch a few lines above already guards against exactly this with
(id, type)tuple keys ("Use (id, type) tuples to avoid collisions between different search_index row types that share the same auto-increment id") — but the primary lookup path doesn't.Concrete example (real database, basic-memory 0.21.6, SQLite backend)
search_indexcontained entity rows with ids 1–4 and relation rows with ids 1, 2, 4, 5:A direct KNN query against
search_vector_embeddingsranked entity 4's chunk #1 for the test query (distance 0.64), confirming the embeddings are fine — the result is lost purely in the id→row hydration step. Searching withmin_similarity=0and verbatim text from the affected notes still returns nothing for them.Expected behavior
Every vector hit should hydrate against the search_index row of the same type. Keying
similarity_by_si_id/chunks_by_si_id/_fetch_search_index_rows_by_idsby(type, id)tuples fixes it (verified locally on 0.21.6 — the dropped notes immediately return at the expected ranks).Environment
main)