feat(workspaces): add hybrid workspace search - #701
Conversation
Add a Vectorize-backed workspace search module (indexer, embeddings, chunking, ranking, query) with a search-workspace operation and AI tool wiring, plus the WORKSPACE_SEARCH Vectorize binding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews. |
|
React Doctor found 5 new issues in 3 files · 5 warnings · score 89 / 100 (Great) · 1 fixed · vs 5 warnings
Reviewed by React Doctor for commit |
📝 WalkthroughWalkthroughAdds workspace search across contracts, chunking, indexing, semantic and keyword querying, vector storage, extraction healing, kernel integration, and the read-only ChangesWorkspace search foundations
Extraction recovery and workspace integration
Search querying and AI tool delivery
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant searchWorkspaceOperation
participant WorkspaceKernel
participant WorkspaceSearchProjection
participant WorkspaceSearchQuery
participant Vectorize
Client->>searchWorkspaceOperation: Submit WorkspaceSearchInput
searchWorkspaceOperation->>WorkspaceKernel: Authorize and call searchWorkspace
WorkspaceKernel->>WorkspaceSearchProjection: Search workspace
WorkspaceSearchProjection->>WorkspaceSearchQuery: Resolve scope and query indexes
WorkspaceSearchQuery->>Vectorize: Run semantic vector search
Vectorize-->>WorkspaceSearchQuery: Return vector matches
WorkspaceSearchQuery-->>WorkspaceSearchProjection: Return results, failures, and status
WorkspaceSearchProjection-->>WorkspaceKernel: Return search output
WorkspaceKernel-->>searchWorkspaceOperation: Return search output
searchWorkspaceOperation-->>Client: Return results and references
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bef73b8bdf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let failed = 0; | ||
|
|
||
| try { | ||
| await this.search.purgeVectors(); |
There was a problem hiding this comment.
If deleteByIds fails transiently here, the error is caught and purgeForDeletion still calls this.ctx.storage.deleteAll() later, destroying the only inventory of chunk IDs and queued deletions. Because workspace deletion invokes this purge only once, the workspace's vectors can then remain in Vectorize indefinitely with no way to retry their deletion; retain the local state or fail the purge before deleting it.
Useful? React with 👍 / 👎.
| ) | ||
| ) | ||
| ORDER BY i.created_at ASC | ||
| LIMIT ${workflowBatchSize} |
There was a problem hiding this comment.
When more than 100 file projections are eligible for repair, this query returns only the oldest batch and the function never schedules or loops for another batch. Any remaining files stay unqueued—and therefore unavailable to search—until some unrelated future start or connection invokes reconciliation again, even if the workspace remains active; continue fetching after each submitted batch or explicitly schedule another pass.
Useful? React with 👍 / 👎.
| if (input.kind === "item") { | ||
| return [{ itemId: input.itemId }]; | ||
| } | ||
| const type = input.types.length === 1 ? input.types[0] : undefined; |
There was a problem hiding this comment.
The input schema allows duplicate enum values, so an input such as types: ["file", "file"] reaches this branch with length 2 and disables the type metadata filter. The semantic query then takes its bounded top-K results across both documents and files before the SQL loader discards documents, which can hide every relevant file in a document-heavy workspace; derive this condition from the unique type set or reject duplicates.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (10)
src/features/workspaces/search/workspace-search-scope.ts (1)
134-158: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueReturn a scope failure instead of throwing when one folder identifier exceeds the filter limit.
batchFolderFiltersthrows at Line 148.resolveWorkspaceSearchScopedoes not catch it, so the error propagates out ofsearch()and fails the whole tool call. A single identifier cannot realistically exceed 2,048 bytes today, so this is defensive only. Consider skipping the oversized identifier and continuing with the remaining batches.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/search/workspace-search-scope.ts` around lines 134 - 158, The batchFolderFilters function should not throw when a single folder identifier exceeds the Vectorize filter limit, since resolveWorkspaceSearchScope does not handle that exception. Skip the oversized identifier and continue processing subsequent folder IDs, while preserving normal batching and filter generation for valid identifiers.src/features/workspaces/ai/workspace-tool-result-adapters.ts (1)
10-24: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winMake
projectOutputfail safe likecollectReferences.
collectReferencesusessafeParseand returns an empty array on a validation failure.projectOutputusesparseand throws.createWorkspaceThreadToolinsrc/features/workspaces/ai/workspace-tools.ts(Lines 34-40) callsprojectOutputfromtoModelOutputafter the operation has already completed, so one unexpected field breaks the assistant turn and discards a successful search. The tool output is never validated against the schema before this point. Fall back to the raw output when parsing fails.♻️ Proposed fail-safe projection
projectOutput: (output: unknown) => { - return input.projectOutput(input.outputSchema.parse(output)) as JSONValue; + const parsed = input.outputSchema.safeParse(output); + return ( + parsed.success ? input.projectOutput(parsed.data) : output + ) as JSONValue; },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/ai/workspace-tool-result-adapters.ts` around lines 10 - 24, Update projectOutput in defineWorkspaceToolResultAdapter to validate with safeParse instead of throwing via parse. Return the projected parsed data on successful validation, and fall back to the raw output when validation fails so createWorkspaceThreadTool can complete successfully.src/features/workspaces/kernel/workspace-kernel.ts (1)
144-157: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winThrottle extraction healing on connect.
onConnectrunsrequestWorkspaceFileExtractionHealingfor every client connection. Each call runs the reconciler query and, when candidates exist, onecreateBatchcall on the workflow binding. Frequent reconnects therefore repeat this work, even though the grace and cooldown windows keep the workflow identifiers stable. Store the last healing timestamp and skip the call inside a short window.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/kernel/workspace-kernel.ts` around lines 144 - 157, The onConnect method currently triggers workspace file extraction healing for every authenticated connection. Add a timestamp field for the last healing request and update requestWorkspaceFileExtractionHealing invocation to skip execution within a short throttle window, while preserving the existing behavior after the window expires and continuing to broadcast presence snapshots.src/features/workspaces/search/workspace-search-schema.ts (2)
72-85: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winAdd an index for the pending-queue ordering.
WorkspaceSearchIndexer.processBatchrunsSELECT item_id FROM kernel_search_pending ORDER BY requested_at ASC LIMIT 2. The primary key isitem_id, so SQLite sorts the whole pending set on every batch. The same applies tokernel_search_vector_deletesordered byrequested_atinflushVectorDeletes. Both queues can hold one row per workspace item during a full rebuild.♻️ Proposed indexes
sql` CREATE TABLE IF NOT EXISTS kernel_search_pending ( item_id TEXT PRIMARY KEY, requested_at INTEGER NOT NULL, attempts INTEGER NOT NULL DEFAULT 0 ) `; + sql`CREATE INDEX IF NOT EXISTS kernel_search_pending_requested_idx + ON kernel_search_pending (requested_at)`; + sql`CREATE INDEX IF NOT EXISTS kernel_search_vector_deletes_requested_idx + ON kernel_search_vector_deletes (requested_at)`;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/search/workspace-search-schema.ts` around lines 72 - 85, Add indexes on requested_at for both kernel_search_pending and kernel_search_vector_deletes in the workspace search schema setup, so WorkspaceSearchIndexer.processBatch and flushVectorDeletes can efficiently execute their ordering queries. Use the existing schema initialization alongside the CREATE TABLE statements and preserve the current queue behavior.
64-71: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftConsider a rowid mapping for
kernel_search_ftsto avoid full-index scans on delete.
chunk_idisUNINDEXED, so it is stored but not searchable.WorkspaceSearchIndexer.deleteLocalChunks(workspace-search-indexer.tslines 418-421) deletes rows withWHERE chunk_id IN (...). FTS5 cannot use an index for that predicate, so each reindex of a single item scans every row in the FTS table. The cost grows with total workspace chunks, not with the chunks of the item.Two options:
- Store the FTS
rowidinkernel_search_chunkswhen inserting, then delete fromkernel_search_ftsbyrowid.- Use an external-content FTS5 table backed by
kernel_search_chunks.Both keep delete cost proportional to the item.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/search/workspace-search-schema.ts` around lines 64 - 71, Update the FTS5 indexing and deletion flow around kernel_search_fts and WorkspaceSearchIndexer.deleteLocalChunks to avoid filtering on the unindexed chunk_id column. Persist a rowid mapping in kernel_search_chunks during insertion and delete FTS rows by rowid, or switch to an external-content table backed by kernel_search_chunks; ensure reindexing deletes only the affected item’s rows without scanning the full FTS table.src/features/workspaces/search/workspace-search-indexer.ts (1)
342-353: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winFail on a missing embedding instead of upserting an empty vector.
Line 335 validates the count, so
embeddings[index]should always exist. The?? []fallback converts an unexpected hole into a zero-dimension vector, which Vectorize rejects with a dimension error that names the batch rather than the cause. Throw a named error instead.♻️ Proposed change
const vectors = chunks.map( - (chunk, index): VectorizeVector => ({ - id: chunk.chunkId, + (chunk, index): VectorizeVector => { + const values = embeddings[index]; + if (!values) { + throw new Error("Workspace search embedding was missing for an indexed chunk."); + } + return { + id: chunk.chunkId, metadata: createWorkspaceSearchVectorMetadata({ itemId: source.itemId, parentId: source.parentId, type: source.type, }), namespace: this.workspaceId(), - values: embeddings[index] ?? [], - }), + values, + }; + }, );🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/search/workspace-search-indexer.ts` around lines 342 - 353, Update the vector mapping in the chunk-to-vector flow to replace the embeddings[index] ?? [] fallback with an explicit named error when an embedding is missing. Preserve the existing validated embedding values and ensure the failure identifies the missing embedding before the upsert reaches Vectorize.wrangler.jsonc (1)
37-44: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRecord the out-of-band Vectorize provisioning steps.
The comment states the metadata index requirement, which is good. Wrangler cannot create metadata indexes from this file, so the requirement is enforced by nothing in the repository. The same applies to the index dimensions and distance metric, which must match the embedding model used by
embedWorkspaceSearchTexts.If a metadata index for
parentIdortypeis missing, scoped queries return no semantic results and fall back to keyword-only, which looks like a ranking bug rather than a provisioning gap. If the dimensions do not match, every upsert fails.Add the exact
wrangler vectorize create-indexandwrangler vectorize create-metadata-indexcommands for both indexes to the deployment documentation, and reference that document here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@wrangler.jsonc` around lines 37 - 44, Document the out-of-band Vectorize provisioning for both indexes configured in the vectorize section, including exact create-index and create-metadata-index commands with the embedding model’s required dimensions, distance metric, and itemId, parentId, and type fields. Add a reference to that deployment document beside the wrangler configuration comment so provisioning requirements remain discoverable.src/features/workspaces/search/workspace-search-projection.ts (1)
37-83: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMake the event switch exhaustive.
The switch lists each handled
WorkspaceRealtimeEventtype and has nodefaultbranch. When a new event type is added to the union, this code compiles and silently ignores it, so the affected item is never re-indexed and search results go stale without any signal.Add a
defaultbranch with aneverassignment so the compiler reports unhandled event types.♻️ Proposed change
case "workspace.item.color.updated": case "workspace.relations.updated": return; + default: { + const unhandled: never = event; + void unhandled; + return; + } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/search/workspace-search-projection.ts` around lines 37 - 83, Add a default branch to the event switch in observe, assigning the event to a never-typed variable so TypeScript reports any newly added WorkspaceRealtimeEvent variants as compile-time errors. Leave the existing event handling and requestRun flow unchanged.src/features/workspaces/search/workspace-search-query.ts (2)
91-113: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winReport degraded semantic retrieval in
status.If embeddings or Vectorize fail,
searchSemanticWithFallbackreturns[]and the response still reports the pre-computedstatus. A workspace with a healthy index then returnsstatus: "ready"while the results are keyword-only. Propagate the fallback so callers and the model can tell that recall was reduced.♻️ Proposed change to surface the fallback
- const semantic = await this.searchSemanticWithFallback({ + const semantic = await this.searchSemanticWithFallback({ candidateLimit, query: input.query, scope, types, });Return a flag from
searchSemanticWithFallback(for example{ candidates, degraded }) and downgradestatusto"partial"whendegradedis true.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/search/workspace-search-query.ts` around lines 91 - 113, Update the search flow around searchSemanticWithFallback to return and consume a degraded indicator alongside semantic candidates. When semantic retrieval falls back to an empty result because embeddings or Vectorize fail, set the response status to "partial" instead of the precomputed status; preserve the existing status and ranking behavior when retrieval succeeds.
199-203: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the source-version freshness expression into one shared SQL fragment.
This CASE expression is duplicated verbatim at Lines 321-325, and the indexer holds a third copy. The contract in
workspace-search-version.tsrequires all copies to mirrorbuildWorkspaceSearchSourceVersion. If one copy drifts, keyword and semantic queries silently return stale chunks or no chunks at all. Define the expression once and reuse it in both methods.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/search/workspace-search-query.ts` around lines 199 - 203, Extract the CASE expression from the workspace search query into a shared SQL fragment, using the canonical source-version expression defined by buildWorkspaceSearchSourceVersion in workspace-search-version.ts. Reuse that fragment in both query methods, including the duplicate around the later source_version predicate, so keyword and semantic paths remain identical.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/features/workspaces/kernel/workspace-kernel.ts`:
- Around line 326-332: Update processWorkspaceSearchIndex and the underlying
search.processBatch flow to atomically claim selected kernel_search_pending rows
before reading or indexing their batches. Ensure concurrent due invocations
cannot select the same pending source while preserving the existing parallel
indexing and scheduling behavior.
In `@src/features/workspaces/search/workspace-search-chunks.ts`:
- Around line 78-86: Update the separator lookup in the chunk-boundary logic to
search from hardEnd minus separator.length, ensuring the returned boundary plus
the separator length never exceeds hardEnd or targetChunkCharacters. Preserve
the existing minimumEnd validation and hardEnd fallback.
In `@src/features/workspaces/search/workspace-search-contract.ts`:
- Around line 22-27: Normalize the optional types array in
workspaceSearchItemTypeSchema before createWorkspaceSearchVectorFilters consumes
it, deduplicating repeated values so identical entries count as one semantic
type. Preserve the existing optional behavior and the .min(1)/.max(2) validation
while ensuring inputs such as duplicate document values produce a single-element
array and retain the type filter.
In `@src/features/workspaces/search/workspace-search-indexer.ts`:
- Around line 158-170: Update purgeVectors so each successfully deleted batch
also removes its corresponding rows from kernel_search_vector_deletes, while
leaving rows for failed or unattempted batches intact. Perform the database
deletion only after vectorize.deleteByIds resolves, and allow failures to
propagate so the caller can retry the remaining ids.
- Around line 446-470: Update flushVectorDeletes to bound retries for failed
vector deletions, matching the existing markIndexAttemptFailed behavior: add or
reuse persistent attempt tracking for kernel_search_vector_deletes, increment it
when vectorize.deleteByIds fails, and remove or park rows after the established
maximum failure count so hasPending and processBatch cannot trigger an unbounded
retry loop.
- Around line 139-156: Update processWorkspaceSearchIndex to catch processBatch
rejections, invoke processWorkspaceSearchIndex again to reschedule retryable
pending work, and return false when attempts are exhausted. Preserve the
existing successful truthy-return scheduling behavior and ensure AggregateError
failures do not leave retryable rows unscheduled.
- Around line 483-501: Update markIndexAttemptFailed so that when the maximum
indexing attempts are exhausted, it queues the item’s vectors for deletion and
removes its local rows from kernel_search_chunks and kernel_search_fts before
setting vector_status to 'failed' and deleting the pending entry. Preserve the
existing retry behavior for attempts below maximumIndexAttempts.
- Around line 84-98: The SQL freshness CASE in seedPendingItems must remain
identical to buildWorkspaceSearchSourceVersion in workspace-search-version.ts.
Add coverage for both document and file rows that computes the SQL CASE output
and compares it with buildWorkspaceSearchSourceVersion, including the
file-specific updated_at and source_hash components, so version changes cannot
cause pending items to be skipped or retained indefinitely.
In `@src/features/workspaces/search/workspace-search-scope.ts`:
- Around line 111-131: The Vectorize indexes required by
createWorkspaceSearchVectorFilters are not being provisioned. Add string
metadata index creation for parentId and type on both
thinkex-workspace-search-staging and thinkex-workspace-search, while preserving
the existing bindings and configuration.
---
Nitpick comments:
In `@src/features/workspaces/ai/workspace-tool-result-adapters.ts`:
- Around line 10-24: Update projectOutput in defineWorkspaceToolResultAdapter to
validate with safeParse instead of throwing via parse. Return the projected
parsed data on successful validation, and fall back to the raw output when
validation fails so createWorkspaceThreadTool can complete successfully.
In `@src/features/workspaces/kernel/workspace-kernel.ts`:
- Around line 144-157: The onConnect method currently triggers workspace file
extraction healing for every authenticated connection. Add a timestamp field for
the last healing request and update requestWorkspaceFileExtractionHealing
invocation to skip execution within a short throttle window, while preserving
the existing behavior after the window expires and continuing to broadcast
presence snapshots.
In `@src/features/workspaces/search/workspace-search-indexer.ts`:
- Around line 342-353: Update the vector mapping in the chunk-to-vector flow to
replace the embeddings[index] ?? [] fallback with an explicit named error when
an embedding is missing. Preserve the existing validated embedding values and
ensure the failure identifies the missing embedding before the upsert reaches
Vectorize.
In `@src/features/workspaces/search/workspace-search-projection.ts`:
- Around line 37-83: Add a default branch to the event switch in observe,
assigning the event to a never-typed variable so TypeScript reports any newly
added WorkspaceRealtimeEvent variants as compile-time errors. Leave the existing
event handling and requestRun flow unchanged.
In `@src/features/workspaces/search/workspace-search-query.ts`:
- Around line 91-113: Update the search flow around searchSemanticWithFallback
to return and consume a degraded indicator alongside semantic candidates. When
semantic retrieval falls back to an empty result because embeddings or Vectorize
fail, set the response status to "partial" instead of the precomputed status;
preserve the existing status and ranking behavior when retrieval succeeds.
- Around line 199-203: Extract the CASE expression from the workspace search
query into a shared SQL fragment, using the canonical source-version expression
defined by buildWorkspaceSearchSourceVersion in workspace-search-version.ts.
Reuse that fragment in both query methods, including the duplicate around the
later source_version predicate, so keyword and semantic paths remain identical.
In `@src/features/workspaces/search/workspace-search-schema.ts`:
- Around line 72-85: Add indexes on requested_at for both kernel_search_pending
and kernel_search_vector_deletes in the workspace search schema setup, so
WorkspaceSearchIndexer.processBatch and flushVectorDeletes can efficiently
execute their ordering queries. Use the existing schema initialization alongside
the CREATE TABLE statements and preserve the current queue behavior.
- Around line 64-71: Update the FTS5 indexing and deletion flow around
kernel_search_fts and WorkspaceSearchIndexer.deleteLocalChunks to avoid
filtering on the unindexed chunk_id column. Persist a rowid mapping in
kernel_search_chunks during insertion and delete FTS rows by rowid, or switch to
an external-content table backed by kernel_search_chunks; ensure reindexing
deletes only the affected item’s rows without scanning the full FTS table.
In `@src/features/workspaces/search/workspace-search-scope.ts`:
- Around line 134-158: The batchFolderFilters function should not throw when a
single folder identifier exceeds the Vectorize filter limit, since
resolveWorkspaceSearchScope does not handle that exception. Skip the oversized
identifier and continue processing subsequent folder IDs, while preserving
normal batching and filter generation for valid identifiers.
In `@wrangler.jsonc`:
- Around line 37-44: Document the out-of-band Vectorize provisioning for both
indexes configured in the vectorize section, including exact create-index and
create-metadata-index commands with the embedding model’s required dimensions,
distance metric, and itemId, parentId, and type fields. Add a reference to that
deployment document beside the wrangler configuration comment so provisioning
requirements remain discoverable.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: c3b89fda-a7c6-494f-b9ae-a9fcedb4ff44
📒 Files selected for processing (33)
src/features/workspaces/ai/ai-tool-registry.tssrc/features/workspaces/ai/workspace-citations.tssrc/features/workspaces/ai/workspace-tool-result-adapters.tssrc/features/workspaces/ai/workspace-tools.tssrc/features/workspaces/extraction/request-workspace-file-extraction.tssrc/features/workspaces/extraction/workspace-file-extraction-reconciler.tssrc/features/workspaces/extraction/workspace-file-extraction-workflow-id.tssrc/features/workspaces/extraction/workspace-page-projection.tssrc/features/workspaces/extraction/workspace-projection-readiness.tssrc/features/workspaces/kernel/workspace-kernel-access.tssrc/features/workspaces/kernel/workspace-kernel-events.tssrc/features/workspaces/kernel/workspace-kernel-item-commands.tssrc/features/workspaces/kernel/workspace-kernel.tssrc/features/workspaces/operations/search-workspace.tssrc/features/workspaces/operations/workspace-operation-observability.tssrc/features/workspaces/operations/workspace-tool-definitions.tssrc/features/workspaces/operations/workspace-tool-schemas.tssrc/features/workspaces/search/workspace-search-batches.tssrc/features/workspaces/search/workspace-search-chunks.tssrc/features/workspaces/search/workspace-search-content.tssrc/features/workspaces/search/workspace-search-contract.tssrc/features/workspaces/search/workspace-search-embeddings.tssrc/features/workspaces/search/workspace-search-indexer.tssrc/features/workspaces/search/workspace-search-projection.tssrc/features/workspaces/search/workspace-search-query.tssrc/features/workspaces/search/workspace-search-ranking.tssrc/features/workspaces/search/workspace-search-references.tssrc/features/workspaces/search/workspace-search-schema.tssrc/features/workspaces/search/workspace-search-scope.tssrc/features/workspaces/search/workspace-search-version.tssrc/features/workspaces/search/workspace-search.test.tsworker-configuration.d.tswrangler.jsonc
| async processWorkspaceSearchIndex() { | ||
| if (await this.search.processBatch()) { | ||
| // The current one-shot schedule is removed after this callback returns, | ||
| // so its successor must not deduplicate onto the executing row. | ||
| await this.scheduleWorkspaceSearchIndexing(false); | ||
| } | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect the indexer batch claiming logic and pending-state transitions.
fd -t f 'workspace-search-indexer.ts' | xargs -r ast-grep outline --items all
fd -t f 'workspace-search-indexer.ts' | while IFS= read -r file; do
rg -n -C 6 'processBatch|markPending|claim|UPDATE|DELETE FROM|status' "$file"
doneRepository: ThinkEx-OSS/thinkex
Length of output: 7464
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workspace-kernel outline =="
fd -t f 'workspace-kernel.ts' | xargs -r ast-grep outline --items all
echo "== relevant workspace-kernel sections =="
file="$(fd -t f 'workspace-kernel.ts' | head -n1)"
sed -n '1,130p' "$file"
sed -n '300,345p' "$file"
echo "== related methods in workspace-search-indexer.ts =="
indexer="$(fd -t f 'workspace-search-indexer.ts' | head -n1)"
sed -n '120,200p' "$indexer"
sed -n '200,300p' "$indexer"
sed -n '470,490p' "$indexer"
echo "== all processBatch implementations =="
rg -n -C 8 'async processBatch\(|processBatch\(' --glob '*.ts'Repository: ThinkEx-OSS/thinkex
Length of output: 14189
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== WorkspaceSearchProjection relevant methods =="
proj="$(fd -t f 'workspace-search-projection.ts' | head -n1)"
sed -n '1,240p' "$proj"
echo "== scheduleWorkspaceSearchIndexing references =="
rg -n -C 4 'scheduleWorkspaceSearchIndexing|requestRun|searchItems|hasPending|processBatch' src/features/workspaces/search src/features/workspaces/kernel --glob '*.ts'Repository: ThinkEx-OSS/thinkex
Length of output: 13423
Claim pending items atomically before reading batches.
processBatch selects from kernel_search_pending, then indexes each item in parallel. If another due row calls processBatch before clearing requested_at, the same source can be embedded and upserted twice. Claim the selected rows in one atomic write before indexing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/features/workspaces/kernel/workspace-kernel.ts` around lines 326 - 332,
Update processWorkspaceSearchIndex and the underlying search.processBatch flow
to atomically claim selected kernel_search_pending rows before reading or
indexing their batches. Ensure concurrent due invocations cannot select the same
pending source while preserving the existing parallel indexing and scheduling
behavior.
| AND ( | ||
| i.type = 'document' | ||
| OR (i.type = 'file' AND p.source_hash IS NOT NULL) | ||
| ) | ||
| AND ( | ||
| s.item_id IS NULL | ||
| OR s.source_version != CASE | ||
| WHEN i.type = 'document' | ||
| THEN ${workspaceSearchIndexVersion} || ':document:' || i.updated_at | ||
| ELSE ${workspaceSearchIndexVersion} || ':file:' || i.updated_at || ':' || p.updated_at || ':' || p.source_hash | ||
| END | ||
| OR s.vector_status != 'ready' | ||
| ) | ||
| ON CONFLICT(item_id) DO NOTHING | ||
| `; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect the declared affinity of the timestamp columns used in the version formula.
fd -e ts -e sql . src/features/workspaces --exec rg -n -C3 'kernel_items *\(|kernel_item_projections *\(' {} \;
# Show the canonical TypeScript version builder for comparison.
fd 'workspace-search-version.ts' --exec cat -n {} \;Repository: ThinkEx-OSS/thinkex
Length of output: 2742
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== search indexer seedWorkspaceSearchSourceVersion context =="
fd 'workspace-search-indexer.ts' src/features/workspaces -x sh -c 'echo "--- $1"; rg -n -C8 "seedWorkspaceSearchSourceVersion|buildWorkspaceSearchSourceVersion|workspaceSearchSourceVersion|search_items|kernel_items|kernel_item_projections" "$1" || true' sh {}
echo "== schema definitions for updated_at =="
rg -n -C4 'kernel_item_projections.*updated_at|kernel_items.*updated_at|CREATE TABLE IF NOT EXISTS kernel_items|CREATE TABLE IF NOT EXISTS kernel_item_projections|updated_at TEXT|updated_at NUMBER|updated_at INTEGER|updated_at REAL' src/features/workspaces src || true
echo "== relevant SQL CREATE TABLE snippets =="
python3 - <<'PY'
from pathlib import Path
for p in Path('src/features/workspaces').rglob('*.ts'):
txt=p.read_text(errors='ignore')
if 'CREATE TABLE IF NOT EXISTS kernel_items' in txt or 'CREATE TABLE IF NOT EXISTS kernel_item_projections' in txt:
print(f'--- {p}')
lines=txt.splitlines()
for i,l in enumerate(lines,1):
if 'CREATE TABLE IF NOT EXISTS kernel_items' in l or 'CREATE TABLE IF NOT EXISTS kernel_item_projections' in l or 'updated_at' in l:
start=max(1,i-3); end=min(len(lines),i+8)
for n in range(start,end+1):
print(f'{n}: {lines[n-1]}')
PY
echo "== tests around buildWorkspaceSearchSourceVersion or workspaceSearchIndexVersion =="
rg -n -C3 "buildWorkspaceSearchSourceVersion|workspaceSearchIndexVersion|source_version" src test __tests__ tests .github -g '*.ts' -g '*.test.ts' -g '*.spec.ts' || true
echo "== SQLite text rendering simulation =="
python3 - <<'PY'
vals = [12, 12.0, 12.5, "12", "12.0", None]
for v in vals:
if isinstance(v, (int, float)) and v is not None:
s = str(int(v)) if (float(v)).is_integer() and isinstance(v, float) else str(v)
# Python's str for integers equals SQLite's text for INTEGER values in this representation
print(v, "to_str", s, "dot", isinstance(v,float) and not isinstance(v,bool))
PYRepository: ThinkEx-OSS/thinkex
Length of output: 30963
Verify the SQL freshness check against the canonical source-version formatter.
seedPendingItems rebuilds the exact buildWorkspaceSearchSourceVersion string in SQL. If this formula diverges from src/features/workspaces/search/workspace-search-version.ts, items can stay in kernel_search_pending indefinitely or skip reindexing. Add coverage comparing the SQL CASE output to buildWorkspaceSearchSourceVersion for document and file rows.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/features/workspaces/search/workspace-search-indexer.ts` around lines 84 -
98, The SQL freshness CASE in seedPendingItems must remain identical to
buildWorkspaceSearchSourceVersion in workspace-search-version.ts. Add coverage
for both document and file rows that computes the SQL CASE output and compares
it with buildWorkspaceSearchSourceVersion, including the file-specific
updated_at and source_hash components, so version changes cannot cause pending
items to be skipped or retained indefinitely.
| const failures: unknown[] = []; | ||
| for (const row of pending) { | ||
| try { | ||
| await this.indexItem(row.item_id); | ||
| } catch (error) { | ||
| this.recordIndexFailure(row.item_id, error); | ||
| if (!this.markIndexAttemptFailed(row.item_id)) { | ||
| failures.push(error); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| await this.flushVectorDeletes(); | ||
| if (failures.length > 0) { | ||
| throw new AggregateError(failures, "Workspace search indexing failed."); | ||
| } | ||
| return this.hasPending(); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find the search-projection run loop and its error handling in the kernel.
fd 'workspace-kernel.ts' --exec rg -n -C10 'processBatch|hasPending|searchProjection|alarm' {} \;Repository: ThinkEx-OSS/thinkex
Length of output: 1874
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect WorkspaceKernel search-projection scheduling and handler boundaries.
fd 'workspace-kernel.ts' -x sh -c '
file="$1"
echo "== $file =="
wc -l "$file"
echo "--- relevant sections ---"
rg -n -C 16 "scheduleWorkspaceSearchIndexing|processWorkspaceSearchIndex|processBatch|search.*alarm|alarm" "$file"
echo "--- line numbers for scheduling symbols ---"
rg -n "scheduleWorkspaceSearchIndexing|processWorkspaceSearchIndex" "$file"
' sh {}
echo "== target file relevant sections =="
fd 'workspace-search-indexer.ts' -x sh -c '
file="$1"
echo "== $file =="
wc -l "$file"
rg -n -C 8 "processBatch|recordIndexFailure|markIndexAttemptFailed|hasPending|select.*pending|pending.*select" "$file"
' sh {}Repository: ThinkEx-OSS/thinkex
Length of output: 8177
Reschedule after processBatch rejects.
processBatch catches an indexing exception, increments attempts, then throws AggregateError. processWorkspaceSearchIndex() only schedules the next batch when the return value is truthy, so a retryable failure leaves the pending row in kernel_search_pending without consuming another scheduled attempt. Catch the projection rejection and call processWorkspaceSearchIndex() for the next batch while returning false for exhausted attempts.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/features/workspaces/search/workspace-search-indexer.ts` around lines 139
- 156, Update processWorkspaceSearchIndex to catch processBatch rejections,
invoke processWorkspaceSearchIndex again to reschedule retryable pending work,
and return false when attempts are exhausted. Preserve the existing successful
truthy-return scheduling behavior and ensure AggregateError failures do not
leave retryable rows unscheduled.
| function createWorkspaceSearchVectorFilters( | ||
| input: | ||
| | { kind: "workspace"; types: WorkspaceSearchItemType[] } | ||
| | { itemId: string; kind: "item" } | ||
| | { folderIds: string[]; kind: "folder"; types: WorkspaceSearchItemType[] }, | ||
| ): WorkspaceSearchVectorFilter[] { | ||
| if (input.kind === "item") { | ||
| return [{ itemId: input.itemId }]; | ||
| } | ||
| const type = input.types.length === 1 ? input.types[0] : undefined; | ||
| if (input.kind === "workspace") { | ||
| return type ? [{ type }] : [null]; | ||
| } | ||
|
|
||
| return batchFolderFilters(input.folderIds, (folderIds) => { | ||
| const filter: VectorizeVectorMetadataFilter = { parentId: { $in: folderIds } }; | ||
| if (type) { | ||
| filter.type = type; | ||
| } | ||
| return filter; | ||
| }); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Cloudflare Vectorize metadata filtering $in operator support and metadata index requirement
💡 Result:
Cloudflare Vectorize provides robust support for the $in operator within metadata filters [1][2]. Operator Support The $in operator allows you to filter for documents where a specified metadata field matches any value within a provided array [1][2]. Its counterpart, the $nin (not in) operator, is also supported [1][2]. These operators accept arrays containing string, number, boolean, or null values [1][2]. Metadata Index Requirements To use metadata filtering (including the $in operator), you must explicitly configure metadata indexes for your Vectorize index [1][3]. Key requirements include: 1. Creation Requirement: You must create metadata indexes for specific properties before inserting the vectors you intend to filter by [1][2][4]. Vectors inserted prior to the creation of a metadata index will not be filterable using that property [4]. 2. Index Limits: You can create up to 10 metadata indexes per Vectorize index [5][4]. 3. Data Types: Metadata indexes support three data types: string, number, and boolean [3][4]. 4. Compatibility: Metadata filtering is only available on Vectorize indexes created on or after December 6, 2023 [1][6]. Legacy indexes cannot be migrated to support this feature [1][6]. You can manage these indexes using Wrangler (e.g., npx wrangler vectorize create-metadata-index --property-name= --type=) or the Cloudflare API [5][3][7]. It is recommended to create these indexes immediately after initializing your Vectorize index to ensure all subsequent data is properly indexed for filtering [3].
Citations:
- 1: https://developers.cloudflare.com/vectorize/reference/metadata-filtering/
- 2: https://developers.cloudflare.com/vectorize/reference/metadata-filtering/index.md
- 3: https://developers.cloudflare.com/vectorize/get-started/intro/index.md
- 4: https://github.com/cloudflare/cloudflare-docs/blob/production/src/content/docs/vectorize/reference/metadata-filtering.mdx
- 5: https://developers.cloudflare.com/api/resources/vectorize/subresources/indexes/subresources/metadata_index/methods/create/
- 6: https://developers.cloudflare.com/vectorize/platform/changelog/
- 7: https://developers.cloudflare.com/vectorize/reference/client-api/index.md
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository files matching workspace-search-scope and Vectorize config:"
fd -i 'workspace-search-scope|wrangler|vectorize|worker' . | sed -n '1,120p'
echo
echo "Search for createWorkspaceSearchVectorFilters and Vectorize metadata usage:"
rg -n "createWorkspaceSearchVectorFilters|batchFolderFilters|Vectorize|VectorizeVectorMetadataFilter|metadataIndex|metadata index|create-metadata-index|property-name|parentId|type" . \
--glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' | sed -n '1,240p'
echo
echo "Target file excerpt:"
fd -a 'workspace-search-scope.ts' . | while read -r f; do
echo "--- ${f#$(pwd)/} ---"
nl -ba "$f" | sed -n '1,170p'
doneRepository: ThinkEx-OSS/thinkex
Length of output: 27158
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Workspace search scope excerpt:"
f="./src/features/workspaces/search/workspace-search-scope.ts"
python3 - <<'PY' "$f"
import sys
p=sys.argv[1]
lines=open(p).read().splitlines()
for i in range(max(0,111-30), min(len(lines),131+10)):
print(f"{i+1:4}\t{lines[i]}")
PY
echo
echo "Vectorize metadata index definitions in wrangler.jsonc:"
python3 - <<'PY'
import json
from pathlib import Path
p=Path("wrangler.jsonc")
text=p.read_text()
clean=text
for pat in [r"// .*"]:
pass
print(text)
PY
echo
echo "Focused wrangler Vectorize sections:"
python3 - <<'PY'
from pathlib import Path
lines=Path("wrangler.jsonc").read_text().splitlines()
for start,end in [(30,45),(95,125),(260,295),(400,430)]:
print(f"--- lines {start}-{end} ---")
for i in range(start-1,end):
if i < len(lines):
print(f"{i+1:4}\t{lines[i]}")
PYRepository: ThinkEx-OSS/thinkex
Length of output: 15544
🌐 Web query:
Cloudflare Vectorize Wrangler metadata indexes array indexing property string metadata-indexes
💡 Result:
Cloudflare Vectorize allows you to perform metadata filtering to narrow down search results by creating metadata indexes for specific properties [1][2]. You can create up to 10 metadata indexes per Vectorize index [1][3]. Metadata indexes are supported for three data types: string, number, and boolean [1][2]. When using the "string" type, Vectorize indexes the first 64 bytes of the string data, truncating it on UTF-8 character boundaries to the longest well-formed substring within that limit [1][2]. To manage these indexes using Wrangler (version 3.71.0 or later), you can use the following commands [4]: 1. Create a metadata index: wrangler vectorize create-metadata-index --property-name= --type=<string|number|boolean> [4][5] 2. List metadata indexes: wrangler vectorize list-metadata-index [4][5] 3. Delete a metadata index: wrangler vectorize delete-metadata-index --property-name= [4] Metadata indexes must be created before inserting vectors to enable filtering for those specific fields [1][2]. Once indexed, you can apply filters during a query using the filter property in the query method [1][3][6]: // Example query with metadata filter let matches = await env.YOUR_INDEX.query(queryVector, { topK: 3, filter: { streaming_platform: "netflix" } }); There is no native concept of an "array indexing" property in the metadata index configuration itself; metadata indexes are defined per property name and type [7][8][9]. For more complex structures, you generally store and filter on the primitive types supported [1][3].
Citations:
- 1: https://developers.cloudflare.com/vectorize/reference/metadata-filtering/
- 2: https://developers.cloudflare.com/vectorize/get-started/intro/index.md
- 3: https://developers.cloudflare.com/vectorize/reference/metadata-filtering/index.md
- 4: https://developers.cloudflare.com/vectorize/reference/client-api/index.md
- 5: https://developers.cloudflare.com/vectorize/reference/wrangler-commands/index.md
- 6: https://developers.cloudflare.com/api/resources/vectorize/subresources/indexes/methods/query/
- 7: https://developers.cloudflare.com/api/resources/vectorize/subresources/indexes/subresources/metadata_index/
- 8: https://developers.cloudflare.com/api/resources/vectorize/subresources/indexes/subresources/metadata_index/methods/create/
- 9: https://github.com/cloudflare/workers-sdk/blob/e643b19d/packages/wrangler/src/vectorize/createMetadataIndex.ts
🌐 Web query:
Cloudflare Vectorize Wrangler configuration metadata-indexes property-name type
💡 Result:
Cloudflare Vectorize metadata indexes are managed via the Wrangler CLI command wrangler vectorize create-metadata-index, rather than through static configuration in the wrangler.toml file [1][2][3]. While wrangler.toml is used to define the Vectorize index bindings for your Workers project [1][4][5], metadata indexes are created and managed as part of the index's persistent configuration [2][3]. When creating a metadata index, you must specify the following parameters: 1. property-name: The name of the metadata field you wish to index [1][6][7]. 2. type: The data type of the metadata property [6][8]. Supported types are 'string', 'number', and 'boolean' [2][6][9]. Example usage: npx wrangler vectorize create-metadata-index --property-name='your-property' --type='string' Key limitations and details include: - You can create up to 10 metadata indexes per Vectorize index [2][9][3]. - Metadata indexes must be created before vectors are inserted into the index [2][3]. - For type 'number', the indexed precision is float64 [2][3]. - For type 'string', the first 64 bytes of the string data are indexed [2][3].
Citations:
- 1: https://developers.cloudflare.com/vectorize/reference/client-api/index.md
- 2: https://developers.cloudflare.com/vectorize/reference/metadata-filtering/
- 3: https://developers.cloudflare.com/vectorize/get-started/intro/index.md
- 4: https://developers.cloudflare.com/pages/functions/wrangler-configuration/
- 5: https://developers.cloudflare.com/vectorize/get-started/embeddings/
- 6: https://github.com/cloudflare/workers-sdk/blob/0277bfa1/packages/wrangler/src/vectorize/createMetadataIndex.ts
- 7: https://developers.cloudflare.com/api/resources/vectorize/subresources/indexes/subresources/metadata_index/
- 8: https://developers.cloudflare.com/api/resources/vectorize/subresources/indexes/subresources/metadata_index/methods/create/
- 9: https://github.com/cloudflare/cloudflare-docs/blob/production/src/content/docs/vectorize/reference/metadata-filtering.mdx
Ensure Vectorize metadata indexes exist for parentId and type.
wrangler.jsonc only binds the indexes and documents the requirement, but Wrangler config does not create the metadata indexes. Add provisioning for string metadata indexes on thinkex-workspace-search-staging and thinkex-workspace-search, so folder-scoped $in filtering and type-based workspace filtering actually narrow results instead of returning no matches.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/features/workspaces/search/workspace-search-scope.ts` around lines 111 -
131, The Vectorize indexes required by createWorkspaceSearchVectorFilters are
not being provisioned. Add string metadata index creation for parentId and type
on both thinkex-workspace-search-staging and thinkex-workspace-search, while
preserving the existing bindings and configuration.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 33 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/features/workspaces/search/workspace-search-indexer.ts">
<violation number="1" location="src/features/workspaces/search/workspace-search-indexer.ts:255">
P2: Rapid document saves within one millisecond can leave newer content permanently unindexed. Use a content revision/hash that changes per checkpoint in `sourceVersion`, rather than `updated_at` alone.</violation>
</file>
<file name="src/features/workspaces/search/workspace-search-references.ts">
<violation number="1" location="src/features/workspaces/search/workspace-search-references.ts:11">
P3: Search citations have no regression coverage for item-level versus PDF-page reference mapping or for stripping `itemId` from model output. Add focused tests for both branches so citation targets remain aligned with search locations.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| shellPath: row.shell_path, | ||
| sourceVersion: buildWorkspaceSearchSourceVersion({ | ||
| type: "document", | ||
| updatedAt: row.updated_at, |
There was a problem hiding this comment.
P2: Rapid document saves within one millisecond can leave newer content permanently unindexed. Use a content revision/hash that changes per checkpoint in sourceVersion, rather than updated_at alone.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/features/workspaces/search/workspace-search-indexer.ts, line 255:
<comment>Rapid document saves within one millisecond can leave newer content permanently unindexed. Use a content revision/hash that changes per checkpoint in `sourceVersion`, rather than `updated_at` alone.</comment>
<file context>
@@ -0,0 +1,502 @@
+ shellPath: row.shell_path,
+ sourceVersion: buildWorkspaceSearchSourceVersion({
+ type: "document",
+ updatedAt: row.updated_at,
+ }),
+ type: "document",
</file context>
| WorkspaceSearchResult, | ||
| } from "#/features/workspaces/search/workspace-search-contract"; | ||
|
|
||
| export function createWorkspaceSearchReferences(results: readonly WorkspaceSearchResult[]) { |
There was a problem hiding this comment.
P3: Search citations have no regression coverage for item-level versus PDF-page reference mapping or for stripping itemId from model output. Add focused tests for both branches so citation targets remain aligned with search locations.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/features/workspaces/search/workspace-search-references.ts, line 11:
<comment>Search citations have no regression coverage for item-level versus PDF-page reference mapping or for stripping `itemId` from model output. Add focused tests for both branches so citation targets remain aligned with search locations.</comment>
<file context>
@@ -0,0 +1,52 @@
+ WorkspaceSearchResult,
+} from "#/features/workspaces/search/workspace-search-contract";
+
+export function createWorkspaceSearchReferences(results: readonly WorkspaceSearchResult[]) {
+ return createWorkspaceReferenceRecords(results.map(getSearchResultLocation));
+}
</file context>
Persist cleanup progress across partial Vectorize failures and bound background deletion retries. Remove abandoned partial chunks and retry failed workspace purges without erasing their recovery inventory. Keep document source revisions monotonic so indexing cannot accept stale checkpoint content.
Cap semantic calls for very large folder scopes and report reduced semantic coverage as partial. Keep stale or incomplete chunks out of results, preserve type filters for repeated values, center excerpts on any matching term, and cover chunk bounds plus model-facing references.
Build deterministic repair jobs concurrently, then submit every eligible file in service-sized batches. This prevents workspaces with more than 100 broken or missing projections from waiting for another connection to continue healing.
Document the required BGE-M3 dimensions, cosine metric, and metadata index commands for staging and production so scoped semantic search infrastructure is reproducible from the repository.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
Use Zod overwrite so duplicate type values collapse at the tool boundary while remaining representable in the MCP JSON Schema.
There was a problem hiding this comment.
All reported issues were addressed across 15 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
♻️ Duplicate comments (1)
src/features/workspaces/kernel/workspace-kernel.ts (1)
329-335: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winClaim pending items atomically before reading batches.
processBatchselects fromkernel_search_pending, then indexes each item. If another due invocation callsprocessBatchbefore the previous run clearsrequested_at, the same source can be embedded and upserted twice. Claim the selected rows in one atomic write before indexing them.This was flagged in a previous review on this method and does not appear resolved in the current diff (the indexer file that contains
processBatchis not part of this review batch, so the fix status cannot be re-verified here).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/kernel/workspace-kernel.ts` around lines 329 - 335, Update the search indexer’s processBatch flow to atomically claim the selected kernel_search_pending rows before reading or indexing their batches, ensuring concurrent invocations cannot process the same source twice. Preserve processWorkspaceSearchIndex’s scheduling behavior and apply the claim within the processBatch implementation or its existing data-access helper.
🧹 Nitpick comments (2)
src/features/workspaces/kernel/workspace-kernel.ts (2)
423-439: 🧹 Nitpick | 🔵 TrivialConsider a distinct signal when purge retries are exhausted.
After
workspacePurgeMaximumAttempts(5) failed attempts, spread across roughly 35 seconds total,purgeForDeletionstops retrying and keeps local storage intact for a later manual retry. Each attempt already reports failures per-resource viarecordOperationalFailure, but there is no distinct event marking that a workspace has exhausted all automated retries and now requires manual intervention.Add an operational event (or alert) specifically for the exhausted-retry case so orphaned workspace resources are surfaced instead of being visible only as a stream of individual per-attempt failure events.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/kernel/workspace-kernel.ts` around lines 423 - 439, Add a distinct operational event or alert in purgeForDeletion when failed remains nonzero and attempt has reached workspacePurgeMaximumAttempts, indicating that automated retries are exhausted and manual intervention is required. Keep the existing retry scheduling for attempts below the maximum and preserve local storage for failed purges.
403-421: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConsider
Promise.allSettledinstead ofPromise.allfor the R2 prefix purge.
Promise.allrejects as soon as onedeleteR2Prefixcall fails. The other four in-flight deletions are not awaited after that point, so they can be abandoned mid-flight if the isolate suspends before they finish.deleteR2Prefixis idempotent, so a retry is safe, but the in-flight work already done in this cycle is discarded, and thefailedcounter cannot reflect how many of the five prefixes actually failed.Use
Promise.allSettledto let all five prefix deletions complete before deciding on retry, and to report failures more precisely.♻️ Proposed refactor
try { - await Promise.all([ + const results = await Promise.allSettled([ deleteR2Prefix( this.env.WORKSPACE_KERNEL_FILES, getChatAttachmentWorkspacePrefix(workspaceId), ), deleteR2Prefix(this.env.WORKSPACE_KERNEL_FILES, `uploads/workspaces/${workspaceId}/`), deleteR2Prefix(this.env.WORKSPACE_KERNEL_FILES, `workspace_kernel_files/${workspaceId}/`), deleteR2Prefix(this.env.WORKSPACE_KERNEL_FILES, `workspace_file_objects/${workspaceId}/`), deleteR2Prefix(this.env.WORKSPACE_KERNEL_FILES, `workspace_file_uploads/${workspaceId}/`), ]); - } catch (error) { - failed += 1; - recordOperationalFailure({ - error, - event: "workspace_r2_purge", - fields: { workspace_id: workspaceId }, - }); - } + const rejected = results.filter( + (result): result is PromiseRejectedResult => result.status === "rejected", + ); + if (rejected.length > 0) { + failed += 1; + for (const result of rejected) { + recordOperationalFailure({ + error: result.reason, + event: "workspace_r2_purge", + fields: { workspace_id: workspaceId }, + }); + } + } + } catch (error) { + failed += 1; + recordOperationalFailure({ + error, + event: "workspace_r2_purge", + fields: { workspace_id: workspaceId }, + }); + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/features/workspaces/kernel/workspace-kernel.ts` around lines 403 - 421, Update the R2 purge block in the workspace deletion method to use Promise.allSettled for all five deleteR2Prefix calls, ensuring every deletion is awaited before handling results. Inspect each rejected result, increment failed for each unsuccessful prefix, and call recordOperationalFailure with the corresponding error and workspace_id so retry accounting reflects individual failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@src/features/workspaces/kernel/workspace-kernel.ts`:
- Around line 329-335: Update the search indexer’s processBatch flow to
atomically claim the selected kernel_search_pending rows before reading or
indexing their batches, ensuring concurrent invocations cannot process the same
source twice. Preserve processWorkspaceSearchIndex’s scheduling behavior and
apply the claim within the processBatch implementation or its existing
data-access helper.
---
Nitpick comments:
In `@src/features/workspaces/kernel/workspace-kernel.ts`:
- Around line 423-439: Add a distinct operational event or alert in
purgeForDeletion when failed remains nonzero and attempt has reached
workspacePurgeMaximumAttempts, indicating that automated retries are exhausted
and manual intervention is required. Keep the existing retry scheduling for
attempts below the maximum and preserve local storage for failed purges.
- Around line 403-421: Update the R2 purge block in the workspace deletion
method to use Promise.allSettled for all five deleteR2Prefix calls, ensuring
every deletion is awaited before handling results. Inspect each rejected result,
increment failed for each unsuccessful prefix, and call recordOperationalFailure
with the corresponding error and workspace_id so retry accounting reflects
individual failures.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 04b08d99-0dd4-40b2-bda1-388fbd3ccd17
📒 Files selected for processing (15)
docs/configuration/deployments.mdxsrc/features/workspaces/extraction/workspace-file-extraction-reconciler.tssrc/features/workspaces/kernel/workspace-kernel-item-commands.tssrc/features/workspaces/kernel/workspace-kernel.tssrc/features/workspaces/operations/workspace-tool-definitions.tssrc/features/workspaces/search/workspace-search-chunks.tssrc/features/workspaces/search/workspace-search-contract.tssrc/features/workspaces/search/workspace-search-indexer.tssrc/features/workspaces/search/workspace-search-projection.tssrc/features/workspaces/search/workspace-search-query.tssrc/features/workspaces/search/workspace-search-references.test.tssrc/features/workspaces/search/workspace-search-schema.tssrc/features/workspaces/search/workspace-search-version.tssrc/features/workspaces/search/workspace-search.test.tswrangler.jsonc
🚧 Files skipped from review as they are similar to previous changes (11)
- wrangler.jsonc
- src/features/workspaces/search/workspace-search-schema.ts
- src/features/workspaces/kernel/workspace-kernel-item-commands.ts
- src/features/workspaces/search/workspace-search-version.ts
- src/features/workspaces/operations/workspace-tool-definitions.ts
- src/features/workspaces/search/workspace-search-chunks.ts
- src/features/workspaces/search/workspace-search-projection.ts
- src/features/workspaces/search/workspace-search-contract.ts
- src/features/workspaces/search/workspace-search-indexer.ts
- src/features/workspaces/extraction/workspace-file-extraction-reconciler.ts
- src/features/workspaces/search/workspace-search-query.ts
There was a problem hiding this comment.
0 issues found across 4 files (changes from recent commits).
Requires human review: Auto-approval blocked by 4 unresolved issues from previous reviews.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Keep the complete local text index after semantic retries are exhausted. Report semantic coverage as partial, and stop requeueing permanently failed vector projections on every object wake. Queue superseded vectors with one SQL write instead of one write per chunk.
Hash and submit extraction repairs in service-sized batches, and key retries to the broken projection revision. Trigger healing from active searches as well as workspace opens. Use Agent.destroy for successful cleanup and await every R2 purge before deciding whether to retry.
Delete the obsolete document timestamp helper and stop exporting search-only implementation types and adapters. Keep the public workspace search contract unchanged.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| if (!object) { | ||
| throw new Error(`Extracted page ${pageNumber} was not found.`); | ||
| } | ||
| const object = await getWorkspacePageProjectionObject({ |
There was a problem hiding this comment.
React Doctor · react-doctor/async-await-in-loop (warning)
This makes the for…of loop slow because each await runs one after another, so collect the independent calls & run them together with await Promise.all(items.map(...))
Fix → Collect the items, then use await Promise.all(items.map(...)) so independent work runs at the same time
| const prefix = getManifestPrefix(input.manifestObjectKey); | ||
|
|
||
| for (let pageNumber = 1; pageNumber <= manifest.pageCount; pageNumber += 1) { | ||
| const object = await getWorkspacePageProjectionObject({ |
There was a problem hiding this comment.
React Doctor · react-doctor/async-await-in-loop (warning)
This makes the for-loop slow because each await runs one after another, so collect the independent calls & run them together with await Promise.all(items.map(...))
Fix → Collect the items, then use await Promise.all(items.map(...)) so independent work runs at the same time
| export async function embedWorkspaceSearchTexts(ai: Ai, texts: string[]) { | ||
| const embeddings: number[][] = []; | ||
| for (const batch of batchWorkspaceSearchValues(texts, embeddingBatchSize)) { | ||
| const output: unknown = await ai.run(workspaceSearchEmbeddingModel, { |
There was a problem hiding this comment.
React Doctor · react-doctor/async-await-in-loop (warning)
This makes the for…of loop slow because each await runs one after another, so collect the independent calls & run them together with await Promise.all(items.map(...))
Fix → Collect the items, then use await Promise.all(items.map(...)) so independent work runs at the same time
| const failures: unknown[] = []; | ||
| for (const row of pending) { | ||
| try { | ||
| await this.indexItem(row.item_id); |
There was a problem hiding this comment.
React Doctor · react-doctor/async-await-in-loop (warning)
This makes the for…of loop slow because each await runs one after another, so collect the independent calls & run them together with await Promise.all(items.map(...))
Fix → Collect the items, then use await Promise.all(items.map(...)) so independent work runs at the same time
| const failures: unknown[] = []; | ||
| for (const batch of batchWorkspaceSearchValues(Array.from(ids), vectorDeleteBatchSize)) { | ||
| try { | ||
| await this.vectorize.deleteByIds(batch); |
There was a problem hiding this comment.
React Doctor · react-doctor/async-await-in-loop (warning)
This makes the for…of loop slow because each await runs one after another, so collect the independent calls & run them together with await Promise.all(items.map(...))
Fix → Collect the items, then use await Promise.all(items.map(...)) so independent work runs at the same time
There was a problem hiding this comment.
1 issue found across 10 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/features/workspaces/search/workspace-search-indexer.ts">
<violation number="1" location="src/features/workspaces/search/workspace-search-indexer.ts:219">
P2: Terminal indexing failures now leave vectors from earlier successful batches orphaned indefinitely. Keep the local chunks for keyword fallback, but queue their vector IDs for deletion when the item becomes `failed`; otherwise they consume Vectorize capacity and can crowd semantic top-K matches without being eligible to return.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| return; | ||
| } | ||
| } catch (error) { | ||
| indexError = error; |
There was a problem hiding this comment.
P2: Terminal indexing failures now leave vectors from earlier successful batches orphaned indefinitely. Keep the local chunks for keyword fallback, but queue their vector IDs for deletion when the item becomes failed; otherwise they consume Vectorize capacity and can crowd semantic top-K matches without being eligible to return.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/features/workspaces/search/workspace-search-indexer.ts, line 219:
<comment>Terminal indexing failures now leave vectors from earlier successful batches orphaned indefinitely. Keep the local chunks for keyword fallback, but queue their vector IDs for deletion when the item becomes `failed`; otherwise they consume Vectorize capacity and can crowd semantic top-K matches without being eligible to return.</comment>
<file context>
@@ -210,19 +210,34 @@ export class WorkspaceSearchIndexer {
+ return;
+ }
+ } catch (error) {
+ indexError = error;
+ }
}
</file context>
1 participant
Summary
workspace_searchtool with keyword and semantic retrieval.Why
Workspace tools could list and read known items but could not efficiently discover relevant content across documents and extracted files. Large content also needed bounded streaming rather than silent per-item cutoffs, and failed historical file projections needed a safe recovery path before they could participate in search.
Changes
@cf/baai/bge-m3embeddings and Vectorize semantic matches, with keyword-only fallback when semantic retrieval is unavailable.itemId,parentId, andtypemetadata; resolve folder membership from the live workspace tree so folder moves do not require descendant reindexing.ready,indexing,partial), projection error isolation, and cleanup for deleted/stale vectors.itemId,parentId, andtype. These are already provisioned on the staging and production indexes.Testing
pnpm checkpnpm test— 206 tests passedpnpm test:workers— 30 tests passedgit diff --check origin/main...HEADReview Notes
Start with
workspace-search-scope.ts, thenworkspace-search-indexer.tsandworkspace-search-query.ts. Search remains a derived projection; no canonical workspace content is moved into FTS5 or Vectorize. The branch intentionally excludes the unrelated persisted-image-attachment commit that existed on the original local development branch.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by CodeRabbit