DL-11: Text Search Index Operations

[spuentesp]

Category: data-layer | Epic: 0 | Priority: low

Summary

Implement text search operations using OpenSearch. Supports indexing, searching,
and deleting text documents including snippets, facts, and narrative content.
Enables keyword and phrase search with filtering and highlighting.

Acceptance Criteria

• opensearch_index_document indexes text with metadata
• opensearch_index_document creates index if not exists
• opensearch_index_document supports update (upsert)
• opensearch_search performs keyword/phrase search
• opensearch_search supports field filters (universe_id, type)
• opensearch_search returns highlighted snippets
• opensearch_search supports pagination
• opensearch_delete_document removes document by ID
• opensearch_delete_by_query removes documents matching filter
• Indices are created with appropriate analyzers
• Unit tests achieve >= 80% coverage

Blocks

This use case blocks:

• Q-4
• I-2

Implementation

Layer: 1

Files to create:

• packages/data-layer/tests/test_tools/test_opensearch_tools.py
  Files to modify:
• packages/data-layer/src/monitor_data/tools/opensearch_tools.py
• packages/data-layer/src/monitor_data/db/opensearch.py

OPENSEARCH Operations:

• opensearch_index_document (authority: *)
• opensearch_search (authority: *)
• opensearch_delete_document (authority: *)
• opensearch_delete_by_query (authority: *)
• opensearch_get_document (authority: *)

Notes:

• OpenSearch provides full-text search capabilities
• Complements vector search for keyword queries
• Consider synonym expansion for domain terms
• Faceted search for filtering by type/universe

Testing Requirements

Minimum coverage: 80%

Unit tests:

• test_index_document: valid body → document indexed
• test_index_creates_index: new index → auto-created
• test_search_keyword: keyword query → matching docs
• test_search_phrase: phrase query → exact matches
• test_search_filter: universe filter → scoped results
• ... and 3 more

Integration tests:

• test_snippet_search: index snippets → search → verify results
• test_fact_search: index facts → keyword search → ranked results

References

Documentation:

• docs/architecture/DATABASE_INTEGRATION.md
  Code:
• packages/data-layer/src/monitor_data/db/opensearch.py

---

Generated from /home/sebas/monitor2/docs/use-cases/data-layer/DL-11.yml

[github-actions]

@copilot please implement this issue following the acceptance criteria and implementation details.