feat: implement complete MCP search and retrieval API with hybrid search engine

[qowlgur121]

Summary of Changes

This PR implements the complete MCP Search & Retrieval API system with hybrid search capabilities and comprehensive Swagger documentation. It establishes the core search infrastructure that enables AI assistants to interact with OpenContext through the two-phase search strategy (exploratory search → focused retrieval).

Related Issues

• Resolves search API implementation requirements for MCP protocol integration
• Addresses UTF-8 encoding issues in Elasticsearch indexing
• Fixes file type hardcoding in document metadata

Testing Instructions

1. Search API Test: Test GET /api/v1/search?query=<your-query>&topK=5 for hybrid search functionality.
2. Content Retrieval Test: Test POST /api/v1/get-content with JSON body {"chunkId": "<chunk-id>", "maxTokens": 8000}.
3. UTF-8 Encoding Test: Verify Korean content is properly indexed and searchable in Elasticsearch.
4. File Type Resolution Test: Confirm metadata.fileType reflects actual uploaded file types (PDF/MARKDOWN).
5. Swagger Documentation Test: Access /swagger-ui.html and verify comprehensive API documentation.
6. File Upload UI Test: Ensure file upload shows proper file selection interface.
7. Error Handling Test: Verify proper error responses with detailed error codes.

Key Implementation Features

Search & Retrieval Engine

• SearchService: Hybrid search combining BM25 (keyword) + vector similarity with Korean Nori analyzer.
• ContentRetrievalService: Token-aware content retrieval with tiktoken-cl100k_base compatibility.
• MCP Protocol Integration: Complete find_knowledge and get_content tool implementation.

API Documentation & UX

• OpenApiConfig: Centralized Swagger configuration with proper API key authentication.
• DocsSearchController: Interface-based documentation following team patterns.
• Enhanced File Upload: Proper multipart form UI instead of text input.
• Comprehensive Examples: Real-world JSON examples for all endpoints.

Technical Improvements

• UTF-8 Encoding Fix: Resolved Korean character corruption in Elasticsearch bulk indexing.
• Type Safety Enhancement: Improved DTO binding with flexible chunkId handling (String vs UUID).
• Error Handling: Detailed error responses with specific error codes and messages.
• Code Deduplication: Resolved ChunkingService duplicate code compilation issues.

Data Flow

Search Flow

MCP Request  → SearchController → SearchService → Elasticsearch → SearchResults
     ↓             ↓                ↓               ↓               ↓
AI Assistant  JSON Validation  Hybrid Search    Vector+BM25     Relevance Ranking

Content Retrieval Flow

Content Request → ContentRetrievalService → Token Limiting → Final Content
       ↓                   ↓                     ↓               ↓
   Chunk ID           ES Document         tiktoken-cl100k    JSON Response

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Additional Notes

MCP Integration Ready

• Phase 1 Search: GET /api/v1/search returns ranked knowledge chunks with titles and snippets.
• Phase 2 Retrieval: POST /api/v1/get-content provides full content with token management.
• No Authentication: MCP endpoints are open for internal AI assistant access.
• Standard Response Format: All responses follow CommonResponse<T> pattern.

Bug Fixes & Improvements

• UTF-8 Encoding: Fixed Korean content corruption in Elasticsearch bulk API.
• File Type Resolution: Removed hardcoded "PDF", now dynamically resolved from database.
• Chunk ID Compatibility: Enhanced DTO binding to handle various chunk ID patterns.
• Documentation Quality: Team-standard Swagger documentation with real examples.

Breaking Changes

• SearchResultItem.chunkId changed from UUID to String type for better compatibility.
• New application.yml configuration parameters for search functionality.
• Additional database migration (V2) for enhanced document chunk schema.

Architecture Alignment

• Interface Pattern: Following team's DocsSourceController pattern for clean separation.
• Service Layer: Proper business logic encapsulation in SearchService and ContentRetrievalService.
• Error Handling: Consistent error responses with GlobalExceptionHandler integration.
• Security: Proper security annotations distinguishing admin vs public endpoints.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.