diff --git a/README.md b/README.md
index d11726c..c67b6d3 100644
--- a/README.md
+++ b/README.md
@@ -7,455 +7,117 @@
[](https://docs.docker.com/compose/)
[](https://modelcontextprotocol.io/)
-**Secure, Self-Hosted RAG System for Confidential Documents**
+**Self-Hosted RAG System for Secure Document Processing**
-An open-source RAG (Retrieval-Augmented Generation) system that eliminates the security risks of cloud-based AI tools while delivering enterprise-grade search capabilities for your most sensitive documents and proprietary code.
+OpenContext is an enterprise-grade RAG (Retrieval-Augmented Generation) system designed for organizations that require complete data sovereignty. Process confidential documents without external dependencies or security compromises.
-## The Problem We Solve
+## Why OpenContext?
-### The Security Dilemma of Modern AI Development
+**The Problem**: Developers need AI assistance but can't upload sensitive code or documents to external cloud services due to security, compliance, or intellectual property concerns.
-Developers today face an impossible choice between productivity and security:
+**The Solution**: OpenContext runs entirely within your infrastructure, ensuring your data never leaves your control while providing enterprise-grade search capabilities.
-```mermaid
-graph TD
- A[Developer with Confidential Documents] --> B{Choose Your Approach}
-
- B --> C[Upload to Cloud AI Services]
- B --> D[Keep Documents Private]
-
- C --> E[High Productivity]
- C --> F[Critical Security Risks
• Proprietary code exposure
• Third-party data mining
• Compliance violations
• Loss of data sovereignty]
-
- D --> G[Security Maintained]
- D --> H[Productivity Loss
• Manual document searches
• Knowledge fragmentation
• Repetitive research tasks]
-
- style F fill:#ffebee
- style H fill:#fff3e0
-```
+### Key Benefits
-### Critical Vulnerabilities in Current Solutions
+- **Zero External Dependencies**: All processing happens within your network
+- **Self-Hosted Security**: Keep sensitive data within your infrastructure
+- **Advanced Search**: Hybrid keyword + semantic search with Korean language support
+- **Production Ready**: Docker Compose deployment with monitoring and logging
+- **Developer Friendly**: MCP protocol integration for AI assistants like Cursor
-**Cloud AI Services (ChatGPT, Claude, etc.)**
-- **Data Harvesting**: Your proprietary code and internal documentation becomes training data for competitors
-- **Vendor Lock-in**: No control over data retention, deletion, or access policies
-- **Compliance Violations**: GDPR, HIPAA, SOX, and other regulations prohibit uploading sensitive data
-- **Industrial Espionage**: Foreign-hosted services may provide government access to your IP
-- **Supply Chain Attacks**: Third-party AI services can inject malicious responses or code
-
-**Third-Party MCP Tools (Context7, etc.)**
-- **Forced Data Upload**: Your private documents must be uploaded to their servers for processing
-- **Opaque Processing**: No visibility into how your data is stored, processed, or shared
-- **Malicious Code Injection**: External code suggestions may contain backdoors or vulnerabilities
-- **Service Dependencies**: Your development workflow becomes dependent on external infrastructure
-- **Cost Escalation**: Per-query pricing models make enterprise usage prohibitively expensive
-
-### Real-World Impact on Organizations
-
-**Financial Services Companies**
-```
-Problem: Trading algorithms and client data cannot be uploaded to external AI
-Impact: Developers manually search through extensive compliance documentation
-Challenge: Maintaining competitive advantage while ensuring regulatory compliance
-```
-
-**Healthcare Technology Organizations**
-```
-Problem: HIPAA regulations prevent using cloud AI with patient data documentation
-Impact: Medical protocol documentation scattered across multiple systems
-Challenge: Balancing research productivity with strict privacy requirements
-```
-
-**Defense and Government Contractors**
-```
-Problem: Classified documentation requires air-gapped environments
-Impact: Engineers rely on manual knowledge retrieval processes
-Challenge: Maintaining productivity in security-restricted environments
-```
-
-## Our Solution: Zero-Compromise Security with Maximum Productivity
-
-OpenContext delivers a RAG system designed to eliminate the false choice between security and productivity:
-
-```mermaid
-graph TD
- A[Your Confidential Documents] --> B[OpenContext Self-Hosted RAG]
-
- B --> C[Air-Gapped Processing]
- B --> D[Enterprise Search Engine]
- B --> E[MCP Protocol Integration]
-
- C --> F[Complete Data Sovereignty]
- D --> F
- E --> F
-
- C --> G[Maximum Productivity]
- D --> G
- E --> G
-
- F --> H[Security + Productivity
Without Compromise]
- G --> H
-
- style H fill:#e8f5e8
-```
-
-### Why Self-Hosting is Non-Negotiable for Sensitive Data
-
-**Complete Infrastructure Control**
-- All processing happens within your network perimeter
-- No external API calls or data transmission to third parties
-- Full audit trail of all data access and processing activities
-- Customizable security policies and access controls
-
-**Regulatory Compliance by Design**
-- GDPR Article 25: Data protection by design and by default
-- HIPAA Technical Safeguards: Access control and audit controls
-- SOX Section 404: Internal controls over financial reporting
-- ISO 27001: Information security management systems
-
-**Protection Against Supply Chain Attacks**
-- Open source codebase available for security auditing
-- No black-box AI services with unknown training data or behavior
-- Deterministic, controllable AI model execution
-- Protection against malicious code injection in AI responses
-
-### Key Innovation: Two-Phase Retrieval
-
-Unlike simple chatbots, OpenContext implements an intelligent two-phase approach:
-
-1. **Exploratory Search**: "Show me what you have about Spring Security"
-2. **Focused Retrieval**: "Give me the complete implementation details for JWT authentication"
-
-This mirrors how human experts actually work - first understanding the landscape, then diving deep into specifics.
-
-## Architecture Overview
-
-OpenContext implements a two-phase retrieval strategy optimized for interactive knowledge discovery:
+## Architecture
```mermaid
graph TB
subgraph "Client Layer"
- AdminUI[Admin Dashboard
React + TypeScript]
- AIAssistant[AI Assistant
MCP Protocol Client]
- end
-
- subgraph "Gateway Layer"
- MCPAdapter[MCP Adapter
Node.js Protocol Gateway]
+ UI[Admin Dashboard]
+ AI[AI Assistant]
end
subgraph "Application Layer"
- Core[OpenContext Core
Spring Boot 3.3]
- Pipeline[Document Pipeline
LangChain4j + Unstructured.io]
+ Core[OpenContext Core
Spring Boot]
+ MCP[MCP Adapter
Node.js]
end
subgraph "Data Layer"
- PostgreSQL[(PostgreSQL
Metadata Store)]
- Elasticsearch[(Elasticsearch
Search Index)]
- MinIO[(MinIO
Object Storage)]
- Ollama[Ollama
Embedding Service]
+ PG[(PostgreSQL)]
+ ES[(Elasticsearch)]
+ MinIO[(MinIO)]
+ Ollama[Ollama]
end
- AdminUI -->|REST API| Core
- AIAssistant -->|MCP Tools| MCPAdapter
- MCPAdapter -->|HTTP Proxy| Core
-
- Core --> Pipeline
- Pipeline --> PostgreSQL
- Pipeline --> Elasticsearch
- Pipeline --> MinIO
- Pipeline --> Ollama
+ UI --> Core
+ AI --> MCP
+ MCP --> Core
+ Core --> PG
+ Core --> ES
+ Core --> MinIO
+ Core --> Ollama
```
-### Component Responsibilities
-
-- **OpenContext Core**: Spring Boot application providing REST APIs and orchestrating document processing
-- **MCP Adapter**: Protocol gateway translating MCP tool calls to HTTP requests
-- **Admin Dashboard**: React-based web interface for document management and system monitoring
-- **Document Pipeline**: LangChain4j-based processing chain for parsing, chunking, and indexing
-- **PostgreSQL**: Relational database storing document metadata and chunk hierarchy
-- **Elasticsearch**: Search engine with hybrid BM25+vector capabilities and Korean language support
-- **MinIO**: S3-compatible object storage for original document files
-- **Ollama**: Local embedding model server running Qwen3-Embedding-0.6B
-
-## Why Organizations Choose OpenContext
-
-### Target Use Cases
-
-OpenContext is designed for organizations that require:
-
-**Financial Services**
-- Protection of proprietary trading algorithms and client data
-- Air-gapped development environments
-- Regulatory compliance (SOX, PCI-DSS, etc.)
-
-**Healthcare & Pharmaceuticals**
-- HIPAA-compliant document processing
-- Protection of research and patient data
-- Secure access to historical clinical trial information
-
-**Government & Defense**
-- Classified document processing capabilities
-- Offline operation in secure environments
-- Full audit trail and access control
-
-**Enterprise Technology**
-- Protection of intellectual property and trade secrets
-- Zero external dependencies for sensitive projects
-- Customizable security controls and deployment options
-
-## Key Features
-
-### Enterprise-Grade Document Processing
-- **Multi-format Support**: PDF and Markdown with preserving document structure
-- **Hierarchical Chunking**: Maintains context relationships across document sections
-- **Real-time Processing**: Live status tracking with detailed error reporting
-- **Smart Deduplication**: Content-based detection prevents redundant processing
-- **Batch Operations**: Efficient handling of large documentation sets
-
-### Advanced Search Intelligence
-- **Hybrid Search Engine**: Combines keyword precision with semantic understanding
-- **Two-Phase Retrieval**: Exploratory discovery → focused content extraction
-- **Multi-language Support**: Korean, Japanese, Chinese text analysis via Elasticsearch Nori
-- **Relevance Tuning**: Configurable ranking algorithms for domain-specific optimization
-- **Search Analytics**: Query performance monitoring and result quality metrics
-
-### Zero-Compromise Security Architecture
-- **Air-Gapped Deployment**: Complete network isolation with zero external dependencies
-- **Data Sovereignty**: Your data never leaves your infrastructure or legal jurisdiction
-- **Source Code Transparency**: Full open-source codebase available for security auditing
-- **Encrypted Everything**: TLS 1.3 for transport, AES-256 for storage, encrypted embeddings
-- **Comprehensive Audit Trail**: Every document access, search query, and system action logged
-- **Configurable Access Controls**: Role-based permissions with API key rotation
-- **Supply Chain Protection**: Verified build process with signed container images
-- **Compliance Ready**: Built-in support for GDPR, HIPAA, SOX, and ISO 27001 requirements
-
## Quick Start
### Prerequisites
-
```bash
-# System requirements
docker --version # 20.10+
docker compose version # 2.0+
-free -h # 4GB+ RAM recommended
-df -h # 10GB+ disk space required
+git --version # Any recent version
```
-### Installation
+**System Requirements:**
+- Minimum 4GB RAM, 10GB disk space recommended
+### Installation
```bash
-git clone https://github.com/your-org/open-context.git
+git clone https://github.com/OpenContextAI/open-context.git
cd open-context
-# Start all services
+# Start all services (model download included)
docker compose up -d
# Verify deployment
-docker compose ps
curl http://localhost:8080/actuator/health
```
-### Service Endpoints
-
-| Service | Port | Purpose |
-|---------|------|---------|
-| Core API | 8080 | Main application REST endpoints |
-| Admin UI | 3001 | Web dashboard for document management |
-| MCP Adapter | 3000 | Protocol gateway for AI assistant integration |
-| Elasticsearch | 9200 | Search engine and vector store |
-| PostgreSQL | 5432 | Relational database |
-| MinIO | 9000 | Object storage service |
-| Kibana | 5601 | Elasticsearch management interface |
-
-## API Documentation
-
-### MCP Protocol Tools
-
-OpenContext exposes two tools via the Model Context Protocol:
-
-#### find_knowledge
-Performs exploratory search returning relevant document chunks.
-
-**Request:**
-```http
-GET /api/v1/search?query=spring security jwt&topK=5
-```
-
-**Response:**
-```json
-{
- "success": true,
- "data": {
- "results": [
- {
- "chunkId": "uuid-string",
- "title": "JWT Authentication Configuration",
- "snippet": "Configure JWT authentication in Spring Security...",
- "relevanceScore": 0.94
- }
- ]
- }
-}
-```
-
-#### get_content
-Retrieves full content of a specific document chunk.
-
-**Request:**
-```http
-POST /api/v1/get-content
-Content-Type: application/json
-
-{
- "chunkId": "uuid-string",
- "maxTokens": 25000
-}
-```
-
-**Response:**
-```json
-{
- "success": true,
- "data": {
- "content": "Complete chunk content here...",
- "tokenInfo": {
- "tokenizer": "tiktoken-cl100k_base",
- "actualTokens": 1247
- }
- }
-}
-```
-
-### Administrative APIs
-
-Document management endpoints requiring API key authentication:
-
-```http
-# Upload document
-POST /api/v1/sources/upload
-X-API-KEY: your-api-key
-Content-Type: multipart/form-data
-
-# List documents with pagination
-GET /api/v1/sources?page=0&size=20&sort=createdAt,desc
-X-API-KEY: your-api-key
-
-# Trigger document reprocessing
-POST /api/v1/sources/{id}/resync
-X-API-KEY: your-api-key
-
-# Delete document and associated chunks
-DELETE /api/v1/sources/{id}
-X-API-KEY: your-api-key
-```
-
-## Technology Stack
-
-### Backend
-- **Java 21**: Latest LTS with virtual thread support
-- **Spring Boot 3.3.11**: Enterprise application framework
-- **LangChain4j 0.35.0**: RAG pipeline orchestration
-- **PostgreSQL 16**: Relational data persistence
-- **Elasticsearch 8.11**: Full-text search with vector capabilities
-- **QueryDSL 5.x**: Type-safe query construction
-- **Flyway**: Database schema migration management
-
-### Frontend
-- **React 19**: Modern component-based UI framework
-- **TypeScript**: Static type checking for enhanced development experience
-- **Vite 7**: Fast build tooling and development server
-- **Tailwind CSS 3**: Utility-first styling framework
-- **TanStack Query v5**: Server state management and caching
-- **Zustand**: Lightweight client state management
-
-### Infrastructure
-- **Docker Compose**: Multi-container application orchestration
-- **Ollama**: Local LLM serving with Qwen3-Embedding-0.6B model
-- **MinIO**: S3-compatible object storage
-- **Unstructured.io**: Document parsing and structure extraction
-
-## Configuration
-
-### API Key Management
-
-**Default API Key**: The system comes with a development API key `dev-api-key-123`. **Change this for production use.**
-
-**To change the API key:**
-
-1. **For Docker deployment**, set environment variable:
-```bash
-# In your shell or .env file
-export OPENCONTEXT_API_KEY="your-secure-production-key"
-
-# Generate a secure key
-openssl rand -hex 32
-```
-
-2. **For local development**, edit `application.yml`:
-```yaml
-opencontext:
- api:
- key: your-secure-production-key
-```
-
-**Using the API key:**
-```bash
-# All Admin API calls require the X-API-KEY header
-curl -H "X-API-KEY: your-secure-production-key" \
- http://localhost:8080/api/v1/sources
-```
+### Access Points
+| Service | URL | Purpose |
+|---------|-----|---------|
+| Admin UI | http://localhost:3001 | Document management dashboard |
+| Core API | http://localhost:8080 | REST API endpoints |
+| MCP Adapter | http://localhost:3000 | AI assistant integration |
+| API Docs | http://localhost:8080/swagger-ui.html | Interactive API documentation |
-### Embedding Model Configuration
+## Usage
-**Default Model**: `dengcao/Qwen3-Embedding-0.6B:F16` (1024 dimensions)
+### 1. Upload Documents
+Access the Admin UI at http://localhost:3001 and upload PDF or Markdown files. The system processes documents through multiple stages:
+- **PENDING**: File uploaded, waiting for processing
+- **PARSING**: Document structure extraction
+- **CHUNKING**: Text segmentation into meaningful chunks
+- **EMBEDDING**: Vector generation for semantic search
+- **INDEXING**: Search index creation
+- **COMPLETED**: Ready for search and retrieval
-**To change embedding models:**
+### 2. Efficient Search and Retrieve
+OpenContext optimizes AI assistant token usage through a two-phase approach:
-1. **Choose a compatible model from Ollama:**
+**Phase 1 - Explore (Low Token Cost):**
```bash
-# List available embedding models
-docker compose exec ollama ollama list
-
-# Pull a different model (example)
-docker compose exec ollama ollama pull nomic-embed-text
-docker compose exec ollama ollama pull mxbai-embed-large
-```
-
-2. **Update configuration** in `application-docker.yml`:
-```yaml
-app:
- ollama:
- embedding:
- model: nomic-embed-text # or your chosen model
+curl "http://localhost:8080/api/v1/search?query=spring%20security&topK=5"
```
+Returns lightweight chunk summaries for the AI to evaluate relevance.
-3. **Important**: Verify embedding dimensions match your Elasticsearch mapping:
+**Phase 2 - Focus (Targeted Token Usage):**
```bash
-# Check model info
-docker compose exec ollama ollama show nomic-embed-text
+curl -X POST http://localhost:8080/api/v1/get-content \
+ -H "Content-Type: application/json" \
+ -d '{"chunkId": "your-chunk-id", "maxTokens": 25000}'
```
+Retrieves full content only for selected chunks, minimizing unnecessary token consumption.
-**Supported embedding models:**
-- `nomic-embed-text` (768 dimensions) - English optimized
-- `mxbai-embed-large` (1024 dimensions) - Multilingual
-- `bge-large-en` (1024 dimensions) - High quality English
-- `bge-m3` (1024 dimensions) - Multilingual, includes Korean
+### 3. AI Assistant Integration
+Configure your AI assistant (Cursor, VSCode) with MCP:
-### MCP Integration Setup
-
-**For AI Assistants (Cursor, VSCode, etc.):**
-
-1. **Install the MCP server:**
-```bash
-# The MCP adapter runs as a Docker service
-docker compose up -d open-context-mcp-adapter
-```
-
-2. **Register with your AI assistant:**
-
-**MCP Configuration:**
-
-**For Cursor/VSCode:**
```json
{
"mcpServers": {
@@ -466,311 +128,114 @@ docker compose up -d open-context-mcp-adapter
}
```
-**For other MCP clients:**
-```json
-{
- "mcpServers": {
- "opencontext": {
- "type": "streamable-http",
- "url": "http://localhost:3000/mcp"
- }
- }
-}
-```
+## Technology Stack
-**Note:** The MCP server runs in HTTP mode and requires an MCP client for communication. Direct execution is not possible.
+**Backend**
+- Java 21 + Spring Boot 3.3.11
+- LangChain4j for RAG pipeline
+- PostgreSQL for metadata
+- Elasticsearch for search
+- Ollama for local embeddings
-3. **Verify MCP connection:**
+**Frontend**
+- React 19 + TypeScript
+- Tailwind CSS + Vite
+- TanStack Query for state management
-**HTTP Endpoint Tests:**
-```bash
-# Health check
-curl http://localhost:3000/health
+**Infrastructure**
+- Docker Compose orchestration
+- MinIO for file storage
+- Unstructured.io for document parsing
-# Server information
-curl http://localhost:3000/info
+## Configuration
-# Ping test
-curl http://localhost:3000/ping
-```
+### API Key Setup
+**Default API key**: `dev-api-key-123` (change for production)
-**MCP Protocol Test (requires MCP client):**
-```bash
-# Note: Direct tool testing is not possible without an MCP client
-# Use Cursor/VSCode MCP integration or MCP Inspector for actual tool testing
-```
+All administrative APIs require API key authentication via `X-API-KEY` header:
-### Advanced Configuration
-
-**Search tuning** in `application.yml`:
-```yaml
-app:
- search:
- snippet-max-length: 50 # Snippet length in characters
- bm25-weight: 0.3 # Keyword search weight (0.0-1.0)
- vector-weight: 0.7 # Semantic search weight (0.0-1.0)
- embedding:
- batch-size: 10 # Documents processed per batch
- content:
- default-max-tokens: 25000 # Maximum tokens per content response
- tokenizer: tiktoken-cl100k_base
-```
+```bash
+# Generate secure key
+openssl rand -hex 32
-**Document processing** configuration:
-```yaml
-opencontext:
- processing:
- chunk-size: 1000 # Characters per chunk
- chunk-overlap: 200 # Overlap between chunks
-```
+# Set environment variable
+export OPENCONTEXT_API_KEY="your-secure-key"
-**File upload limits:**
-```yaml
-spring:
- servlet:
- multipart:
- max-file-size: 100MB # Maximum single file size
- max-request-size: 100MB # Maximum total request size
+# Use in API calls
+curl -H "X-API-KEY: your-secure-key" http://localhost:8080/api/v1/sources
```
-### Environment Variables
+### Embedding Model
+Default: `dengcao/Qwen3-Embedding-0.6B:F16`
```bash
-# Core application
-SPRING_PROFILES_ACTIVE=docker
-OPENCONTEXT_API_KEY=your-secure-key
-
-# Database
-POSTGRES_DB=opencontext
-POSTGRES_USER=user
-POSTGRES_PASSWORD=secure-password
-
-# Object storage
-MINIO_ROOT_USER=minioadmin
-MINIO_ROOT_PASSWORD=secure-password
+# Install different model
+docker compose exec ollama ollama pull nomic-embed-text
-# Optional: GPU acceleration for Ollama
-OLLAMA_RUNTIME=nvidia
+# Update configuration in application-docker.yml
```
## Development
-### Local Development Setup
-
+### Local Development
```bash
-# Backend development
-cd core
-./gradlew bootRun
-
-# Frontend development
-cd admin-ui
-npm install
-npm run dev
-
-# MCP adapter development
-cd mcp-adapter
-npm install
-npm run dev
-```
+# Backend
+cd core && ./gradlew bootRun
-### Database Management
+# Frontend
+cd admin-ui && npm install && npm run dev
-```bash
-# Apply migrations
-./gradlew flywayMigrate
-
-# View migration status
-./gradlew flywayInfo
-
-# Repair migration checksums
-./gradlew flywayRepair
+# MCP Adapter
+cd mcp-adapter && npm install && npm run dev
```
### Testing
-
-```bash
-# Unit tests
-./gradlew test
-
-# Integration tests with TestContainers
-./gradlew integrationTest
-
-# Test coverage report
-./gradlew jacocoTestReport
-
-# Frontend tests
-cd admin-ui
-npm test
-```
-
-## Monitoring and Operations
-
-### Health Checks
-
-```mermaid
-graph LR
- A[Application] --> B[Health Endpoint]
- B --> C{Dependencies}
- C --> D[PostgreSQL]
- C --> E[Elasticsearch]
- C --> F[Ollama]
- C --> G[MinIO]
-
- D --> H{Status}
- E --> H
- F --> H
- G --> H
-
- H -->|All UP| I[Service Healthy]
- H -->|Any DOWN| J[Service Degraded]
-```
-
-Access monitoring endpoints:
-- Health: `http://localhost:8080/actuator/health`
-- Metrics: `http://localhost:8080/actuator/metrics`
-- Application info: `http://localhost:8080/actuator/info`
-
-### Performance Considerations
-
-| Component | Resource Requirements | Scaling Notes |
-|-----------|---------------------|---------------|
-| Core Application | 2GB RAM, 2 CPU cores | Stateless, horizontally scalable |
-| PostgreSQL | 1GB RAM, SSD storage | Read replicas for query scaling |
-| Elasticsearch | 2GB RAM, fast disk | Cluster deployment for HA |
-| Ollama | 4GB RAM, GPU optional | Model caching, batch processing |
-
-### Log Management
-
```bash
-# View application logs
-docker compose logs -f open-context-core
-
-# Search-specific logging
-docker compose logs -f elasticsearch
-
-# System-wide logging
-docker compose logs --tail=100
+./gradlew test # Unit tests
+./gradlew integrationTest # Integration tests
+cd admin-ui && npm test # Frontend tests
```
-## Production Deployment
-
-### Docker Compose Production
-
-```yaml
-# docker-compose.prod.yml
-version: '3.8'
-services:
- open-context-core:
- image: opencontext/core:1.0.0
- environment:
- SPRING_PROFILES_ACTIVE: prod
- OPENCONTEXT_API_KEY: ${API_KEY}
- deploy:
- replicas: 2
- resources:
- limits:
- memory: 4G
- cpus: '2'
-```
+## Documentation
-### Security Hardening
+- **[Core Backend Guide](core/README.md)** - Backend development and API details
+- **[MCP Adapter Guide](mcp-adapter/README.md)** - MCP protocol implementation
+- **[Admin UI Guide](admin-ui/README.md)** - Frontend development guide
+- **[Installation and Setup](https://github.com/OpenContextAI/open-context/wiki/Installation-and-Setup)** - Production deployment and configuration
+- **[API Reference](https://github.com/OpenContextAI/open-context/wiki/API)** - Complete API documentation
+- **[Interactive API Docs](http://localhost:8080/swagger-ui.html)** - Swagger UI for testing
-```bash
-# Generate secure API key
-openssl rand -hex 32
+## Contributing
-# Configure TLS termination
-# Use reverse proxy (nginx/traefik) for HTTPS
+We welcome contributions! Please see our [Contributing Guide](https://github.com/OpenContextAI/open-context/wiki/How-to-Contribute) for complete details.
-# Network isolation
-# Deploy in isolated Docker network
-# Configure firewall rules for port access
-```
+### Development Workflow
+1. Fork the repository and clone locally
+2. Create feature branch: `feature/123-brief-description`
+3. Implement changes with comprehensive tests
+4. Follow commit convention: `type(scope): subject`
+5. Submit pull request with detailed description
+6. Address code review feedback
-### Backup Strategy
+### Branch Strategy
+- **main**: Production releases
+- **develop**: Integration branch for features
+- **feature/***: New features and improvements
+- **hotfix/***: Critical production fixes
-```bash
-# Database backup
-docker compose exec postgres pg_dump -U user opencontext > backup.sql
+### Security
+For security vulnerabilities, please email security@opencontext.ai instead of opening a public issue.
-# Elasticsearch snapshot
-curl -X PUT "localhost:9200/_snapshot/backup_repo/snapshot_1"
-
-# MinIO data backup
-mc mirror minio/opencontext-documents /backup/documents/
-```
-
-## Open Source Community
-
-### Why We Open Sourced OpenContext
-
-**Transparency Builds Trust**: Security-critical software must be auditable. Every line of code is open for inspection, ensuring no backdoors or data harvesting mechanisms exist.
-
-**Community-Driven Security**: Open source transparency enables security professionals to review the code, helping identify and address vulnerabilities through collaborative development.
-
-**Customization Freedom**: Organizations can modify OpenContext to meet their specific security requirements, compliance standards, and operational needs.
-
-**No Vendor Lock-in**: You own your deployment, your data, and your destiny. No licensing fees, usage limits, or forced upgrades.
-
-### Contributing to OpenContext
-
-We welcome contributions from security professionals, developers, and organizations who understand the critical importance of data sovereignty.
-
-#### High-Impact Contribution Areas
-
-**Security Enhancements**
-- Advanced encryption implementations
-- Zero-knowledge proof integration
-- Hardware security module (HSM) support
-- Differential privacy mechanisms
-
-**Enterprise Features**
-- SAML/OIDC authentication integration
-- Advanced role-based access control (RBAC)
-- Multi-tenant deployment architectures
-- Kubernetes operator development
-
-**Performance Optimizations**
-- Distributed search capabilities
-- GPU acceleration for embeddings
-- Caching layer implementations
-- Query optimization algorithms
-
-**Integration Adapters**
-- Additional MCP protocol tools
-- Enterprise document management systems
-- Version control system integrations
-- Monitoring and alerting systems
-
-#### Development Workflow
-
-1. **Fork and Branch**: Create feature branch from latest `develop`
-2. **Security Review**: All code changes undergo security-focused review
-3. **Comprehensive Testing**: Unit, integration, and security tests required
-4. **Documentation**: Update API docs, security considerations, and deployment guides
-5. **Community Review**: Pull requests reviewed by maintainers and community
-
-#### Code Standards and Security Requirements
-
-- **Java**: Google Java Style Guide with additional security annotations
-- **TypeScript**: Prettier + ESLint with security-focused rules
-- **Security**: OWASP secure coding standards, dependency vulnerability scanning
-- **Testing**: 90% code coverage requirement, security test cases mandatory
-- **Audit Trail**: All code changes must maintain audit logging capabilities
-
-#### Community Support
+## Support
-- **GitHub Discussions**: Architecture decisions, feature planning, security considerations
-- **Security Advisory**: Responsible disclosure of vulnerabilities
-- **Documentation Wiki**: Community-maintained deployment guides and best practices
-- **Monthly Community Calls**: Feature roadmap discussions and security updates
+- **Issues**: [GitHub Issues](../../issues)
+- **Discussions**: [GitHub Discussions](../../discussions)
+- **Documentation**: [Project Wiki](https://github.com/OpenContextAI/open-context/wiki)
## License
-Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) file for full license text.
+Licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) file for details.
-## Support
+---
-- **Bug Reports**: [GitHub Issues](../../issues)
-- **Feature Requests**: [GitHub Discussions](../../discussions)
-- **Documentation**: [Project Wiki](../../wiki)
\ No newline at end of file
+**Enterprise Support**: Need help with deployment, customization, or security reviews? Contact us at enterprise@opencontext.ai
\ No newline at end of file
diff --git a/admin-ui/README.md b/admin-ui/README.md
index 2743947..1955365 100644
--- a/admin-ui/README.md
+++ b/admin-ui/README.md
@@ -27,7 +27,8 @@ React-based administration interface for OpenContext knowledge management system
- Node.js 18+
- npm or yarn
-- OpenContext backend running on localhost:8080
+- OpenContext backend running on localhost:8080 (for development)
+- Docker and Docker Compose (for production deployment)
### Setup
@@ -45,6 +46,21 @@ npm run build
npm run preview
```
+### Docker Deployment
+
+The admin UI is containerized and automatically deployed with the full stack:
+
+```bash
+# Build and start all services (from project root)
+docker compose up -d
+
+# Build admin UI only
+docker compose build open-context-admin-ui
+
+# View logs
+docker compose logs -f open-context-admin-ui
+```
+
### Environment Variables
```bash
@@ -54,12 +70,23 @@ VITE_API_BASE_URL=http://localhost:8080/api/v1
## Usage
+### Development Mode
1. Start the backend server (OpenContext Core)
-2. Run the frontend development server
+2. Run the frontend development server: `npm run dev`
3. Navigate to http://localhost:5173
4. Configure API key in Settings page
5. Upload documents and test search functionality
+### Production Mode (Docker)
+1. Start all services: `docker compose up -d`
+2. Navigate to http://localhost:3001
+3. Configure API key in Settings page (default: `dev-api-key-123`)
+4. Upload documents and test search functionality
+
+### Port Information
+- **Development**: http://localhost:5173 (Vite dev server)
+- **Production**: http://localhost:3001 (Docker container)
+
## API Integration
The admin UI integrates with OpenContext backend APIs:
diff --git a/core/README.md b/core/README.md
new file mode 100644
index 0000000..e6ca3c7
--- /dev/null
+++ b/core/README.md
@@ -0,0 +1,404 @@
+# OpenContext Core Backend
+
+[](https://openjdk.org/projects/jdk/21/)
+[](https://spring.io/projects/spring-boot)
+[](https://github.com/langchain4j/langchain4j)
+
+**Enterprise-grade RAG backend service built with Spring Boot and LangChain4j**
+
+This is the core backend service of OpenContext, implementing document processing pipelines, hybrid search capabilities, and MCP protocol APIs for secure, self-hosted RAG systems.
+
+## Architecture Overview
+
+The core service follows a layered architecture with clear separation of concerns:
+
+```
+com.opencontext/
+├── config/ # Spring configuration classes
+├── common/ # Common response objects and utilities
+├── controller/ # REST API endpoints
+├── service/ # Business logic layer
+├── repository/ # Data access layer with QueryDSL
+├── entity/ # JPA entities
+├── dto/ # Data transfer objects
+├── enums/ # Enum definitions (IngestionStatus, ErrorCode)
+├── exception/ # Custom exceptions and global handlers
+└── pipeline/ # Document processing pipeline components
+```
+
+## Key Components
+
+### Document Processing Pipeline
+- **DocumentParsingService**: Integrates with Unstructured.io for PDF/Markdown parsing
+- **ChunkingService**: Hierarchical text chunking with LangChain4j
+- **EmbeddingService**: Ollama API integration for text embeddings
+- **IndexingService**: Elasticsearch indexing with hybrid search setup
+
+### Search & Retrieval Engine
+- **SearchService**: Hybrid BM25 + vector search with Korean language support
+- **ContentRetrievalService**: Chunk content retrieval with token limit handling
+- **SearchController**: MCP protocol API endpoints (`/search`, `/get-content`)
+
+### Data Layer
+- **PostgreSQL**: Source document metadata and chunk hierarchy storage
+- **Elasticsearch**: Document search index with Korean Nori analyzer
+- **MinIO**: Original file storage with S3-compatible API
+
+## Technology Stack
+
+### Core Framework
+- **Java 21**: Latest LTS with virtual thread support
+- **Spring Boot 3.3.11**: Main application framework
+- **Spring Security**: API key authentication
+- **Spring Data JPA + Hibernate 6.x**: ORM with PostgreSQL
+- **QueryDSL 5.x**: Type-safe query construction
+
+### RAG Pipeline
+- **LangChain4j 0.36.2**: RAG pipeline orchestration framework
+- **Ollama**: Local embedding model serving (Qwen3-Embedding-0.6B)
+- **Elasticsearch 8.x**: Hybrid search engine with vector capabilities
+- **Unstructured.io**: Document parsing service (Docker API)
+
+### Testing & Quality
+- **JUnit 5**: Unit testing framework
+- **TestContainers**: Integration testing with real databases
+- **Mockito**: Mocking framework
+- **Flyway**: Database migration management
+
+## API Endpoints
+
+### MCP Protocol APIs (No Authentication)
+These endpoints are used by the MCP adapter for AI assistant integration:
+
+```http
+# Phase 1: Exploratory search
+GET /api/v1/search?query={query}&topK={count}
+
+# Phase 2: Content retrieval
+POST /api/v1/get-content
+Content-Type: application/json
+{
+ "chunkId": "uuid-string",
+ "maxTokens": 25000
+}
+```
+
+### Administrative APIs (API Key Required)
+Document management endpoints requiring `X-API-KEY` header:
+
+```http
+# Upload documents
+POST /api/v1/sources/upload
+X-API-KEY: your-api-key
+Content-Type: multipart/form-data
+
+# List documents with pagination
+GET /api/v1/sources?page=0&size=20&sort=createdAt,desc
+X-API-KEY: your-api-key
+
+# Trigger document reprocessing
+POST /api/v1/sources/{id}/resync
+X-API-KEY: your-api-key
+
+# Delete document
+DELETE /api/v1/sources/{id}
+X-API-KEY: your-api-key
+```
+
+## Configuration
+
+### Database Configuration
+```yaml
+# application.yml
+spring:
+ datasource:
+ url: jdbc:postgresql://localhost:5432/opencontext
+ username: user
+ password: password
+ jpa:
+ hibernate:
+ ddl-auto: validate # Production: validate only
+ show-sql: false
+ flyway:
+ baseline-on-migrate: true
+```
+
+### Elasticsearch Configuration
+```yaml
+app:
+ elasticsearch:
+ url: http://localhost:9200
+ index: document_chunks_index
+```
+
+### Embedding Service Configuration
+```yaml
+app:
+ ollama:
+ api:
+ url: http://localhost:11434
+ embedding:
+ model: dengcao/Qwen3-Embedding-0.6B:F16
+ embedding:
+ batch-size: 10
+```
+
+### API Security
+```yaml
+opencontext:
+ api:
+ key: dev-api-key-123 # Change for production
+```
+
+## Database Schema
+
+### Source Documents Table
+Stores original document metadata and processing status:
+
+```sql
+CREATE TABLE source_documents (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ original_filename VARCHAR(255) NOT NULL,
+ file_storage_path VARCHAR(1024) NOT NULL,
+ file_type VARCHAR(50) NOT NULL,
+ file_size BIGINT NOT NULL,
+ file_checksum VARCHAR(64) NOT NULL UNIQUE,
+ ingestion_status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
+ error_message TEXT,
+ last_ingested_at TIMESTAMPTZ,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### Document Chunks Table
+Maintains hierarchical chunk structure and relationships:
+
+```sql
+CREATE TABLE document_chunks (
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+ source_document_id UUID NOT NULL REFERENCES source_documents(id) ON DELETE CASCADE,
+ parent_chunk_id UUID REFERENCES document_chunks(id) ON DELETE CASCADE,
+ sequence_in_document INT NOT NULL,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### Elasticsearch Index Mapping
+Document chunks are stored in Elasticsearch with the following structure:
+
+```json
+{
+ "mappings": {
+ "properties": {
+ "chunkId": { "type": "keyword" },
+ "content": {
+ "type": "text",
+ "analyzer": "korean_nori"
+ },
+ "embedding": {
+ "type": "dense_vector",
+ "dims": 1024,
+ "index": "true",
+ "similarity": "cosine"
+ },
+ "metadata": {
+ "properties": {
+ "title": { "type": "text", "analyzer": "korean_nori" },
+ "hierarchyLevel": { "type": "integer" },
+ "originalFilename": { "type": "keyword" },
+ "fileType": { "type": "keyword" }
+ }
+ }
+ }
+ }
+}
+```
+
+## Development
+
+### Prerequisites
+```bash
+java --version # OpenJDK 21
+./gradlew --version # Gradle 8+
+```
+
+### Local Development Setup
+```bash
+# Start dependencies with Docker Compose
+docker compose up -d postgres elasticsearch ollama minio
+
+# Run application
+./gradlew bootRun
+
+# Run with specific profile
+./gradlew bootRun --args='--spring.profiles.active=dev'
+```
+
+### Testing
+```bash
+# Unit tests
+./gradlew test
+
+# Integration tests (requires TestContainers)
+./gradlew integrationTest
+
+# Test with coverage
+./gradlew jacocoTestReport
+
+# Build without tests
+./gradlew build -x test
+```
+
+### Database Migrations
+```bash
+# Apply migrations
+./gradlew flywayMigrate
+
+# Check migration status
+./gradlew flywayInfo
+
+# Repair migration checksums (if needed)
+./gradlew flywayRepair
+```
+
+## Monitoring & Operations
+
+### Health Checks
+The application exposes Spring Boot Actuator endpoints:
+
+```bash
+# Overall health status
+curl http://localhost:8080/actuator/health
+
+# Database connectivity
+curl http://localhost:8080/actuator/health/db
+
+# Elasticsearch connectivity
+curl http://localhost:8080/actuator/health/elasticsearch
+
+# Application info
+curl http://localhost:8080/actuator/info
+```
+
+### Logging Configuration
+```yaml
+# application.yml
+logging:
+ level:
+ com.opencontext: DEBUG # Application logs
+ org.hibernate.SQL: DEBUG # SQL queries
+ org.hibernate.orm.jdbc.bind: TRACE # SQL parameters
+```
+
+### Performance Tuning
+```yaml
+# application.yml
+spring:
+ datasource:
+ hikari:
+ maximum-pool-size: 10
+ minimum-idle: 2
+ connection-timeout: 20000
+
+server:
+ tomcat:
+ threads:
+ max: 200
+ min-spare: 10
+```
+
+## Error Handling
+
+The application uses structured error responses with specific error codes:
+
+### Common Error Codes
+- `VALIDATION_FAILED` (400): Request validation errors
+- `INSUFFICIENT_PERMISSION` (403): Invalid API key
+- `SOURCE_DOCUMENT_NOT_FOUND` (404): Document not found
+- `CHUNK_NOT_FOUND` (404): Chunk not found
+- `DUPLICATE_FILE_UPLOADED` (409): File already exists
+- `CONTENT_NOT_AVAILABLE` (422): Chunk content unavailable
+- `EXTERNAL_SERVICE_UNAVAILABLE` (503): Ollama/Elasticsearch connection error
+
+### Error Response Format
+```json
+{
+ "success": false,
+ "data": null,
+ "message": "Human-readable error message",
+ "errorCode": "ERROR_CODE_ENUM_VALUE",
+ "timestamp": "2025-08-21T12:00:00Z"
+}
+```
+
+## Security Considerations
+
+### API Key Authentication
+- All administrative endpoints require `X-API-KEY` header
+- Default development key: `dev-api-key-123`
+- Generate production keys with: `openssl rand -hex 32`
+
+### Data Protection
+- All data processing happens locally
+- No external API calls for sensitive document content
+- PostgreSQL stores only metadata and relationships
+- Elasticsearch contains indexed content for search
+
+### Network Security
+- MCP endpoints (`/search`, `/get-content`) have no authentication for internal use
+- Administrative endpoints require API key authentication
+- Configure reverse proxy (nginx) for HTTPS termination in production
+
+## Troubleshooting
+
+### Common Issues
+
+**Elasticsearch Connection Failed**
+```bash
+# Check Elasticsearch status
+curl http://localhost:9200/_cluster/health
+
+# Verify Korean analyzer plugin
+curl http://localhost:9200/_cat/plugins?v
+```
+
+**Ollama Model Not Found**
+```bash
+# Pull embedding model
+docker compose exec ollama ollama pull dengcao/Qwen3-Embedding-0.6B:F16
+
+# List available models
+docker compose exec ollama ollama list
+```
+
+**Database Migration Issues**
+```bash
+# Check migration status
+./gradlew flywayInfo
+
+# Repair checksums after manual changes
+./gradlew flywayRepair
+```
+
+## Contributing
+
+### Code Style
+- Follow Google Java Style Guide
+- Use Lombok annotations for boilerplate reduction
+- Prefer constructor injection over field injection
+- Write comprehensive JavaDoc for public APIs
+
+### Testing Requirements
+- Unit tests for all service classes
+- Integration tests using TestContainers for repository classes
+- API tests for all controller endpoints
+- Minimum 80% code coverage required
+
+### Commit Guidelines
+- Use conventional commits format
+- Include issue numbers in commit messages
+- Keep commits focused on single concerns
+- Write clear, descriptive commit messages
+
+For more information, see the main [Contributing Guide](../CONTRIBUTING.md).
\ No newline at end of file