Feature/min io by Yoo-SH · Pull Request #10 · OpenContextAI/open-context

Yoo-SH · 2025-08-11T15:21:49Z

Summary of Changes

Implemented a comprehensive document management system with MinIO object storage integration. Added REST APIs for file upload, retrieval, re-processing, and deletion with X-API-KEY authentication system for admin operations.

Type of Change

New feature (non-breaking change which adds functionality)

Related Issues

Closes #issue-number

Testing Instructions

Start the application: ./gradlew bootRun
Test APIs using POSTMAN with the following endpoints:
- POST /api/v1/sources/upload - File upload (requires X-API-KEY: dev-api-key-123 header)
- GET /api/v1/sources - List all documents with pagination
- POST /api/v1/sources/{id}/resync - Re-process document
- DELETE /api/v1/sources/{id} - Delete document
Verify 403 error when X-API-KEY header is missing
Verify 415 error when uploading unsupported file formats
Test file upload with PDF, Markdown, and text files

Screenshots (if applicable)

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Additional Notes

Key Implementation Features:

MinIO Integration: Object storage for file management with configurable endpoints
API Authentication: X-API-KEY header authentication for admin endpoints
Asynchronous Processing: Background document ingestion pipeline
Error Handling: Comprehensive error codes and structured response format
Swagger Documentation: Complete API documentation with examples
Clean Architecture: Interface-based design with Lombok optimization

Configuration Requirements:

MinIO server configuration (endpoint, access/secret keys in application.yml)
API key configuration (default: dev-api-key-123)

Files Added/Modified:

MinIOConfig.java - MinIO client configuration
ApiKeyAuthenticationFilter.java - API key authentication filter
SecurityConfig.java - Spring Security integration
SourceController.java & DocsSourceController.java - REST API controllers
FileStorageService.java & SourceDocumentService.java - Business logic services
SourceDocumentDto.java & SourceUploadResponse.java - API response DTOs
CommonResponse.java & ErrorCode.java - Error handling infrastructure

- Implement X-API-KEY header validation for /api/v1/sources endpoints - Add proper error responses for missing/invalid API keys - Support configurable API key via application properties - Secure admin APIs while keeping MCP APIs public

- Configure MinIO client bean with endpoint and credentials - Support configurable properties via application.yml - Enable file storage operations for document management

- Add ApiKeyAuthenticationFilter to Spring Security filter chain - Configure protected endpoints for admin APIs (/api/v1/sources) - Keep MCP APIs public while securing admin operations - Enable X-API-KEY header validation for document management

- Add SourceDocumentDto for document list API response - Add SourceUploadResponse for file upload API response

- Add FileStorageService for MinIO file operations - Add SourceDocumentService for document lifecycle management - Implement upload, retrieval, resync, and deletion operations - Support asynchronous document processing pipeline

- Add DocsSourceController interface with Swagger documentation - Add SourceController implementation for document management - Implement upload, list, resync, and delete endpoints - Support /api/v1/sources REST API with proper error handling

- Add error response factory methods for ErrorCode enum integration - Support both custom error messages and predefined error code - Improve error handling consistency across APIs

- Define error codes for authentication, document, file storage operations - Add HTTP status mapping for consistent error responses - Support categorized error codes (COMMON, AUTH, DOC, CTX, FILE, INFRA)

Copilot

Pull Request Overview

Implements a comprehensive document management system with MinIO object storage integration for the OpenContext project. The PR adds REST APIs for file upload, retrieval, re-processing, and deletion with X-API-KEY authentication system for admin operations, along with a complete document ingestion pipeline foundation.

MinIO integration for object storage with automatic bucket management
API Key authentication system using Spring Security for admin endpoints
Complete document ingestion pipeline with async processing support

Reviewed Changes

Copilot reviewed 42 out of 44 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`mcp-adapter/.gitignore`	Adds Node.js project gitignore configuration
`docker-compose.yml`	Adds platform architecture specification for containers
`core/build.gradle`	Updates dependencies and adds MinIO, LangChain4j, QueryDSL support
Database migration files	Creates PostgreSQL schema for document and chunk storage
Entity classes	Implements JPA entities for source documents and document chunks
Service layer	Implements document management and file storage services
Controller layer	Provides REST API endpoints with comprehensive Swagger documentation
Configuration classes	Sets up MinIO, security, async processing, and JPA auditing
Test files	Adds comprehensive unit and integration tests

Copilot · 2025-08-11T15:23:18Z

+<<<<<<< HEAD
+@rem SPDX-License-Identifier: Apache-2.0
+@rem
+=======
+@rem SPDX-License-Identifier: Apache-2.0
+@rem


The file contains unresolved merge conflicts. The merge conflict markers (<<<<<<< HEAD, =======, >>>>>>> 2b49148) need to be resolved before merging.

Suggested change

<<<<<<< HEAD

@rem SPDX-License-Identifier: Apache-2.0

@rem

=======

@rem SPDX-License-Identifier: Apache-2.0

@rem

@rem SPDX-License-Identifier: Apache-2.0

@rem

Copilot · 2025-08-11T15:23:18Z

+<<<<<<< HEAD
+echo. 1>&2
+echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
+echo. 1>&2
+echo Please set the JAVA_HOME variable in your environment to match the 1>&2
+echo location of your Java installation. 1>&2
+=======
+echo.
+echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
+echo.
+echo Please set the JAVA_HOME variable in your environment to match the
+echo location of your Java installation.
+>>>>>>> 2b49148 (build: regenerate gradle wrapper for consistent build environment)


Additional merge conflict marker found. All merge conflicts throughout the file need to be resolved.

Suggested change

<<<<<<< HEAD

echo. 1>&2

echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2

echo. 1>&2

echo Please set the JAVA_HOME variable in your environment to match the 1>&2

echo location of your Java installation. 1>&2

=======

echo.

echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.

echo.

echo Please set the JAVA_HOME variable in your environment to match the

echo location of your Java installation.

>>>>>>> 2b49148 (build: regenerate gradle wrapper for consistent build environment)

echo. 1>&2

echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2

echo. 1>&2

echo Please set the JAVA_HOME variable in your environment to match the 1>&2

echo location of your Java installation. 1>&2

Copilot · 2025-08-11T15:23:18Z

+        if (file.getSize() > 100 * 1024 * 1024) { // 100MB
+            throw new BusinessException(ErrorCode.PAYLOAD_TOO_LARGE, 
+                    "File size exceeds maximum limit of 100MB");


The file size limit (100MB) is hardcoded. Consider making this configurable through application properties to allow different limits in different environments.

Suggested change

if (file.getSize() > 100 * 1024 * 1024) { // 100MB

throw new BusinessException(ErrorCode.PAYLOAD_TOO_LARGE,

"File size exceeds maximum limit of 100MB");

if (file.getSize() > maxFileSize) {

throw new BusinessException(ErrorCode.PAYLOAD_TOO_LARGE,

String.format("File size exceeds maximum limit of %.2fMB", maxFileSize / (1024.0 * 1024.0)));

Copilot · 2025-08-11T15:23:19Z

+    @Value("${opencontext.api.key:dev-api-key-123}")
+    private String validApiKey;
+
+    // Endpoints that require API Key authentication


The default API key 'dev-api-key-123' is weak and predictable. Consider using a stronger default or requiring the API key to be explicitly configured in production environments.

Suggested change

@Value("${opencontext.api.key:dev-api-key-123}")

private String validApiKey;

// Endpoints that require API Key authentication

@Value("${opencontext.api.key}")

private String validApiKey;

// Endpoints that require API Key authentication

// Ensure the API key is set and not weak

@jakarta.annotation.PostConstruct

private void validateApiKey() {

if (!StringUtils.hasText(validApiKey) || "dev-api-key-123".equals(validApiKey)) {

throw new IllegalStateException("API key must be explicitly configured and not use a default/weak value.");

}

}

Yoo-SH added 8 commits August 11, 2025 23:42

feat: Add MinIO configuration for object storage

b0903e2

- Configure MinIO client bean with endpoint and credentials - Support configurable properties via application.yml - Enable file storage operations for document management

feat: Add DTOs for source document API responses

670bbae

- Add SourceDocumentDto for document list API response - Add SourceUploadResponse for file upload API response

feat: Add core services for document management

0fcdb27

- Add FileStorageService for MinIO file operations - Add SourceDocumentService for document lifecycle management - Implement upload, retrieval, resync, and deletion operations - Support asynchronous document processing pipeline

feat: Enhance CommonResponse with error handling support

a7719fe

- Add error response factory methods for ErrorCode enum integration - Support both custom error messages and predefined error code - Improve error handling consistency across APIs

feat: Add comprehensive error code definitions

e0dd7d9

- Define error codes for authentication, document, file storage operations - Add HTTP status mapping for consistent error responses - Support categorized error codes (COMMON, AUTH, DOC, CTX, FILE, INFRA)

Yoo-SH requested review from Copilot, kAYI0019 and qowlgur121 August 11, 2025 15:21

Yoo-SH assigned qowlgur121, Yoo-SH and kAYI0019 Aug 11, 2025

Copilot AI reviewed Aug 11, 2025

View reviewed changes

Yoo-SH changed the base branch from main to develop August 11, 2025 15:23

qowlgur121 merged commit 575a754 into develop Aug 11, 2025

Yoo-SH mentioned this pull request Aug 21, 2025

# Pull Request: develop → main #36

Merged

53 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/min io#10

Feature/min io#10
qowlgur121 merged 8 commits into
developfrom
feature/minIO

Yoo-SH commented Aug 11, 2025 •

edited by qowlgur121

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Aug 11, 2025

Uh oh!

Copilot AI Aug 11, 2025

Uh oh!

Copilot AI Aug 11, 2025

Uh oh!

Copilot AI Aug 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

-    @Value("${opencontext.api.key:dev-api-key-123}")
-    private String validApiKey;
-    // Endpoints that require API Key authentication
+    @Value("${opencontext.api.key}")
+    private String validApiKey;
+    // Endpoints that require API Key authentication
+    // Ensure the API key is set and not weak
+    @jakarta.annotation.PostConstruct
+    private void validateApiKey() {
+        if (!StringUtils.hasText(validApiKey) || "dev-api-key-123".equals(validApiKey)) {
+            throw new IllegalStateException("API key must be explicitly configured and not use a default/weak value.");
+        }
+    }

Uh oh!

Conversation

Yoo-SH commented Aug 11, 2025 • edited by qowlgur121 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Type of Change

Related Issues

Testing Instructions

Screenshots (if applicable)

Checklist

Additional Notes

Key Implementation Features:

Configuration Requirements:

Files Added/Modified:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Yoo-SH commented Aug 11, 2025 •

edited by qowlgur121

Loading