Feature/ingestion pipeline by Yoo-SH · Pull Request #12 · OpenContextAI/open-context

Yoo-SH · 2025-08-13T17:11:10Z

Here's the PR template in English based on the git log:

Summary of Changes

This PR implements a complete document processing pipeline that transforms the existing legacy architecture into a new modular service architecture. It supports the entire workflow from document upload to semantic search, including asynchronous processing and comprehensive error handling.

Type of Change

New feature (non-breaking change which adds functionality)
Code refactoring
Performance improvement
Bug fix (non-breaking change which fixes an issue)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Test addition

Related Issues

Testing Instructions

Document Upload Test: Upload various format documents (PDF, Markdown) to verify the pipeline works correctly
Pipeline Stage Test: Verify that PARSING → CHUNKING → EMBEDDING → INDEXING stages execute sequentially
Async Processing Test: Confirm asynchronous processing works correctly for large document processing
Error Handling Test: Verify appropriate handling of potential errors at each stage
Elasticsearch Integration Test: Ensure generated vectors are properly stored in Elasticsearch
PostgreSQL Integration Test: Verify document hierarchy is properly stored in PostgreSQL

Screenshots (if applicable)

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Additional Notes

🚀 Key Implementation Features

DocumentParsingService: Document parsing through Unstructured API
ChunkingService: Splitting documents into meaningful chunks
EmbeddingService: Vector generation using LangChain4j + Ollama
IndexingService: Dual storage in Elasticsearch + PostgreSQL
SourceController: Complete ingestion pipeline orchestration

🔧 Technical Improvements

Asynchronous Processing: Performance optimization through @Async
Modularization: Service separation following single responsibility principle
Error Handling: Detailed error logging and status management at each stage
Batch Processing: Optimization for large data processing

Data Flow

Document Upload → Parsing → Chunking → Embedding → Indexing → Completion
      ↓            ↓         ↓          ↓          ↓
    MinIO     Unstructured  Hierarchy  Ollama    ES + PG

⚠️ Important Notes

Legacy controller and service files have been removed
New environment variables may be required (Ollama, Elasticsearch, etc.)
Database schema changes are included

🏗️ Architecture Changes

Before: Monolithic controller with mixed responsibilities
After: Modular services with clear separation of concerns
Benefits: Better testability, maintainability, and scalability

This PR represents a complete redesign of the document processing system, so please review it from an overall architecture perspective. The changes include significant refactoring and new service implementations that work together to provide a robust document ingestion pipeline.

- Create new service for parsing uploaded documents - Integrate with Unstructured API for text extraction - Support multiple document formats (PDF, Markdown, etc.) - Handle document parsing errors and validation

- Update CommonResponse for better API response formatting - Improve WebConfig for enhanced web configuration - Add new error codes for document processing pipeline

…ion-pipeline

- Modify DTO structure to support document chunking - Update fields for better chunk processing - Prepare data model for chunking service integration

- Rename parent_chunk_id to parent_chunk_uuid for consistency - Add source_document_uuid for direct access - Improve JPA relationship mapping structure

- Add document status management methods - Implement complete document deletion pipeline - Improve logging with emojis and categories - Add Elasticsearch and PostgreSQL cleanup integration

- Implement document chunking logic and strategies - Add hierarchical structure management - Support various document types and formats

…n-pipeline

- Add updateLastIngestedAt for tracking ingestion completion - Add updateErrorMessage and clearErrorMessage for error handling - Improve entity state management capabilities

- Integrate LangChain4j with Ollama for embedding generation - Implement batch processing for performance optimization - Generate semantic vectors for document chunks - Support configurable batch sizes and model parameters

…stion-pipeline

- Implement Elasticsearch bulk indexing for vector data - Add PostgreSQL hierarchy storage for document structure - Include content sanitization to prevent JSON parsing errors - Support batch processing for performance optimization

…n-pipeline

…line - Add full document processing pipeline (PARSING → CHUNKING → EMBEDDING → INDEXING) - Implement asynchronous processing for better performance - Integrate with new modular services (ChunkingService, EmbeddingService, IndexingService) - Add comprehensive error handling and status management

…estion-pipeline

Copilot

Pull Request Overview

This PR implements a complete document ingestion pipeline that transforms the legacy monolithic architecture into a modular service-based system. It establishes an end-to-end workflow for document processing from upload through semantic search indexing, with comprehensive async processing and error handling.

Pipeline Implementation: Transforms single-responsibility controller into orchestrated pipeline (PARSING → CHUNKING → EMBEDDING → INDEXING)
Service Modularization: Breaks down monolithic SourceDocumentService into specialized services (DocumentParsingService, ChunkingService, EmbeddingService, IndexingService)
Dual Storage Strategy: Implements hybrid storage with Elasticsearch for vector search and PostgreSQL for hierarchical structure

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
SourceDocumentService.java	Complete removal of legacy monolithic service
SourceController.java	New unified controller with embedded pipeline orchestration
FileStorageService.java	Enhanced with document metadata management and comprehensive deletion
DocumentParsingService.java	New service for Unstructured API integration
ChunkingService.java	New service for hierarchical document chunking
EmbeddingService.java	New service for LangChain4j/Ollama embedding generation
IndexingService.java	New service for dual Elasticsearch/PostgreSQL indexing
Entity/DTO updates	Schema changes to support new pipeline architecture

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-13T17:14:03Z

+                        .hierarchyLevel(chunk.getHierarchyLevel())
+                        .parentChunkId(parentChunkUuid)
+                        .elementType(chunk.getElementType())
+                        .sequenceInDocument(0) // 기본값 설정, 실제로는 순서에 따라 설정해야 함


The sequenceInDocument field is hardcoded to 0 with a Korean comment indicating this should be set based on actual order. This could lead to incorrect document structure representation and should be properly calculated.

Suggested change

.sequenceInDocument(0) // 기본값 설정, 실제로는 순서에 따라 설정해야 함

.sequenceInDocument(i) // 실제 순서에 따라 설정

Copilot · 2025-08-13T17:14:03Z

+            json.append("}");
+            return json.toString();
+        }
+        return obj.toString();


The manual JSON serialization implementation is fragile and incomplete. Consider using Jackson ObjectMapper for proper JSON serialization, especially since the comment on line 261 acknowledges this should use ObjectMapper.

Suggested change

return obj.toString();

private static final ObjectMapper objectMapper = new ObjectMapper();

/**

* 객체를 JSON 문자열로 변환합니다.

* Jackson ObjectMapper를 사용하여 안전하게 변환합니다.

*/

private String toJsonString(Object obj) {

try {

return objectMapper.writeValueAsString(obj);

} catch (JsonProcessingException e) {

log.error("JSON 직렬화 오류", e);

throw new RuntimeException("JSON 직렬화 오류", e);

}

Copilot · 2025-08-13T17:14:03Z

+    /**
+     * content 필드의 Java 코드나 특수문자를 정리하여 JSON 파싱 에러를 방지합니다.
+     */
+    private String sanitizeContent(String content) {


The content sanitization logic is overly aggressive and removes potentially meaningful content. The regex patterns could inadvertently remove legitimate text that contains Java keywords or similar patterns.

Copilot · 2025-08-13T17:14:05Z

+    /**
+     * 문서 수집 파이프라인을 비동기적으로 실행합니다.
+     */
+    @Async


The @async annotation on a public method within the same class may not work as expected due to Spring AOP proxy limitations. Consider moving async methods to a separate service or using @async on interface methods.

Copilot · 2025-08-13T17:14:05Z

+    /**
+     * 문서 삭제 파이프라인을 비동기적으로 실행합니다.
+     */
+    @Async


The @async annotation on a public method within the same class may not work as expected due to Spring AOP proxy limitations. Consider moving async methods to a separate service or using @async on interface methods.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ice.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

….java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Yoo-SH added 16 commits August 14, 2025 01:01

feat: Add DocumentParsingService for document processing pipeline

04be713

- Create new service for parsing uploaded documents - Integrate with Unstructured API for text extraction - Support multiple document formats (PDF, Markdown, etc.) - Handle document parsing errors and validation

refactor: enhance common response structure and error handling

7407fb3

- Update CommonResponse for better API response formatting - Improve WebConfig for enhanced web configuration - Add new error codes for document processing pipeline

Merge branch 'feature/ingestion-pipeline-parsing' into feature/ingest…

613f9be

…ion-pipeline

feat: update StructuredChunk DTO for chunking pipeline

e5cecec

- Modify DTO structure to support document chunking - Update fields for better chunk processing - Prepare data model for chunking service integration

refactor: improve DocumentChunk entity structure and relationships

4b88310

- Rename parent_chunk_id to parent_chunk_uuid for consistency - Add source_document_uuid for direct access - Improve JPA relationship mapping structure

feat: deleteBySourceDocumentId function

8f1d5c2

feat: enhance FileStorageService with comprehensive document management

8289e2f

- Add document status management methods - Implement complete document deletion pipeline - Improve logging with emojis and categories - Add Elasticsearch and PostgreSQL cleanup integration

feat: add ChunkingService for document segmentation

698abfc

- Implement document chunking logic and strategies - Add hierarchical structure management - Support various document types and formats

Merge branch 'feature/ingestion-pipeline-chunk' into feature/ingestio…

ad46592

…n-pipeline

feat: add utility methods to SourceDocument entity

8e95c7c

- Add updateLastIngestedAt for tracking ingestion completion - Add updateErrorMessage and clearErrorMessage for error handling - Improve entity state management capabilities

feat: add EmbeddingService for semantic vector generation

a2096c0

- Integrate LangChain4j with Ollama for embedding generation - Implement batch processing for performance optimization - Generate semantic vectors for document chunks - Support configurable batch sizes and model parameters

Merge branch 'feature/ingestion-pipeline-embedding' into feature/inge…

5b5e541

…stion-pipeline

feat: add IndexingService for document chunk storage

2c71181

- Implement Elasticsearch bulk indexing for vector data - Add PostgreSQL hierarchy storage for document structure - Include content sanitization to prevent JSON parsing errors - Support batch processing for performance optimization

Merge branch 'feature/ingestion-pipeline-index' into feature/ingestio…

81545a1

…n-pipeline

Merge branch 'feature/ingestion-pipeline-controller' into feature/ing…

1700f0b

…estion-pipeline

Yoo-SH requested review from Copilot, kAYI0019 and qowlgur121 August 13, 2025 17:11

Yoo-SH assigned qowlgur121, Yoo-SH, kAYI0019 and Copilot Aug 13, 2025

Copilot AI reviewed Aug 13, 2025

View reviewed changes

Yoo-SH and others added 5 commits August 14, 2025 02:16

Update core/src/main/java/com/opencontext/service/ChunkingService.java

c8a392f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update core/src/main/java/com/opencontext/service/DocumentParsingServ…

51af28c

…ice.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update core/src/main/java/com/opencontext/service/IndexingService.java

03e99e8

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update core/src/main/java/com/opencontext/service/EmbeddingService.java

219588c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update core/src/main/java/com/opencontext/controller/SourceController…

4e24575

….java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Yoo-SH merged commit 6453bf5 into develop Aug 13, 2025

Yoo-SH mentioned this pull request Aug 21, 2025

# Pull Request: develop → main #36

Merged

53 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/ingestion pipeline#12

Feature/ingestion pipeline#12
Yoo-SH merged 21 commits into
developfrom
feature/ingestion-pipeline

Yoo-SH commented Aug 13, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Aug 13, 2025

Uh oh!

Copilot AI Aug 13, 2025

Uh oh!

Copilot AI Aug 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Aug 13, 2025

Uh oh!

Copilot AI Aug 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	.sequenceInDocument(0) // 기본값 설정, 실제로는 순서에 따라 설정해야 함
	.sequenceInDocument(i) // 실제 순서에 따라 설정

-        return obj.toString();
+    private static final ObjectMapper objectMapper = new ObjectMapper();
+    /**
+     * 객체를 JSON 문자열로 변환합니다.
+     * Jackson ObjectMapper를 사용하여 안전하게 변환합니다.
+     */
+    private String toJsonString(Object obj) {
+        try {
+            return objectMapper.writeValueAsString(obj);
+        } catch (JsonProcessingException e) {
+            log.error("JSON 직렬화 오류", e);
+            throw new RuntimeException("JSON 직렬화 오류", e);
+        }

Uh oh!

Conversation

Yoo-SH commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Type of Change

Related Issues

Testing Instructions

Screenshots (if applicable)

Checklist

Additional Notes

🚀 Key Implementation Features

🔧 Technical Improvements

Data Flow

⚠️ Important Notes

🏗️ Architecture Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Yoo-SH commented Aug 13, 2025 •

edited

Loading