Code chunking breaks line-by-line causing semantic loss and poor search quality' labels: bug assignees:

[Yoo-SH]

---

name: Bug Report
about: Create a report to help us improve
title: '[BUG] Code chunking breaks line-by-line causing semantic loss and poor search quality'
labels: bug
assignees: ''

Bug Description

The current chunking service is breaking code into individual lines instead of meaningful semantic units, resulting in loss of context, poor search results, and unusable code fragments. This significantly degrades the RAG system's ability to provide accurate and complete code examples.

Steps to Reproduce

Steps to reproduce the behavior:

1. Go to the document ingestion system
2. Upload a code file containing multi-line methods or classes
3. Process the document through the chunking service
4. Check the generated chunks in the vector database
5. See that code is split line-by-line instead of by logical blocks

Expected Behavior

Code should be chunked into meaningful semantic units that preserve:

• Complete methods or functions
• Logical code blocks with proper context
• Import statements with their related code
• Class definitions with their methods
• Syntactically complete and executable code snippets

Actual Behavior

Code is being chunked line-by-line, resulting in:

• Individual lines like "public static void main(String[] args) {" as separate chunks
• Broken syntax with missing braces, semicolons, or method signatures
• Loss of semantic context between related code lines
• Poor search quality where users get incomplete code fragments
• Inability to understand the full logic or execute the code examples

Screenshots

Current Problematic Chunking:

Chunk 1: "public static void main(String[] args) {"
Chunk 2: "int sum = numbers.stream()"
Chunk 3: ".mapToInt(Integer::intValue)"
Chunk 4: ".sum();"
Chunk 5: "System.out.println("Sum: " + sum);"
Chunk 6: "}"

Expected Chunking:

Chunk 1: Complete method with full context and executable code

Environment

• OS: Server environment
• System: Document processing/chunking service
• Component: RAG chunking pipeline
• Language: Java (but affects all programming languages)

Additional Context

This issue severely impacts:

1. Search Quality: Users searching for "stream API sum calculation" only get partial, unusable code
2. User Experience: Developers receive incomplete code that doesn't compile or run
3. Vector Embeddings: Poor semantic representation due to fragmented context
4. Learning Effectiveness: Students/developers can't understand complete programming concepts

Suggested Solutions:

• Implement AST-based chunking for code files
• Set minimum chunk sizes (100-200+ characters)
• Preserve syntactic completeness (matching braces, complete statements)
• Group related imports, class definitions, and method implementations
• Add metadata to link related chunks when necessary

Priority: High - This significantly degrades the core functionality of the RAG system for code-related queries.