feat: implement structured chunk data structures and development security config by qowlgur121 · Pull Request #9 · OpenContextAI/open-context

qowlgur121 · 2025-08-11T03:12:43Z

Summary

Implement core data structures for structured chunk processing and search functionality
Create comprehensive DTO layer with validation and API documentation support
Establish development security configuration for unrestricted API testing
Configure application settings for local development environment integration

Key Implementation Notes

Pattern: Applied standard DTO pattern with Jakarta validation and Swagger documentation
Testing: DTOs designed for seamless integration with future Elasticsearch indexing
API Docs: Complete Swagger annotations for automated API documentation generation
Performance: Optimized data structures with 1024-dimension embedding support

Major Components

DTO Layer

StructuredChunk: Core Elasticsearch document structure with embedding and metadata
ChunkMetadata: Hierarchical navigation with breadcrumbs and level tracking
SearchResultItem: Search API response format with relevance scoring
GetContentResponse: Content retrieval response with token information
TokenInfo: Token counting and management for content length control

Security Configuration

SecurityConfig: Development-focused security setup allowing Swagger UI access
API Access Control: Configured stateless REST API with proper endpoint permissions
MCP Integration: Prepared unrestricted access for future MCP API endpoints

Application Configuration

Database Settings: PostgreSQL connection with optimized HikariCP pool configuration
Service Endpoints: Configured Elasticsearch, Ollama, MinIO, and Unstructured.io connections
Development Logging: Enhanced logging levels for effective debugging

Files Added

core/src/main/java/com/opencontext/dto/StructuredChunk.java - Main document structure
core/src/main/java/com/opencontext/dto/ChunkMetadata.java - Hierarchical metadata
core/src/main/java/com/opencontext/dto/SearchResultItem.java - Search response format
core/src/main/java/com/opencontext/dto/GetContentResponse.java - Content retrieval response
core/src/main/java/com/opencontext/dto/TokenInfo.java - Token management info
core/src/main/java/com/opencontext/config/SecurityConfig.java - Security configuration

Files Modified

core/src/main/resources/application.yml - Development environment settings

Validation Features

All DTOs include comprehensive validation:

Required field validation with @NotNull and @notblank
Embedding dimension validation (1024-dimensional vectors)
Hierarchical structure validation with proper level constraints
Swagger documentation for all fields with examples

Related Issues

Closes #7

- Implement StructuredChunk with embedding support and metadata - Add hierarchical navigation with breadcrumbs and level tracking - Create response objects for search results and content delivery - Include token counting for content length management

- Allow unrestricted access to API documentation endpoints - Enable MCP API access without authentication - Configure stateless session management for REST APIs - Prepare foundation for API key authentication on admin endpoints

- Configure PostgreSQL connection with optimized pool settings - Add Elasticsearch, Ollama, and MinIO service endpoints - Enable Flyway baseline migration for existing databases - Set appropriate logging levels for development debugging

Copilot

Pull Request Overview

This PR implements core data structures for a structured chunk-based search system and establishes development security configuration. The changes introduce a comprehensive DTO layer with validation, Swagger documentation, and configure the application for local development.

Implements structured chunk DTOs with embedding support and hierarchical metadata for Elasticsearch integration
Creates API response DTOs for search results and content retrieval with token management
Establishes development security configuration allowing unrestricted API access and Swagger UI

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
application.yml	Updates database migration strategy and adds Flyway configuration
TokenInfo.java	Token counting and management DTO for LLM context control
StructuredChunk.java	Main Elasticsearch document structure with embeddings and metadata
SearchResultItem.java	Search API response format with relevance scoring
GetContentResponse.java	Content retrieval response with token information
ChunkMetadata.java	Hierarchical metadata structure for document navigation
SecurityConfig.java	Development security configuration with unrestricted API access

Copilot · 2025-08-11T03:13:34Z

+
+        log.info("Security configuration completed successfully");
+        return http.build();
+    }


The commented-out API key authentication filter import and usage on line 11 and 73 indicates incomplete security implementation. Admin APIs at /api/v1/sources/** are currently allowing unrestricted access (line 66) which poses a security risk in production environments.

Suggested change

}

.requestMatchers("/api/v1/sources/**").authenticated() // Require authentication (API key)

// All other requests require authentication

.anyRequest().authenticated()

);

// Add API Key authentication filter for Admin APIs

http.addFilterBefore(apiKeyAuthenticationFilter, UsernamePasswordAuthenticationFilter.class);

log.info("Security configuration completed successfully");

return http.build();

}

/**

* Simple API Key Authentication Filter for Admin APIs.

* Checks for X-API-KEY header and authenticates if valid.

*/

private class ApiKeyAuthenticationFilter extends OncePerRequestFilter {

private final AntPathRequestMatcher adminApiMatcher = new AntPathRequestMatcher("/api/v1/sources/**");

@Override

protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)

throws ServletException, IOException {

if (adminApiMatcher.matches(request)) {

String apiKey = request.getHeader(API_KEY_HEADER);

if (ADMIN_API_KEY.equals(apiKey)) {

Authentication auth = new PreAuthenticatedAuthenticationToken(

"admin", apiKey, Collections.singletonList(new SimpleGrantedAuthority("ROLE_ADMIN")));

SecurityContextHolder.getContext().setAuthentication(auth);

} else {

response.setStatus(HttpServletResponse.SC_UNAUTHORIZED);

response.getWriter().write("Unauthorized: Invalid API Key");

return;

}

}

filterChain.doFilter(request, response);

}

}

Copilot · 2025-08-11T03:13:34Z

+     * Generated by Qwen3-Embedding-0.6B model via Ollama.
+     * Uses float[] for optimal performance with Elasticsearch dense_vector type.
+     */
+    @Schema(description = "Embedding vector for semantic search (1024 dimensions)")


The embedding field lacks validation to ensure it contains exactly 1024 dimensions as described in the documentation comment. Consider adding a validation annotation like @Size(min = 1024, max = 1024) to enforce the documented constraint.

Suggested change

@Schema(description = "Embedding vector for semantic search (1024 dimensions)")

@Schema(description = "Embedding vector for semantic search (1024 dimensions)")

@jakarta.validation.constraints.Size(min = 1024, max = 1024)

Copilot · 2025-08-11T03:13:34Z

+import org.springframework.security.config.annotation.web.configurers.AbstractHttpConfigurer;
+import org.springframework.security.config.http.SessionCreationPolicy;
+import org.springframework.security.web.SecurityFilterChain;
+import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;


The import for UsernamePasswordAuthenticationFilter is unused since the API key authentication filter is commented out. This unused import should be removed to maintain clean code.

Suggested change

import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;

Yoo-SH

All components have been verified to function as intended. This pull request is approved for merge.

qowlgur121 added 3 commits August 11, 2025 12:06

config: configure security settings for development access

04cb857

- Allow unrestricted access to API documentation endpoints - Enable MCP API access without authentication - Configure stateless session management for REST APIs - Prepare foundation for API key authentication on admin endpoints

qowlgur121 requested review from Yoo-SH, Copilot and kAYI0019 August 11, 2025 03:12

Copilot AI reviewed Aug 11, 2025

View reviewed changes

Yoo-SH approved these changes Aug 11, 2025

View reviewed changes

Yoo-SH merged commit 899b6f3 into develop Aug 11, 2025

Yoo-SH mentioned this pull request Aug 21, 2025

# Pull Request: develop → main #36

Merged

53 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: implement structured chunk data structures and development security config - #9

feat: implement structured chunk data structures and development security config#9
Yoo-SH merged 3 commits into
developfrom
feature/7-implement-structured-chunk-mock-data-generation

qowlgur121 commented Aug 11, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Aug 11, 2025

Uh oh!

Copilot AI Aug 11, 2025

Uh oh!

Copilot AI Aug 11, 2025

Uh oh!

Yoo-SH left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-    }
+                .requestMatchers("/api/v1/sources/**").authenticated() // Require authentication (API key)
+                // All other requests require authentication
+                .anyRequest().authenticated()
+            );
+        // Add API Key authentication filter for Admin APIs
+        http.addFilterBefore(apiKeyAuthenticationFilter, UsernamePasswordAuthenticationFilter.class);
+        log.info("Security configuration completed successfully");
+        return http.build();
+    }
+    /**
+     * Simple API Key Authentication Filter for Admin APIs.
+     * Checks for X-API-KEY header and authenticates if valid.
+     */
+    private class ApiKeyAuthenticationFilter extends OncePerRequestFilter {
+        private final AntPathRequestMatcher adminApiMatcher = new AntPathRequestMatcher("/api/v1/sources/**");
+        @Override
+        protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
+                throws ServletException, IOException {
+            if (adminApiMatcher.matches(request)) {
+                String apiKey = request.getHeader(API_KEY_HEADER);
+                if (ADMIN_API_KEY.equals(apiKey)) {
+                    Authentication auth = new PreAuthenticatedAuthenticationToken(
+                        "admin", apiKey, Collections.singletonList(new SimpleGrantedAuthority("ROLE_ADMIN")));
+                    SecurityContextHolder.getContext().setAuthentication(auth);
+                } else {
+                    response.setStatus(HttpServletResponse.SC_UNAUTHORIZED);
+                    response.getWriter().write("Unauthorized: Invalid API Key");
+                    return;
+                }
+            }
+            filterChain.doFilter(request, response);
+        }
+    }

	@Schema(description = "Embedding vector for semantic search (1024 dimensions)")
	@Schema(description = "Embedding vector for semantic search (1024 dimensions)")
	@jakarta.validation.constraints.Size(min = 1024, max = 1024)

Uh oh!

Conversation

qowlgur121 commented Aug 11, 2025

Summary

Key Implementation Notes

Major Components

DTO Layer

Security Configuration

Application Configuration

Files Added

Files Modified

Validation Features

Related Issues

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

Yoo-SH left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants