feat: implement Kuzu backend with global connection pooling and SEC XBRL compatibility

[jfrench9]

Summary

This PR introduces comprehensive enhancements to the Kuzu graph database backend, implementing global connection pooling, improved memory management, and SEC XBRL processing compatibility. The changes refactor the existing Kuzu integration to be more robust, efficient, and production-ready.

Key Accomplishments

🔧 Backend Architecture Improvements

• Global Connection Pooling: Implemented centralized connection management for Kuzu backend to improve resource utilization and reduce connection overhead
• Enhanced Memory Management: Refactored cleanup processes to prevent memory leaks and ensure proper resource disposal
• Query Execution Optimization: Streamlined query processing with improved error handling and performance monitoring

🏗️ Configuration and Environment

• Enhanced Environment Configuration: Updated configuration files with clearer documentation and additional Kuzu-specific settings
• Improved Factory Pattern: Refactored backend factory with better initialization logic and test support
• Docker Compose Updates: Updated container configuration for better Kuzu integration

🧪 Testing and Reliability

• Enhanced Test Backend Factory: Improved test infrastructure with better mocking and setup for Kuzu backend testing
• Logging Improvements: Added comprehensive logging throughout the Kuzu integration for better debugging and monitoring

🔗 SEC XBRL Integration

• Orchestration Updates: Modified SEC XBRL processing pipeline to leverage the enhanced Kuzu backend capabilities
• Graph Engine Integration: Updated middleware to work seamlessly with the new connection pooling architecture

Breaking Changes

⚠️ Configuration Changes Required

• Environment variables have been updated - please review .env.example for new Kuzu-specific configuration options
• Existing Kuzu database connections may need to be reconfigured to use the new connection pooling system

Testing Notes

• All existing unit tests have been updated to work with the new backend architecture
• Backend factory tests now include comprehensive coverage for Kuzu-specific scenarios
• Integration testing should focus on connection pooling behavior and memory usage patterns

Infrastructure Considerations

• Memory Usage: The new connection pooling may change memory usage patterns - monitor resource consumption after deployment
• Connection Limits: Review database connection limits as the global pool may maintain persistent connections
• Startup Time: Initial application startup may be slightly longer due to connection pool initialization
• Configuration Validation: Ensure all Kuzu-related environment variables are properly set before deployment

Files Changed

• Configuration files (.env.example, compose.yaml)
• Core Kuzu backend implementation with 297 lines of changes
• Factory and middleware components for improved integration
• Test infrastructure updates for better coverage

---

🤖 Generated with Claude Code

Branch Info:

• Source: feature/kuzu-sec-compatible
• Target: main
• Type: feature

Co-Authored-By: Claude noreply@anthropic.com

[jfrench9]

@claude please review this PR

[claude]

Claude finished @jfrench9's task —— View job

---

Code Review Complete ✓

I've reviewed the PR implementing Kuzu backend with global connection pooling and SEC XBRL compatibility. Here's my comprehensive feedback:

---

✅ Strengths

Architecture Improvements

1. Global Connection Pooling - The refactoring to use a centralized ConnectionPool singleton (via get_connection_pool()) is excellent. This eliminates the per-instance connection management issues and properly shares Database objects across operations.

2. Clean Abstraction - The backend factory pattern in __init__.py cleanly supports both Kuzu and Neo4j backends, making the system truly multi-backend capable.

3. Improved Resource Management - The explicit connection cleanup in delete_database() with force_database_cleanup(aggressive=True) is a significant improvement for preventing memory leaks.

4. Checkpoint Integration - Adding CHECKPOINT; execution after COPY operations (kuzu.py:228-233) is critical for ensuring data visibility after SEC ingestion.

---

🔍 Issues Found

Critical Issues

1. Missing Connection Pool Import (kuzu.py:7)

   • The code imports get_connection_pool but there's no visible definition or import statement for this in the file
   • Impact: Code will fail at runtime with NameError
   • Fix: Add import like from robosystems.graph_api.core.connection_pool import get_connection_pool
   • Location: robosystems/graph_api/backends/kuzu.py:7

2. Database Path Confusion (kuzu.py:68)

   • Line 68: db_path = self.data_path / f"{database_name}.kuzu"
   • But the global connection pool likely uses its own path configuration
   • Concern: Potential mismatch between where the pool creates databases vs where delete looks for them
   • Impact: delete_database might fail to find/delete the correct files
   • Recommendation: Verify the ConnectionPool uses the same path convention or make this path configurable

High Priority Issues

3. Hardcoded Memory Settings (engine.py:71-76)

   • Different checkpoint thresholds for "sec" vs other databases:

     if db_name == "sec":
         checkpoint_threshold = 134217728  # 128MB for SEC
     else:
         checkpoint_threshold = 536870912  # 512MB for regular databases

   • Issue: This special-casing should probably align with the tier-based memory configuration in env.py
   • Recommendation: Use the tier configuration system instead of hardcoded values

4. No Error Handling for Pool Cleanup (kuzu.py:65)

   • force_database_cleanup() is called without try/catch
   • If cleanup fails, the physical file deletion will proceed anyway
   • Impact: Could leave orphaned connections or partial cleanup
   • Recommendation: Add error handling and logging

5. Inconsistent Database Name Handling (factory.py:799, 878)

   • The client stores _database_name for subgraph support, but it's unclear if all backend methods honor this
   • Concern: Subgraph routing might not work correctly if backend methods ignore this attribute
   • Recommendation: Verify all backend execute methods support database name parameter

Medium Priority Issues

6. Missing Type Hints (kuzu.py:18, 30, 56, etc.)

   • Several places use the ConnectionPool without explicit type hints
   • Impact: Reduced IDE support and type safety
   • Recommendation: Add type annotations for self.connection_pool

7. Incomplete Documentation (.env.example:101-105)

   • The new GRAPH_SHARED_REPOSITORY_BACKEND variable is added but not fully documented
   • Should clarify when/why you'd use this vs the default tier configuration
   • Location: .env.example:101-105

8. Test Coverage Gap (test_backend_factory.py:5-21)

   • Tests mock the ConnectionPool but don't verify actual connection pool behavior
   • No integration tests for the global connection pool lifecycle
   • Recommendation: Add integration tests that verify:
     • Multiple KuzuBackend instances share the same pool
     • Connection cleanup works correctly
     • Concurrent access patterns work

Low Priority Issues

9. Inconsistent Comment Style (kuzu.py:16, 133)

   • Mix of # Comment and # CRITICAL: and # NOTE: prefixes
   • Recommendation: Standardize to CLAUDE.md style (no comments unless necessary)

10. Verbose Logging (factory.py:319, 408, etc.)

    • Many logger.info() calls that might be better as logger.debug() in production
    • Impact: Potential log noise
    • Recommendation: Review log levels for production readiness

11. Magic Numbers (kuzu.py:228)

    • Line 228: CHECKPOINT; execution happens inside ingestion but could be configurable
    • Recommendation: Consider making checkpoint behavior configurable per tier

---

🎯 Specific Recommendations

Configuration Changes

1. Environment Variable Clarity

   • The .env.example additions are good but need better documentation
   • Add comments explaining the relationship between:
     • GRAPH_SHARED_REPOSITORY_BACKEND (dev override)
     • graph.yml tier configuration (AWS environments)
     • Default fallback behavior

2. Kuzu Home Directory

   • New KUZU_HOME variable added (env.py:437) but not used in kuzu.py
   • Either use it or remove it from the configuration

Testing Improvements

1. Add Integration Test

   def test_global_connection_pool_sharing():
       """Verify multiple backends share the same pool"""
       backend1 = get_backend()
       backend2 = get_backend()
       assert backend1.connection_pool is backend2.connection_pool

2. Test Cleanup Behavior

   async def test_database_deletion_cleanup():
       """Verify force_database_cleanup is called before file deletion"""
       # Test that connections are properly closed

Documentation Needs

1. Connection Pool Architecture

   • Add docstring to KuzuBackend.__init__() explaining the global pool pattern
   • Document the lifecycle: when pools are created, when they're cleaned up

2. Migration Guide

   • Since this changes how connections work, add migration notes for existing deployments
   • Explain breaking changes (if any) and upgrade path

---

🔧 Required Changes Before Merge

1. Fix the missing import for get_connection_pool (Critical Inital open source release of robosystems #1)
2. Verify database path consistency between pool and delete operation (Critical Fix MCP tools query results to include units and improve dimensions #2)
3. Add error handling to cleanup operations (High chore: improve documentation and suppress build output #4)
4. Review and test subgraph routing with new connection pool (High Remove unused Account and Disclosure transaction nodes from graph schema #5)

---

💡 Optional Improvements

1. Consider adding a connection pool health check endpoint
2. Add metrics for connection pool utilization
3. Document the memory management strategy in detail
4. Add configuration validation at startup to catch mismatches early

---

Summary

This is a solid architectural improvement that addresses real production issues with connection management and memory cleanup. The global connection pooling pattern is the right approach, and the integration with SEC XBRL processing looks sound.

However, the missing import for get_connection_pool is a blocking issue that must be resolved before merge. Additionally, the database path consistency needs verification to prevent runtime failures during database deletion.

Once the critical issues are addressed, this will be a significant improvement to the Kuzu backend reliability and performance.

Recommendation: Request changes for the critical issues, then approve once fixed and tested.

---

Job Run | Branch: feature/kuzu-sec-compatible

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bind query parameters when executing Kuzu queries

execute_query accepts a parameters argument but calls conn.execute(cypher) without passing it through. Any query that includes placeholders (e.g. MATCH (n) WHERE n.id=$id) will now raise because the parameters are never provided. The previous implementation delegated to Engine.execute_query, which handled parameter binding, so this change breaks parameterised queries across the API.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Query results no longer serializable to dicts

The new implementation appends each result.get_next() to rows and returns that list directly. These row objects/tuples do not expose dictionary semantics, yet downstream code (e.g. BackendClusterService) expects each item to support .keys() to build column metadata. Returning raw rows will therefore trigger AttributeError and prevent responses from being serialized. The prior code returned a list of dictionaries via Engine.execute_query.

Useful? React with 👍 / 👎.