Refactor: Archive Kuzu for ongoing support

[jfrench9]

Summary

This refactor updates the Kuzu database dependency to version 0.11.2 and transitions from remote dependency management to using a local archive for the Kuzu httpfs extension. This change improves build reliability and ensures consistent extension availability across different environments.

Key Changes

• Kuzu Version Update: Upgraded Kuzu dependency from previous version to 0.11.2 in project configuration
• Local Extension Archive: Added local httpfs extension binary for linux_arm64 architecture to project repository
• Docker Build Process: Modified containerization setup to copy Kuzu httpfs extension from local archive instead of downloading during build
• Dependency Lock Updates: Refreshed dependency lock file to reflect new Kuzu version and related package changes

Key Accomplishments

• ✅ Eliminated external dependency on remote extension downloads during builds
• ✅ Ensured consistent httpfs extension availability across deployments
• ✅ Improved build reliability and reduced potential network-related build failures
• ✅ Updated to latest stable Kuzu version (0.11.2) with bug fixes and improvements

Breaking Changes

• Container builds now require the local extension archive to be present in the repository
• Projects depending on previous Kuzu versions may need compatibility verification

Testing Notes

• Verify that httpfs functionality works correctly with the locally archived extension
• Test container builds in different environments to ensure extension loading works properly
• Validate that database operations dependent on httpfs extension continue to function as expected
• Confirm compatibility of existing code with Kuzu v0.11.2 API changes (if any)

Infrastructure Considerations

• Repository size increased due to inclusion of binary extension file (~6.4MB)
• Build process now has reduced external network dependencies
• Container image builds should be more reliable in network-restricted environments
• Consider implications for CI/CD pipelines that may cache or distribute these binaries

---

🤖 Generated with Claude Code

Branch Info:

• Source: refactor/kuzu-archive
• Target: main
• Type: refactor

Co-Authored-By: Claude noreply@anthropic.com

[jfrench9]

@claude please review this PR

[claude]

Claude finished @jfrench9's task —— View job

---

PR Review Complete ✅

I've completed a thorough review of this PR. Here's my feedback:

---

Summary

This PR upgrades Kuzu from >=0.11.0,<0.12 to pinned version 0.11.2 and transitions from downloading the httpfs extension during Docker build to bundling it locally. Overall, this is a solid improvement that enhances build reliability, but there are several considerations to address.

---

✅ Strengths

1. Improved Build Reliability: Eliminating the external download during build removes a network dependency and potential point of failure
2. Version Pinning: Changing from kuzu>=0.11.0,<0.12 to kuzu==0.11.2 provides better reproducibility and prevents unexpected breaking changes
3. Clean Implementation: The Dockerfile changes are well-structured, maintaining the same copy pattern to both user home and data directories
4. Documentation: PR description clearly explains the motivation and implications

---

⚠️ Concerns & Recommendations

1. Binary Provenance & Security (HIGH Priority)

Issue: A 6.2MB binary (libhttpfs.kuzu_extension) is being committed to the repository without verification of its source or integrity.

Concerns:

• No checksum/signature verification documented
• No documented source of this binary (was it extracted from a previous build? downloaded from official Kuzu?)
• Binary blobs in git increase repository size permanently

Recommendations:

# In PR description or commit message, add:
- Source: Official Kuzu extension repository
- Version: v0.11.2 for linux_arm64
- SHA256: <checksum>
- Downloaded from: https://extension.kuzudb.com/v0.11.2/linux_arm64/httpfs/libhttpfs.kuzu_extension

Consider adding a verification step in CI:

# Add checksum verification
RUN echo "EXPECTED_SHA256  bin/kuzu-extensions/v0.11.2/linux_arm64/httpfs/libhttpfs.kuzu_extension" | sha256sum -c -
COPY bin/kuzu-extensions/v0.11.2/linux_arm64/httpfs/libhttpfs.kuzu_extension \
    /kuzu-extension/0.11.2/linux_arm64/httpfs/libhttpfs.kuzu_extension

2. Multi-Architecture Support (MEDIUM Priority)

Issue: The bundled extension is ARM64-only (linux_arm64), but the Dockerfile uses python:3.12.10-slim base image, which is multi-arch.

Question: Do you ever build for x86_64/AMD64? The CLAUDE.md mentions "ECS Fargate ARM64" and "EC2 ARM64", suggesting ARM-only deployment.

Recommendations:

• If ARM64-only: Consider using FROM python:3.12.10-slim-bookworm or explicitly ARM64 base
• If multi-arch needed: Use build args to select appropriate extension binary

ARG TARGETARCH=arm64
COPY bin/kuzu-extensions/v0.11.2/linux_${TARGETARCH}/httpfs/libhttpfs.kuzu_extension \
    /kuzu-extension/0.11.2/linux_${TARGETARCH}/httpfs/libhttpfs.kuzu_extension

3. Version Compatibility (MEDIUM Priority)

Issue: Comments in database_manager.py:50 reference "Kuzu 0.11.0" but we're upgrading to 0.11.2:

# Construct safe path (Kuzu 0.11.0 uses .kuzu files)

Recommendation: Update comment to reflect current version or make it version-agnostic:

# Construct safe path (Kuzu 0.11.x uses .kuzu files)

4. Testing & Validation (HIGH Priority)

Missing:

• No evidence of testing the httpfs functionality with the bundled extension
• No CI/test updates to validate the extension loads correctly
• PR description mentions testing notes but doesn't confirm tests were run

Recommendations:

• Run integration tests that exercise httpfs functionality
• Add a simple smoke test in CI that verifies extension loading:

import kuzu
db = kuzu.Database("/tmp/test.kuzu")
conn = db.conn()
conn.execute("INSTALL httpfs")  # Should not fail
conn.execute("LOAD EXTENSION httpfs")  # Should not fail

5. Extension Update Strategy (LOW Priority)

Issue: No documentation on how to update the extension in the future.

Recommendation: Add a script or documentation (perhaps in bin/kuzu-extensions/README.md) explaining:

• How to download new versions
• How to verify integrity
• When to update (with Kuzu version bumps)

6. Lock File Changes (INFO)

The uv.lock file shows +1011/-732 lines, which is typical for a dependency version update. The large diff suggests transitive dependency updates, which is expected with Kuzu's upgrade.

---

🔍 Additional Questions

1. Breaking Changes: Have you verified that Kuzu 0.11.2 has no breaking API changes from 0.11.0? The code references .kuzu file format which appears consistent, but I'd recommend checking Kuzu's changelog.

2. Extension Functionality: Which parts of the codebase actually use httpfs? I found references in:

   • robosystems/routers/graphs/copy/strategies.py
   • robosystems/kuzu_api/routers/databases/ingest.py

   Have these been tested with the new bundled extension?

3. Production Deployment: Given the infrastructure is on EC2 instances (per CLAUDE.md), will existing Kuzu databases on running instances need any migration or are they compatible with 0.11.2?

---

📋 Suggested Action Items

Before merging:

• Document binary provenance (source URL, checksum)
• Run integration tests for httpfs functionality
• Update version comments in code (database_manager.py:50)
• Consider adding checksum verification to Dockerfile
• Add documentation for future extension updates

Nice to have:

• Create bin/kuzu-extensions/README.md with update procedures
• Add CI smoke test for extension loading
• Verify no breaking changes in Kuzu 0.11.0 → 0.11.2

---

🎯 Verdict

Conditional Approval - This is a good change that improves build reliability, but I recommend addressing the binary provenance/security concern before merging to production. The approach is sound, and with proper documentation of the binary's source and verification, this will be a solid improvement.

The version pinning is particularly valuable for reproducibility, and the local bundling approach aligns well with your Docker-based deployment model.

---

Job: View Run
Branch: refactor/kuzu-archive