Skip to content

feat: state manager manifest integration with TTL-based reconciliation#52

Merged
deanq merged 15 commits intomainfrom
deanq/ae-1747-state-manager-manifest-integration
Jan 27, 2026
Merged

feat: state manager manifest integration with TTL-based reconciliation#52
deanq merged 15 commits intomainfrom
deanq/ae-1747-state-manager-manifest-integration

Conversation

@deanq
Copy link
Copy Markdown
Contributor

@deanq deanq commented Jan 22, 2026

Summary

  • Implement TTL-based manifest reconciliation for state manager integration
  • Convert tetra-rp from git submodule to pip dependency for better package management
  • Update all imports and documentation to reflect dependency model changes
  • Add peer-to-peer cross-endpoint routing with ServiceRegistry
  • Add ARM64 Docker image build support

See Manifest Reconciliation Documentation for technical details on the TTL-based caching strategy.

Test plan

  • Run make quality-check to verify all tests pass
  • Run make test-handler to validate handler functionality
  • Verify Docker builds complete successfully for both amd64 and arm64
  • Test manifest reconciliation with TTL-based triggers

deanq added 6 commits January 21, 2026 18:33
- Add build-arm64, build-cpu-arm64, build-lb-cpu-arm64 targets
- Add push-arm64 target for pushing ARM64 images to Docker Hub
- Fix Makefile awk regex to properly match target names with numbers
- Update version to 0.7.3 in uv.lock
…viceRegistry

Replace hub-and-spoke architecture with peer-to-peer model where all endpoints
query State Manager directly for dynamic endpoint discovery. Integrates tetra-rp's
ServiceRegistry for consistent cross-endpoint function routing.

Key Changes:
- Update tetra-rp ServiceRegistry to use StateManagerClient directly (peer-to-peer)
- Replace manifest-based routing with async ServiceRegistry in RemoteExecutor
- Support provisioning-time endpoint IDs through State Manager
- Add optional tetra-rp import with graceful fallback for local execution
- Remove old static routing methods (_get_target_endpoint, _is_local_endpoint)
- Update tests to verify new ServiceRegistry-based routing

Architecture:
- All endpoints query State Manager directly (no single point of failure)
- 300s cache reduces State Manager queries to ~1 per endpoint per 5 minutes
- ServiceRegistry determines local vs remote based on FLASH_RESOURCE_NAME
- Graceful degradation if State Manager unavailable (fallback to local)

Benefits:
✓ Handles provisioning-time endpoint IDs correctly
✓ No hub-and-spoke dependency (more resilient)
✓ Consistent with tetra-rp client-side routing
✓ Automatic manifest updates via State Manager
✓ Simpler error handling with clear fallback paths

Test Results:
- All 205 tests pass
- Code coverage: 78.78% (required: 35%)
- All quality checks pass (ruff format, ruff check, mypy, pytest)
- Handler tests: 14/14 pass
- Removed .gitmodules and tetra-rp submodule registration
- Added tetra-rp as a git-based dependency in pyproject.toml
- Removed tetra-rp exclusions from mypy and ruff config
- For local development, install with: uv pip install -e ~/Github/python/tetra-rp

All 205 worker-tetra tests pass with 78% coverage.

Benefits:
- Cleaner project structure
- Easier dependency management
- Allows flexible local development with -e flag
- Both projects can have independent release cycles
- Updated DEVELOPMENT.md with local development workflow using pip -e flag
- Renamed 'Submodule Management' to 'tetra-rp Dependency Management'
- Added instructions for switching between remote and local tetra-rp
- Updated CI/CD workflows to remove all submodule references:
  - Removed 'submodules: recursive' from checkout actions
  - Removed 'git submodule update' commands
  - Removed 'cp tetra-rp/src/...' protocol sync commands
- Simplified setup workflow - uv sync now handles all dependencies
Update all code, tests, and documentation to use tetra-rp as a pip
dependency instead of a git submodule. This includes updating import
paths, test fixtures, and documentation references across the codebase.
- Add manifest_reconciliation.py with request-scoped refresh
- Integrate refresh into RemoteExecutor before routing
- Add comprehensive unit and integration tests
- Update remote_executor to call refresh_manifest_if_stale()

Serverless-compatible approach using TTL-based staleness detection.
No boot-time blocking, no background threads. Manifest refreshes
from State Manager only when stale (>5 min) before cross-endpoint routing.

Test coverage: 30+ unit tests, 15+ integration tests
Quality checks: All 241 tests pass, 79% code coverage
@deanq deanq requested a review from Copilot January 22, 2026 20:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements TTL-based manifest reconciliation for state manager integration and converts tetra-rp from a git submodule to a pip dependency. The changes enable peer-to-peer cross-endpoint routing with automatic manifest refresh while simplifying dependency management and adding ARM64 Docker support.

Changes:

  • Convert tetra-rp from git submodule to pip dependency for better package management
  • Add TTL-based manifest reconciliation with State Manager integration
  • Implement peer-to-peer cross-endpoint routing via ServiceRegistry
  • Add ARM64 Docker image build support
  • Update all imports from local remote_execution.py to tetra_rp.protos.remote_execution

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tetra-rp Removed git submodule reference
.gitmodules Removed submodule configuration
pyproject.toml Added tetra-rp as pip dependency and aiohttp for HTTP routing
src/remote_executor.py Added manifest reconciliation, ServiceRegistry integration, and cross-endpoint routing
src/manifest_reconciliation.py New module for TTL-based manifest refresh from State Manager
src/constants.py Added manifest path and timeout constants
Makefile Removed submodule commands, added ARM64 build targets
.github/workflows/*.yml Removed submodule checkout steps
tests/unit/test_manifest_reconciliation.py New comprehensive unit tests for manifest reconciliation
tests/integration/test_manifest_state_manager.py New integration tests for State Manager interaction
All test files Updated imports to use tetra_rp.protos.remote_execution
docs/Manifest_Reconciliation.md New comprehensive documentation
DEVELOPMENT.md Updated for pip dependency workflow
CLAUDE.md Updated references from submodule to pip dependency

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/remote_executor.py Outdated
@deanq deanq requested a review from Copilot January 22, 2026 21:04
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 36 out of 37 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread Makefile Outdated
- Remove unnecessary fallback in remote_executor.py line 383: function_name is
  guaranteed non-None by FunctionRequest validation, so the 'or ""' fallback
  is dead code that would fail anyway
- Improve Makefile help regex pattern to use /^[a-zA-Z_][a-zA-Z0-9_-]*:.*# /
  instead of /^[a-zA-Z0-9_-]+:.*# / to ensure targets start with a letter or
  underscore, preventing unintended matches of targets starting with digits
Comment thread docs/Manifest_Reconciliation.md Outdated
Comment thread docs/Manifest_Reconciliation.md Outdated
Comment thread src/remote_executor.py Outdated
@deanq deanq requested review from KAJdev and Copilot January 27, 2026 03:10
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 38 out of 39 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@deanq deanq merged commit 3683d6d into main Jan 27, 2026
18 checks passed
@deanq deanq deleted the deanq/ae-1747-state-manager-manifest-integration branch January 27, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants