Refactor configuration management for shared repository deployment

[jfrench9]

Summary

This PR consolidates and enhances the configuration management system to support shared repository deployment patterns. The changes streamline environment variable handling, improve configuration constants organization, and enhance snapshot management capabilities.

Key Accomplishments

Configuration System Overhaul

• Consolidated configuration constants: Reduced complexity in robosystems/config/constants.py and robosystems/config/env.py with cleaner environment variable management
• Enhanced environment settings: Updated .env.example with 55 configuration changes to support shared repository patterns
• Improved configuration validation: Strengthened validation logic across billing, credits, and external services configuration modules

Infrastructure Enhancements

• Added CloudFormation template: New graph-ladybug-replicas.yaml template for managing graph replica infrastructure
• Enhanced Dagster job capabilities: Significantly expanded shared repository job functionality with 188+ line additions for better job orchestration
• Improved snapshot management: Enhanced snapshot handling and configuration management across the system

Service Layer Improvements

• Streamlined service integrations: Updated authentication, graph API backends, middleware components, and routing layers to use consolidated configuration
• Enhanced database connectivity: Improved database configuration management and connection handling
• Optimized rate limiting and admission control: Updated middleware components to use new configuration patterns

Breaking Changes

⚠️ Configuration Variables: Some environment variable names and default values have changed. Teams should review and update their environment configurations based on the updated .env.example.

Testing Notes

• All 46 modified files include corresponding test updates
• Configuration tests have been updated to reflect new validation patterns
• Integration tests for graph API, middleware, and authentication components have been adjusted
• Test coverage maintained across billing, credits, and external services modules

Infrastructure Considerations

• New CloudFormation template requires deployment for replica management
• Environment variables should be updated in deployment environments
• Database connection configurations may need adjustment
• Setup scripts have been enhanced to support the new configuration patterns

The changes maintain backward compatibility where possible while providing a more robust and maintainable configuration system for shared repository deployments.

---

🤖 Generated with Claude Code

Branch Info:

• Source: chore/shared-replicas-deployment
• Target: main
• Type: chore

Co-Authored-By: Claude noreply@anthropic.com

[jfrench9]

@claude please review this PR

[claude]

Claude finished @jfrench9's task in 1m 45s —— View job

---

PR Review: Configuration Management Refactor

I've completed my review of this comprehensive configuration management refactoring. Overall, this is a well-structured PR that significantly improves configuration organization and sets up infrastructure for shared repository replicas. Below are my findings:

---

✅ Strengths

1. Excellent Configuration Consolidation

• env.py:311-431: Great reduction from 431 to 156 deletions - configuration is now more focused and organized
• constants.py:1-258: Clear separation between fixed constants and tunable parameters
• billing/core.py:18-83: Single source of truth for subscription tiers with inline credit allocation calculations
• Section organization in env.py mirrors .env.example structure, making it easy to navigate

2. Well-Documented Infrastructure Changes

• graph-ladybug-replicas.yaml:1-634: Comprehensive CloudFormation template with:
  • Clear comments explaining DLM vs Dagster snapshots (lines 237-244)
  • Proper separation of concerns (replica reads vs master writes)
  • Health check configuration with appropriate timeouts
  • Auto-scaling with CPU-based policies

3. Solid Test Coverage

• All 46 modified files include corresponding test updates
• test_env.py:140-240: Tests cover tier config fallbacks and environment detection
• Configuration tests properly updated for new patterns

4. Improved Type Safety

• Helper functions (get_int_env, get_float_env, get_bool_env) provide consistent error handling
• Safe type conversion with fallback to defaults (env.py:96-147)

---

⚠️ Issues & Recommendations

1. Configuration Variable Naming Inconsistency

Location: env.py:528-530

LBUG_DATABASE_PATH = get_str_env("LBUG_DATABASE_PATH", "./data/lbug-dbs")
LBUG_ACCESS_PATTERN = get_str_env("LBUG_ACCESS_PATTERN", "api_auto")
LBUG_NODE_TYPE = get_str_env("LBUG_NODE_TYPE", "writer")

Issue: The config uses LBUG_ prefix for LadybugDB-specific settings, but GRAPH_ prefix for generic graph settings. This could be confusing when both backends are in use.

Recommendation: Document the naming convention clearly or consider consistent prefixing.

2. Shared Repository Configuration Ambiguity

Location: env.py:415-421

SHARED_REPLICA_ALB_URL = get_str_env("SHARED_REPLICA_ALB_URL", "")
SHARED_REPOSITORIES = get_list_env("SHARED_REPOSITORIES", "")

Issue: The comment says "For application logic, use GraphTypeRegistry.SHARED_REPOSITORIES" but the variable SHARED_REPOSITORIES is still present. This dual-source could lead to confusion.

Recommendation: Either remove SHARED_REPOSITORIES entirely or add validation that ensures both sources match in production.

3. Database Connection Management Concern

Location: database.py:88-96

engine = create_engine(
  get_database_url(),
  pool_size=DEFAULT_POOL_SIZE,
  max_overflow=DEFAULT_MAX_OVERFLOW,
  pool_timeout=DEFAULT_POOL_TIMEOUT,
  pool_recycle=DEFAULT_POOL_RECYCLE,
  pool_pre_ping=True,
  echo=env.DATABASE_ECHO,
)

Issue: The engine is created at module load time. If get_database_url() depends on environment variables that change after import, this could cause issues.

Recommendation: This is acceptable if environment variables are always set before any imports, but document this dependency in CLAUDE.md.

4. CloudFormation Snapshot Update Process

Location: graph-ladybug-replicas.yaml:42-47

SnapshotId:
  Type: String
  Description: EBS Snapshot ID to restore data from (updated by Dagster shared_repository_snapshot job)
  AllowedPattern: ^snap-[0-9a-f]+$
  ConstraintDescription: Must be a valid EBS snapshot ID

Concern: The description mentions "updated by Dagster job" but there's no clear mechanism shown for how Dagster updates this CloudFormation parameter.

Recommendation:

• Add a comment or documentation explaining the update mechanism (CloudFormation API call? Stack update?)
• Consider adding error handling if snapshot is missing or invalid

5. Credits Configuration Overlap

Location: credits.py:102-112 and billing/core.py:141-151

Issue: Both files have get_monthly_allocation / get_monthly_credits methods that delegate to each other, creating potential circular dependency.

Recommendation: Consolidate into a single location (preferably billing/core.py since it's the "single source of truth").

6. Missing Validation for Tier Limits

Location: billing/core.py:29-66

Issue: The DEFAULT_GRAPH_BILLING_PLANS define max_queries_per_hour but there's no runtime validation that enforces these limits.

Recommendation: Add validation in BillingConfig.validate_configuration() or document where these limits are enforced.

7. JWT Configuration Default Derivation

Location: env.py:314-321

_jwt_default_domain = (
  os.getenv("ROBOSYSTEMS_API_URL", "https://api.robosystems.ai")
  .replace("https://", "")
  .replace("http://", "")
)
JWT_ISSUER = get_secret_value("JWT_ISSUER", _jwt_default_domain)
JWT_AUDIENCE = get_secret_list_value("JWT_AUDIENCE", _jwt_default_domain)

Issue: Direct use of os.getenv instead of the get_str_env helper function breaks the consistent pattern.

Recommendation: Use get_str_env("ROBOSYSTEMS_API_URL", "https://api.robosystems.ai") for consistency.

8. Dagster Job Timeout Configuration

Location: shared_repository.py:46-48

# Increased from 300s to 600s - large datasets need more warmup time
instance_warmup_seconds: int = 600

Issue: This is hardcoded in the config class. For very large datasets, 600s might still not be enough.

Recommendation: Make this configurable via environment variable or SSM parameter for production flexibility.

---

🔍 Security Considerations

1. Secrets Manager Fallback (env.py:29-57)

✅ Good: Graceful fallback when secrets manager is unavailable
⚠️ Consider: Log a warning in production if fallback is used (indicates misconfiguration)

2. Database SSL Configuration (database.py:17-35)

✅ Good: SSL is required for staging/prod
⚠️ Minor: The string manipulation approach could be fragile with complex connection strings

3. IAM Roles Over Credentials (env.py:605-622)

✅ Good: Properly documented that IAM roles should be used in production
✅ Good: Dev credentials are clearly marked as development-only

---

📋 Additional Observations

Breaking Changes

The PR description mentions breaking changes for environment variables. I recommend:

1. Create a migration guide showing the mapping from old → new variable names
2. Add startup validation that fails early if deprecated variables are detected

Documentation

• The .env.example file is well-organized with clear sections
• Consider adding a CHANGELOG.md entry documenting the configuration refactoring

Testing

• Test coverage appears comprehensive
• Consider adding integration tests that verify:
  • Shared replica boot process
  • Snapshot-based data loading
  • ALB health check behavior

---

📊 Summary

Category	Status	Notes
Code Quality	✅ Good	Clean refactoring with clear organization
Test Coverage	✅ Good	Comprehensive test updates
Documentation	⚠️ Adequate	Could use migration guide
Security	✅ Good	Proper secrets handling
Breaking Changes	⚠️ Document	Needs migration guide
Infrastructure	✅ Good	CloudFormation template is solid

---

🎯 Recommendation

Approve with minor changes

The refactoring is well-executed and provides significant value. Address the configuration ambiguity issues (#2, #5, #7) and add a migration guide for the breaking changes. The other recommendations can be addressed in follow-up PRs.

Quick Wins Before Merge

1. Fix JWT configuration to use get_str_env (env.py:316)
2. Add migration guide for environment variable changes
3. Clarify SHARED_REPOSITORIES vs GraphTypeRegistry.SHARED_REPOSITORIES usage
4. Document snapshot update mechanism in CloudFormation template

---

| Branch: chore/shared-replicas-deployment