Improve local development setup and management

[jfrench9]

Summary

This PR significantly improves the local development experience by refactoring configuration management, enhancing administrative tooling, and standardizing code patterns across the codebase. The changes focus on making the system more maintainable and easier to work with during development.

Key Accomplishments

Development Environment Enhancements

• Improved Configuration Management: Enhanced environment variable setup with better examples and local development templates
• Streamlined Build System: Refactored task automation for improved consistency and usability
• Enhanced Documentation: Updated API documentation and developer guides with clearer instructions and examples

Credit System & Administrative Tooling

• Enhanced Credit Management: Improved credit administration tools with better error handling, validation, and user experience
• Repository Access Management: Streamlined repository access controls and permissions handling
• Administrative Scripts: Significant improvements to user creation, credit allocation, and system management utilities

Code Quality & Consistency

• Import Standardization: Consolidated import patterns across 25+ files for better consistency and maintainability
• Enhanced Error Handling: Improved error reporting and validation throughout the credit and billing systems
• Script Refactoring: Major improvements to administrative scripts with better argument parsing, logging, and user interaction

Infrastructure & Maintenance

• Cleaned Up Legacy Components: Removed deprecated queue management utilities and outdated API descriptions
• Updated Static Resources: Refreshed API documentation to reflect current system capabilities
• Test Alignment: Updated test suites to match refactored code patterns

Breaking Changes

None. All changes are backward compatible and focus on internal improvements.

Testing Notes

• All existing tests have been updated to reflect import and structural changes
• Administrative scripts should be tested in development environments before production use
• Credit management functionality maintains existing API contracts while improving internal reliability

Infrastructure Considerations

The enhanced configuration management and administrative tooling will improve operational efficiency and reduce setup time for new development environments. The standardized import patterns will make future maintenance and refactoring efforts more straightforward.

---

🤖 Generated with Claude Code

Branch Info:

• Source: chore/local-dev-experience
• Target: main
• Type: chore

Co-Authored-By: Claude noreply@anthropic.com

[jfrench9]

@claude please review this PR

[claude]

Claude finished @jfrench9's task —— View job

---

PR Review: Improve local development setup and enhance credit management tooling

I've completed a thorough review of this PR. Overall, this is a well-executed refactoring that improves the developer experience and code maintainability. The changes are backward compatible and focused on internal improvements.

---

✅ Strengths

Configuration Management

• Excellent addition of .env.local.example: This solves a real pain point for developers by providing localhost URL overrides for justfile commands while Docker services use container hostnames
• Clear documentation: Both .env.example and .env.local.example have excellent inline comments explaining their purpose and usage
• Proper justfile integration: The _default_env := ".env.local" pattern is smart and well-documented

Administrative Tooling

• credit_admin.py improvements (robosystems/scripts/credit_admin.py): Significant enhancement with better safety guards, dry-run mode, and clear warnings about automated vs manual operations
• create_test_user.py refactoring (robosystems/scripts/create_test_user.py): The new --modes flag (e.g., --modes json,sec,industry) is much more user-friendly than multiple separate flags
• graph_query.py enhancements (robosystems/scripts/graph_query.py): Better error handling, improved output formatting, and clearer help text

Code Quality

• Terminology standardization: Changing "FREE" to "included" throughout is more accurate and professional (robosystems/config/billing/repositories.py:41-44, robosystems/operations/graph/credit_service.py:805-814)
• Import consistency: Systematic cleanup across 25+ files improves maintainability
• Documentation accuracy: Removing emoji clutter from scripts makes logs more parseable

---

⚠️ Issues Found

1. Inconsistent Emoji Removal (robosystems/scripts/arelle_cache_manager.py vs robosystems/scripts/create_test_user.py)

The PR removes emojis from logging statements in arelle_cache_manager.py, but retains them in create_test_user.py:

# create_test_user.py still has:
print("🔄 Registering user: {email}")
print("✅ User registered successfully")

# arelle_cache_manager.py now has:
logger.info("Downloading XBRL schemas...")
logger.info("Schema download complete...")

Recommendation: Be consistent - either keep emojis in user-facing scripts or remove them from all scripts. I'd suggest keeping them in interactive scripts like create_test_user.py but removing from background/logging scripts (which you've done correctly).

Note: This inconsistency may actually be intentional (interactive vs logging context), but it should be documented if so.

---

💡 Suggestions for Improvement

1. Enhanced Error Messages in graph_query.py (line 136)

The timeout for queries is hardcoded to 300 seconds. Consider making this configurable:

parser.add_argument("--timeout", type=int, default=300, 
                   help="Query timeout in seconds (default: 300)")

2. Justfile Command Organization

The justfile has grown quite large (419 lines). Consider documenting the organization or adding section markers for better navigation. The current organization is logical but could benefit from clearer section headers.

3. Missing Validation in create_test_user.py (line 399)

The script accepts repository names from the modes but doesn't validate them against supported_repos before attempting to grant access. Add validation:

# Validate repository names
invalid_repos = [r for r in repositories_to_grant if r not in supported_repos]
if invalid_repos:
    print(f"❌ Invalid repository names: {', '.join(invalid_repos)}")
    print(f"   Supported: {', '.join(supported_repos)}")
    sys.exit(1)

4. CLAUDE.md Clarity Improvements

The CLAUDE.md changes improve brevity, but some removed details might be valuable:

• Line 11: Changed "all database ops are FREE" to removed entirely - the new credit system documentation is clearer
• Consider adding a note about when to use .env vs .env.local in the "Environment Variables" section

5. Static File Rename Documentation

The rename from static/kuzu-api-description.md to static/graph-api-description.md is good (reflects the backend-agnostic approach), but ensure any references in code or documentation are updated. The new content is much better at explaining the multi-backend architecture.

---

🔍 Code-Specific Observations

Positive Changes

1. Better credit service error messages (robosystems/operations/graph/credit_service.py:805-814)

   • The change from "operation_free": True to "operation_included": True is more professional
   • I verified this field isn't used elsewhere in the codebase, so no breaking changes

2. Improved repository billing config (robosystems/config/billing/repositories.py:41-76)

   • Replacing "(FREE)" with "(included)" is clearer and more accurate
   • Maintains consistency with the credit model where database ops don't consume credits

3. Better test descriptions (tests/operations/graph/test_credit_service.py:181-220)

   • Updated test names and comments reflect the new terminology
   • Makes the test intent clearer

4. Removed deprecated DLQ script (robosystems/scripts/manage_dlq.py deleted)

   • Good cleanup if this functionality is now handled elsewhere
   • Verify: Ensure no CI/CD scripts or documentation reference this file

Areas to Verify

1. Import verification passed: Confirmed that robosystems/tasks/billing/credit_allocation.py exports all functions imported by credit_admin.py

2. Test coverage: The PR updates existing tests to match terminology changes, which is good

3. Backward compatibility: Verified that the operation_free → operation_included change won't break anything (field not used elsewhere)

---

📋 Checklist Before Merge

• Run just test-all to ensure no regressions
• Verify no CI/CD scripts reference the deleted manage_dlq.py
• Consider documenting the emoji policy (interactive scripts keep them, logging scripts don't)
• Optionally add validation for repository names in create_test_user.py
• Ensure any documentation that references kuzu-api-description.md is updated to graph-api-description.md

---

🎯 Summary

Overall Assessment: ✅ Approve

This PR delivers on its promises:

• ✅ Improved local development experience with .env.local pattern
• ✅ Enhanced administrative tooling with better UX and safety guards
• ✅ Better code consistency through terminology standardization
• ✅ Cleaner documentation throughout

The issues identified are all minor and mostly stylistic. The code quality is high, the changes are well-thought-out, and backward compatibility is maintained. The refactoring improves maintainability without introducing breaking changes.

Recommendation: Merge - This is ready to go. The suggestions above are optional improvements for future PRs, not blockers.

---

📊 Review Stats

• Files reviewed: 40
• Critical issues: 0
• Warnings: 1 (emoji inconsistency - may be intentional)
• Suggestions: 5 (all optional)
• Lines added: 783
• Lines removed: 772

---

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[jfrench9]

@claude can you reivew the recent changes

[claude]

Claude finished @jfrench9's task —— View job

---

PR Review: Recent Changes Analysis

I've completed a thorough review of the recent changes in this PR. Here's my detailed feedback:

---

🔴 Critical Issue: .env File Handling Bug

Status: Must be fixed before merge

The Codex reviewer identified a P1 critical issue that will break fresh checkouts:

Problem: The justfile now defaults to .env.local (line 1: _default_env := ".env.local"), but compose.yaml still requires .env (line 15: env_file: - .env). On a fresh checkout:

1. User runs just start
2. compose-up creates .env.local but NOT .env (justfile:31-32)
3. Docker Compose fails with: ERROR: .env is not a file

Root Cause: justfile:31-32 in compose-up:

test -f .env || cp .env.example .env          # ✓ Creates .env
test -f {{env}} || cp .env.example {{env}}     # ✓ Creates .env.local

However, when _default_env := ".env.local" is used, the compose-up recipe receives .env.local as the env parameter, but Docker containers explicitly reference .env in compose.yaml.

Solution: The justfile correctly creates both files (lines 31-32), so this should work. Let me verify the actual flow:

Actually, reviewing more carefully: The compose-up command at lines 31-32 does create .env unconditionally on line 31, so this should not be a problem. The Codex warning may be a false positive because:

1. Line 31: test -f .env || cp .env.example .env - Always ensures .env exists
2. Line 32: test -f {{env}} || cp .env.example {{env}} - Ensures .env.local exists
3. Line 33: Docker uses --env-file {{env}} for runtime, but compose.yaml uses env_file: .env for defaults

Recommendation: Test this on a clean checkout to confirm it works. The logic looks correct, but the dual .env / .env.local pattern is complex and deserves validation.

---

✅ Strengths

1. Excellent Configuration Management Improvements

The new .env.local pattern solves a real developer pain point:

robosystems/.env.local.example:1-32

• Clear documentation explaining the dual-file pattern
• Docker services use container hostnames (pg, valkey, kuzu-api)
• justfile commands need localhost URLs
• .env.local provides localhost overrides

This is a smart separation of concerns.

2. Enhanced Administrative Tooling

robosystems/scripts/credit_admin.py:1-100

• Excellent header documentation (lines 7-28) explaining automated vs manual operations
• Clear warning that monthly allocation is automated via Celery Beat
• Proper imports from centralized task modules (lines 36-45)
• Good safety guards with dry-run mode (line 52)

robosystems/scripts/create_test_user.py:1-100

• Better argument parsing with --modes flag instead of multiple separate flags
• Clean session management with retry logic (lines 35-54)
• Emojis retained for user-facing feedback (lines 65, 72, 90) - appropriate for interactive scripts

3. Terminology Improvements

robosystems/operations/graph/credit_service.py:805-814

• Changed "FREE" to "included" throughout
• More accurate and professional terminology
• "operation_free": True → "operation_included": True
• Comments updated: "FREE" → "Included" (lines 1224-1230)

This is more accurate because these operations are included in the subscription, not truly "free."

4. Static Documentation Improvements

static/graph-api-description.md (new file, replaces kuzu-api-description.md):

• Backend-agnostic terminology
• Better explanation of multi-backend architecture (Kuzu primary, Neo4j optional)
• Clearer infrastructure tier descriptions
• More comprehensive feature documentation

---

⚠️ Issues & Suggestions

1. Justfile Complexity (Medium Priority)

The justfile is now 419 lines and uses a complex environment variable pattern. While the logic appears correct, consider:

Recommendation:

• Add section comments to organize the 419-line file
• Document the .env vs .env.local pattern in comments
• Consider a quickstart comment block at the top

Example:

# Configuration:
#   - .env: Container hostnames for Docker services (required by compose.yaml)
#   - .env.local: Localhost overrides for justfile commands (default for most recipes)
_default_env := ".env.local"

2. Test User Script Repository Validation (Low Priority)

robosystems/scripts/create_test_user.py

The --modes flag accepts repository names but doesn't validate them against SharedRepository enum before attempting access grant. This could lead to confusing errors.

Recommendation: Add validation similar to:

# After parsing modes
from robosystems.config.billing.repositories import SHARED_REPOSITORIES
invalid_repos = [r for r in repositories_to_grant if r not in SHARED_REPOSITORIES]
if invalid_repos:
    print(f"❌ Invalid repository names: {', '.join(invalid_repos)}")
    print(f"   Supported: {', '.join(SHARED_REPOSITORIES.keys())}")
    sys.exit(1)

3. Graph Query Script Timeout (Low Priority)

robosystems/scripts/graph_query.py

The query timeout appears to be hardcoded. Consider making it configurable:

parser.add_argument("--timeout", type=int, default=300,
                   help="Query timeout in seconds (default: 300)")

4. CLAUDE.md Documentation Gap (Low Priority)

The CLAUDE.md changes improve brevity but removed some context about the dual .env pattern. Consider adding:

### Environment Files
- `.env`: Container hostnames for Docker services (required)
- `.env.local`: Localhost URL overrides for justfile commands (default for most commands)

**Setup**: Both files are created automatically by `just start` or `just venv`

---

📊 Code Quality Assessment

Import Standardization ✅

• 25+ files updated to use centralized imports
• Consistent pattern across the codebase
• No breaking changes detected

Backward Compatibility ✅

• operation_free → operation_included field change verified safe
• Grepped codebase: field only used in credit_service.py
• Test updates reflect terminology changes correctly

Test Coverage ✅

• All test files updated to match new terminology
• Test descriptions improved for clarity
• No test functionality changes, only nomenclature

---

🔍 Specific File Reviews

justfile Changes

Lines 1-33: Environment variable refactoring

• ✅ _default_env := ".env.local" - Good pattern
• ✅ Lines 31-32 create both .env and .env.local
• ⚠️ Dual-file pattern adds complexity - needs validation
• ✅ Removed complex _dev variable override logic (simpler now)

robosystems/scripts/credit_admin.py

Lines 1-100: Dry-run and safety improvements

• ✅ Excellent header documentation
• ✅ Proper dry-run mode with detailed output
• ✅ Centralized task imports
• ✅ Clear warnings about automated vs manual operations

robosystems/scripts/create_test_user.py

Lines 1-100: Enhanced user creation

• ✅ --modes flag replaces multiple separate flags
• ✅ Emoji usage appropriate for interactive script
• ✅ Better session management with retry logic
• ⚠️ Missing repository name validation (see suggestion Fix MCP tools query results to include units and improve dimensions #2)

static/graph-api-description.md (new)

Full file review:

• ✅ Backend-agnostic terminology throughout
• ✅ Clear multi-backend architecture explanation
• ✅ Better infrastructure tier documentation
• ✅ More comprehensive than previous kuzu-api-description.md

---

📋 Pre-Merge Checklist

• Code Quality: All changes maintain high quality
• Backward Compatibility: No breaking changes
• Test Coverage: Tests updated appropriately
• Critical Bug: Verify .env handling on fresh checkout (see critical issue above)
• Optional Improvements: Consider suggestions 1-4 (all optional, not blockers)

---

🎯 Final Recommendation

Status: ⚠️ Conditional Approval

This PR delivers excellent improvements to developer experience, administrative tooling, and code consistency. However:

1. MUST DO: Verify the .env / .env.local handling works on a fresh checkout

   • Test: Delete both files, run just start, confirm Docker starts successfully
   • If it fails, the fix is simple: ensure line 31 of justfile always runs

2. SHOULD DO (optional): Consider implementing suggestions 1-4 for polish

If the .env creation is verified working, this PR is ready to merge. The suggested improvements are nice-to-haves for future PRs.

---

📈 Review Statistics

• Files Reviewed: 87
• Critical Issues: 1 (.env handling - needs verification)
• Warnings: 0
• Suggestions: 4 (all optional)
• Lines Added: 933
• Lines Removed: 875
• Net Impact: Improved maintainability and developer experience

---