Feature/vector log retention

config/vector.yaml

@@ -75,22 +75,22 @@ transforms:
 
 # Sinks - write to files
 sinks:
-  # Platform services log file
+  # Platform services log file (daily rotation)
   platform_logs:
     type: file
     inputs:
       - route_platform
-    path: /data/logs/platform.json
+    path: /data/logs/platform-%Y-%m-%d.json
     encoding:
       codec: json
     compression: none
 
-  # Agent containers log file
+  # Agent containers log file (daily rotation)
   agent_logs:
     type: file
     inputs:
       - route_agents
-    path: /data/logs/agents.json
+    path: /data/logs/agents-%Y-%m-%d.json
     encoding:
       codec: json
     compression: none

docker-compose.yml

@@ -31,13 +31,20 @@ services:
       - EMAIL_PROVIDER=${EMAIL_PROVIDER:-resend}
       - RESEND_API_KEY=${RESEND_API_KEY:-}
       - SMTP_FROM=${SMTP_FROM:-noreply@trinity.example.com}
+      # Log Retention & Archival Configuration
+      - LOG_RETENTION_DAYS=${LOG_RETENTION_DAYS:-90}
+      - LOG_ARCHIVE_ENABLED=${LOG_ARCHIVE_ENABLED:-true}
+      - LOG_CLEANUP_HOUR=${LOG_CLEANUP_HOUR:-3}
+      - LOG_ARCHIVE_PATH=/data/archives  # Local path for archived logs
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
       - ./src/backend:/app
       - ./config/agent-templates:/agent-configs/templates:ro
       - ./config/trinity-meta-prompt:/trinity-meta-prompt:ro
       - agent-configs:/agent-configs
       - trinity-data:/data
+      - trinity-logs:/data/logs:ro  # Read-only access to Vector logs
+      - trinity-archives:/data/archives  # Log archives storage
     depends_on:
       redis:
         condition: service_started
@@ -181,6 +188,7 @@ volumes:
   redis-data:
   trinity-data:
   trinity-logs:  # Vector log storage
+  trinity-archives:  # Compressed log archives
 
 networks:
   trinity-network:

docker/backend/Dockerfile

@@ -32,6 +32,7 @@ COPY ../../src/backend/db_models.py /app/
 COPY ../../src/backend/dependencies.py /app/
 COPY ../../src/backend/credentials.py /app/
 COPY ../../src/backend/database.py /app/
+COPY ../../src/backend/logging_config.py /app/
 
 # Copy module directories
 COPY ../../src/backend/routers /app/routers/

docs/memory/changelog.md

@@ -1,3 +1,105 @@
+### 2026-01-06 18:30:00
+♻️ **Refactor: Sovereign Archive Storage Architecture**
+
+**Refactoring**: Removed external S3 dependency and implemented pluggable storage interface with local-only default for sovereign deployments.
+
+**Motivation**: Following architectural feedback to maintain full data sovereignty - no system logs or archives should leave the company's infrastructure. The platform now stores all data locally within mounted volumes, giving operators full control over backup strategies.
+
+**What Changed**:
+1. **New Storage Interface** - Abstract `ArchiveStorage` base class with pluggable backend support
+2. **Local Storage Default** - `LocalArchiveStorage` implementation stores archives in mounted Docker volume
+3. **S3 Code Removed** - Deleted S3-specific code (`s3_storage.py`), boto3 dependency, and S3 env vars
+4. **Sovereign Backup Strategies** - Documentation now covers Docker volume backups, NAS/NFS mounts, rsync, cross-instance copies
+
+**New Files**:
+- `src/backend/services/archive_storage.py` - Storage abstraction layer
+
+**Deleted Files**:
+- `src/backend/services/s3_storage.py` - S3 integration (removed)
+
+**Modified Files**:
+- `src/backend/services/log_archive_service.py` - Uses storage interface instead of S3 directly
+- `docker/backend/Dockerfile` - Removed boto3 dependency
+- `docker-compose.yml` - Removed S3 env vars, added `LOG_ARCHIVE_PATH`
+- `docs/memory/feature-flows/vector-logging.md` - Replaced S3 docs with sovereign backup strategies
+- `tests/test_log_archive.py` - Removed S3 tests, updated to use storage interface
+
+**Configuration Changes**:
+```bash
+# Removed:
+LOG_S3_ENABLED, LOG_S3_BUCKET, LOG_S3_ACCESS_KEY, LOG_S3_SECRET_KEY,
+LOG_S3_ENDPOINT, LOG_S3_REGION
+
+# Added:
+LOG_ARCHIVE_PATH=/data/archives  # Local path for archived logs
+```
+
+**Backup Strategies** (Sovereign):
+- Mount NAS/NFS to archives volume
+- rsync to backup server via cron
+- Docker volume backup/restore
+- Cross-instance volume copy
+
+**Impact**:
+- **Breaking**: S3 configuration no longer supported (by design)
+- **Non-Breaking**: Archives still stored in `trinity-archives` volume
+- **Benefit**: Full data sovereignty - no external dependencies
+- **Extensible**: Storage interface allows future backends (NFS, SFTP, etc.) without S3
+
+**Architecture Philosophy**: Keep all data (logs, archives, backups) within operator-controlled infrastructure. External storage can be implemented via volume mounts or custom scripts, not hardcoded cloud dependencies.
+
+---
+
+### 2026-01-05 18:30:00
+✨ **Feature: Vector Log Retention and Archival**
+
+**New Feature**: Automated log retention, rotation, and archival system for Vector logs with local storage.
+
+**Compliance Gap Addressed**: Vector logging lacked retention policy and archival capabilities required for SOC2/ISO27001 compliance.
+
+**What Was Added**:
+1. **Daily Log Rotation** - Vector now writes to date-stamped files (`platform-2026-01-05.json`)
+2. **Automated Archival** - Nightly job compresses and archives logs older than retention period
+3. **Local Storage** - Archives stored in mounted Docker volume (`trinity-archives`)
+4. **API Endpoints** - Admin-only endpoints for stats, config, and manual archival
+5. **Integrity Verification** - SHA256 checksums ensure archive integrity
+
+**New Files**:
+- `src/backend/services/log_archive_service.py` - Archival logic + APScheduler
+- `src/backend/routers/logs.py` - Log management API endpoints
+- `tests/test_log_archive.py` - Comprehensive test coverage
+
+**Modified Files**:
+- `config/vector.yaml` - Date-based file paths for daily rotation
+- `src/backend/main.py` - Register logs router, start archive scheduler
+- `docker-compose.yml` - Add log retention env vars, trinity-archives volume
+- `docs/memory/feature-flows/vector-logging.md` - Document retention features
+
+**Configuration**:
+```bash
+LOG_RETENTION_DAYS=90       # Days to keep logs
+LOG_ARCHIVE_ENABLED=true    # Enable automated archival
+LOG_CLEANUP_HOUR=3          # Hour (UTC) to run nightly job
+LOG_ARCHIVE_PATH=/data/archives  # Local path for archived logs
+```
+
+**API Endpoints**:
+- `GET /api/logs/stats` - Log file statistics
+- `GET /api/logs/retention` - Retention configuration
+- `PUT /api/logs/retention` - Update retention (runtime)
+- `POST /api/logs/archive` - Manual archival trigger
+- `GET /api/logs/health` - Service health check
+
+**Impact**:
+- No breaking changes to existing Vector functionality
+- Existing single-file logs (`platform.json`, `agents.json`) can be manually archived
+- New date-based files start after Vector restart
+- Disk space remains stable after retention period
+
+**Compliance**: Addresses SOC2/ISO27001 requirement for log retention policy and secure archival.
+
+---
+
 ### 2026-01-05 10:15:00
 🐛 **Fix: ReplayTimeline infinite loop causing browser hang**
 

docs/memory/feature-flows/vector-logging.md

@@ -313,12 +313,159 @@ curl http://localhost:8686/health
 curl http://localhost:8686/api/v1/topology
 ```
 
+## Log Retention & Archival
+
+### Overview
+
+Automated log retention system that:
+1. Rotates logs daily to date-stamped files
+2. Archives old logs with compression to local storage
+3. Deletes originals after successful archive
+
+### Configuration
+
+Environment variables in `docker-compose.yml`:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `LOG_RETENTION_DAYS` | `90` | Days to keep logs before archival |
+| `LOG_ARCHIVE_ENABLED` | `true` | Enable automated archival |
+| `LOG_CLEANUP_HOUR` | `3` | Hour (UTC) to run nightly archival |
+| `LOG_ARCHIVE_PATH` | `/data/archives` | Local path for archived logs |
+
+### Daily Rotation
+
+Vector writes logs to date-stamped files:
+- `platform-2026-01-05.json` (today's platform logs)
+- `agents-2026-01-05.json` (today's agent logs)
+
+Files rotate automatically at midnight UTC.
+
+### Automated Archival
+
+The backend runs a nightly job (default: 3 AM UTC) that:
+
+1. **Finds old files**: Identifies logs older than retention period
+2. **Compresses**: Gzip level 9 (~90% size reduction)
+3. **Verifies**: SHA256 integrity check
+4. **Stores locally**: Moves to mounted archive volume
+5. **Cleans up**: Deletes original files after successful archive
+
+### API Endpoints
+
+Admin-only endpoints for log management:
+
+```bash
+# Get log statistics
+curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/logs/stats
+
+# Get retention configuration
+curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/logs/retention
+
+# Update retention (runtime only)
+curl -X PUT -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"retention_days": 30, "archive_enabled": true, "cleanup_hour": 3}' \
+  http://localhost:8000/api/logs/retention
+
+# Manually trigger archival
+curl -X POST -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"retention_days": 90, "delete_after_archive": true}' \
+  http://localhost:8000/api/logs/archive
+
+# Check archival service health
+curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/logs/health
+```
+
+### Custom Backup Strategies
+
+Trinity archives logs locally by default, keeping all data within your infrastructure (sovereign deployment). You can implement custom backup strategies using standard Docker volume management:
+
+#### Option 1: Mount NAS/NFS for Archives
+
+Mount a network storage device to the archives volume:
+
+```yaml
+# In docker-compose.yml
+services:
+  backend:
+    volumes:
+      - type: bind
+        source: /mnt/nas/trinity-archives  # Your NAS mount point
+        target: /data/archives
+```
+
+#### Option 2: rsync to Backup Server
+
+Create a cron job to sync archives to another server:
+
+```bash
+# /etc/cron.daily/trinity-archive-sync
+#!/bin/bash
+rsync -avz --delete \
+  /var/lib/docker/volumes/trinity-archives/_data/ \
+  backup-server:/backups/trinity-logs/
+```
+
+#### Option 3: Docker Volume Backup
+
+Use Docker's built-in volume backup:
+
+```bash
+# Backup archives volume to tarball
+docker run --rm \
+  -v trinity-archives:/data \
+  -v $(pwd):/backup \
+  alpine tar czf /backup/trinity-archives-$(date +%Y%m%d).tar.gz /data
+
+# Restore from tarball
+docker run --rm \
+  -v trinity-archives:/data \
+  -v $(pwd):/backup \
+  alpine tar xzf /backup/trinity-archives-20260105.tar.gz -C /
+```
+
+#### Option 4: Cross-Instance Volume Copy
+
+Copy archives between Docker instances:
+
+```bash
+# Export from instance A
+docker run --rm \
+  -v trinity-archives:/source \
+  -v $(pwd):/dest \
+  alpine sh -c "cd /source && tar czf - ." > trinity-archives.tar.gz
+
+# Import to instance B (transfer trinity-archives.tar.gz via scp/rsync)
+docker run --rm \
+  -v trinity-archives:/dest \
+  -v $(pwd):/source \
+  alpine sh -c "cd /dest && tar xzf /source/trinity-archives.tar.gz"
+```
+
+### Storage Locations
+
+| Location | Contents | Purpose |
+|----------|----------|---------|
+| `/data/logs/` | Active log files | Daily logs (current + retention period) |
+| `/data/archives/` | Compressed archives | Local storage of archived logs |
+| Docker volume `trinity-archives` | Persistent storage | Mounted at `/data/archives` in backend |
+
+### Disk Space Management
+
+Example with 90-day retention:
+- Daily log size: ~150 MB
+- 90 days uncompressed: ~13.5 GB
+- 90 days compressed: ~1.35 GB (10% of original)
+- Active logs stay at ~13.5 GB (stable after 90 days)
+
 ## Side Effects
 
-- **File Output**: Logs written to Docker volume `trinity-logs`
+- **File Output**: Logs written to Docker volume `trinity-logs` with daily rotation
+- **Compressed Archives**: Old logs archived to Docker volume `trinity-archives`
 - **No Database**: Logs are file-based, not persisted to SQLite
 - **No WebSocket**: No real-time log streaming to UI (query only)
-- **No Rotation**: Currently no automatic rotation configured (monitor file sizes)
 
 ## Error Handling
 
@@ -442,4 +589,6 @@ No cleanup required - logs accumulate until manually cleared.
 
 | Date | Change |
 |------|--------|
+| 2026-01-06 | Refactored to sovereign architecture - removed external S3 dependency, added pluggable storage interface |
+| 2026-01-05 | Added log retention, rotation, and archival capabilities |
 | 2025-12-31 | Initial implementation - replaced audit-logger with Vector |

src/backend/main.py

@@ -45,6 +45,7 @@
 from routers.public import router as public_router
 from routers.setup import router as setup_router
 from routers.telemetry import router as telemetry_router
+from routers.logs import router as logs_router
 
 # Import scheduler service
 from services.scheduler_service import scheduler_service
@@ -55,6 +56,9 @@
 # Import system agent service
 from services.system_agent_service import system_agent_service
 
+# Import log archive service
+from services.log_archive_service import log_archive_service
+
 # Import credentials manager for GitHub PAT initialization
 from credentials import CredentialManager, CredentialCreate
 
@@ -195,6 +199,13 @@ async def lifespan(app: FastAPI):
     except Exception as e:
         print(f"Error initializing scheduler: {e}")
 
+    # Initialize log archive service
+    try:
+        log_archive_service.start()
+        print("Log archive service started")
+    except Exception as e:
+        print(f"Error starting log archive service: {e}")
+
     yield
 
     # Shutdown the scheduler
@@ -204,6 +215,13 @@ async def lifespan(app: FastAPI):
     except Exception as e:
         print(f"Error shutting down scheduler: {e}")
 
+    # Shutdown log archive service
+    try:
+        log_archive_service.stop()
+        print("Log archive service stopped")
+    except Exception as e:
+        print(f"Error stopping log archive service: {e}")
+
 
 # Create FastAPI app
 app = FastAPI(
@@ -242,6 +260,7 @@ async def lifespan(app: FastAPI):
 app.include_router(public_router)
 app.include_router(setup_router)
 app.include_router(telemetry_router)
+app.include_router(logs_router)
 
 
 # WebSocket endpoint