fix(logs): retention can delete archived originals — rw mount + dir ownership (#1478)#1509
Merged
Conversation
…wnership (#1478) LOG_RETENTION_DAYS archived old log files but never freed disk: the backend mounted the logs volume read-only, so log_archive_service.unlink() failed per file (`Errno 30: Read-only file system`) and the volume grew unbounded (~900MB/day; eu2 hit 32G). Two layers were blocking the delete: 1. **Read-only mount** — docker-compose.prod.yml AND docker-compose.yml both mounted `trinity-logs:/data/logs:ro` on the backend. Changed to read-write. Vector stays the sole writer of new log lines; the backend only reads (to gzip) + deletes the archived originals. 2. **Directory ownership** — Vector runs as root and creates `/data/logs` root:root 0755, so even rw the backend (UID 1000) can't unlink (deleting a file needs write on the *directory*). Vector can't fix it itself (cap_drop: ALL → no CAP_CHOWN). Added a one-shot `logs-init` container (root, alpine) that `chown 1000:1000` + `chmod 775` the dir before the backend starts (depends_on: completed). Chowns the dir only (fast, not -R); Vector's future root-owned files stay deletable via the now-writable dir. Also: log_archive_service.start() logs a WARNING at startup if LOG_DIR isn't writable, so the failure is visible instead of silent per-file errors at cleanup_hour. Verified live (dev): reproduced the `:ro` failure; after the fix the dir is 1000:1000 775, backend writes, and a root-owned file in /data/logs is deletable by UID 1000 (Errno 30 gone). Closes #1478. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Resolve by running |
vybe
approved these changes
Jul 8, 2026
vybe
left a comment
vybe
left a comment
Contributor
There was a problem hiding this comment.
Validated via /validate-pr: required checks green, no secrets/PII, scope focused. Approving for sequential merge to dev.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1478.
Problem
LOG_RETENTION_DAYSarchived old logs but never freed disk — every delete failedErrno 30: Read-only file system, volume grew unbounded (~900MB/day; eu2 hit 32G).Two blocking layers (both fixed)
docker-compose.prod.ymlanddocker-compose.ymlboth mountedtrinity-logs:/data/logs:roon the backend (the issue assumed dev was rw — it wasn't). → changed to rw. Vector stays the sole writer of new lines; backend only reads (gzip) + deletes archived originals./data/logsroot:root 0755, so even rw the backend (UID 1000) can'tunlink(deleting needs write on the directory). Vector can't self-fix (cap_drop: ALL→ no CAP_CHOWN). → added a one-shotlogs-initcontainer (root alpine) thatchown 1000:1000+chmod 775the dir before the backend starts (depends_on: service_completed_successfully). Chowns the dir only (fast, not -R); Vector's future root-owned files stay deletable via the writable dir.Plus:
log_archive_service.start()now logs a WARNING ifLOG_DIRisn't writable — visible at startup instead of silent per-file errors at cleanup_hour.Verified live (dev)
:ro:touch /data/logs/... → Read-only file system.1000:1000 775; backend writes; a root-owned file in /data/logs is deletable by UID 1000 (Errno 30 gone).logs-initexits 0.Notes
logs-initruns as root (needs CAP_CHOWN) but uses a stock alpine image, not a Trinity-built one — Invariant Client/Viewer User Role (AUTH-002) #17 ("Trinity-built images non-root") doesn't apply; it's ephemeral (exits immediately)./var/lib/docker).🤖 Generated with Claude Code