feat: install Grafana Alloy on host to ship all site logs to Loki

[jonesrussell]

Status 2026-03-09

Alloy is not installed on razor-crest (which alloy → not found, grafana apt source not present). Requires interactive sudo (password) to install. Can't automate non-interactively.

Ready-to-run — paste this on razor-crest as jones

Step 1: Install Alloy

# Add Grafana apt repo
wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install -y alloy

Step 2: Write consolidated config (Option B — single system service)

Active deployer log paths confirmed:

• streetcode-laravel, coforge, diidjaaheer.live, goformx, movies-of-war.com (active logs)
• minoo, northops, oneredpaperclip, orewire-laravel, northcloud.biz (check if logs exist)

sudo mkdir -p /etc/alloy
sudo tee /etc/alloy/config.alloy << 'EOF'
logging {
  level  = "info"
  format = "logfmt"
}

loki.source.file "streetcode" {
  targets = [{ __path__ = "/home/deployer/streetcode-laravel/current/storage/logs/*.log", service = "streetcode" }]
  forward_to = [loki.write.loki.receiver]
  tail_from_end = true
}

loki.source.file "coforge" {
  targets = [{ __path__ = "/home/deployer/coforge/current/storage/logs/*.log", service = "coforge" }]
  forward_to = [loki.write.loki.receiver]
  tail_from_end = true
}

loki.source.file "diidjaaheer" {
  targets = [{ __path__ = "/home/deployer/diidjaaheer.live/current/storage/logs/*.log", service = "diidjaaheer" }]
  forward_to = [loki.write.loki.receiver]
  tail_from_end = true
}

loki.source.file "goformx" {
  targets = [{ __path__ = "/home/deployer/goformx/current/storage/logs/*.log", service = "goformx" }]
  forward_to = [loki.write.loki.receiver]
  tail_from_end = true
}

loki.source.file "movies_of_war" {
  targets = [{ __path__ = "/home/deployer/movies-of-war.com/current/storage/logs/*.log", service = "movies-of-war" }]
  forward_to = [loki.write.loki.receiver]
  tail_from_end = true
}

loki.source.file "northops" {
  targets = [{ __path__ = "/home/deployer/northops/current/storage/logs/*.log", service = "northops" }]
  forward_to = [loki.write.loki.receiver]
  tail_from_end = true
}

loki.source.file "orewire" {
  targets = [{ __path__ = "/home/deployer/orewire-laravel/current/storage/logs/*.log", service = "orewire" }]
  forward_to = [loki.write.loki.receiver]
  tail_from_end = true
}

loki.source.file "minoo" {
  targets = [{ __path__ = "/home/deployer/minoo/current/storage/logs/*.log", service = "minoo" }]
  forward_to = [loki.write.loki.receiver]
  tail_from_end = true
}

loki.write "loki" {
  endpoint {
    url = "http://127.0.0.1:3100/loki/api/v1/push"
    remote_timeout = "10s"
    batch_wait = "1s"
    batch_size = "100KiB"
  }
}
EOF

Step 3: Give alloy read access to deployer logs

# alloy system service runs as 'alloy' user — needs read access
sudo usermod -aG deployer alloy  # or set ACLs
# If that doesn't work:
sudo setfacl -R -m u:alloy:rX /home/deployer/*/current/storage/logs/

Step 4: Enable and start

sudo systemctl enable alloy
sudo systemctl start alloy
sudo systemctl status alloy

# Verify logs appear in Grafana (Explore → Loki → {service="streetcode"})

Step 5: Cap journald to prevent log accumulation

echo 'SystemMaxUse=512M' | sudo tee -a /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

Notes

• The deployer user's per-site systemd services (streetcode-alloy-loki.service) can remain disabled — the system alloy service supersedes them
• Alloy v1.13.2 on proxy-nyc1 uses ~108MB RSS; should be similar here
• If alloy user can't read deployer's logs, use setfacl (requires acl package)