feat: allow deploying to hetzner by tefkah · Pull Request #1432 · knowledgefutures/pubplatform

tefkah · 2026-04-02T11:19:32Z

Issue(s) Resolved

Deploy to hetzner instead of s3 a la v6

High-level Explanation of PR

Pushes all images to ghcr instead of ECR
Uses hetzner box to deploy preview images
Uses hetzner box to deploy sandbox
Uses hetzner box to deploy prod

Todo:

setup GH secrets SSH_PRIVATE_KEY, SSH_USER, SSH_HOST_PROD, SSH_HOST_STAGING, SSH_HOST_PREVIEW, GHCR_USER, GHCR_TOKEN.
manually point test cloudflare domains to the hetnzer ips
maybe create a new box for platform? can reuse devbox for now

Test Plan

Run ci a bunch to see if that still checks out
Try and deploy preview server
Try and deploy sandbox
Merge it and testttt

Screenshots (if applicable)

Notes

Copilot

Pull request overview

This PR migrates the repo’s CI/build/deploy pipeline from AWS ECR/ECS + PullPreview to GHCR + SSH-based Docker Swarm deployments on Hetzner, and introduces SOPS-managed env files plus Caddy-based proxying for preview and production stacks.

Changes:

Replace ECR build workflows with GHCR build workflows and update e2e to pull images from GHCR.
Replace PullPreview-based preview deployments with an SSH-driven Swarm stack deploy/teardown workflow.
Add Hetzner deployment workflow + new Swarm stack/Caddy/SOPS infra assets.

Reviewed changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
package.json	Adds SOPS encrypt/decrypt helper scripts.
infra/stack.yml	New Swarm stack definition for non-preview deployments.
infra/stack.preview.yml	New Swarm stack definition for PR previews (includes MinIO, init).
infra/import-backup.sh	Adds a helper to import DB dumps into the Swarm DB container.
infra/Caddyfile.preview	Adds preview Caddy config (on-demand TLS, routes).
infra/Caddyfile	Adds prod/staging Caddy config (routes to platform/site-builder/assets).
infra/.sops.yaml	Adds SOPS rules for encrypted env files.
infra/.env.preview.enc	Adds encrypted preview env file.
infra/.env.example	Adds infra env template for prod-like deployments.
docker-compose.preview.yml	Removes old PullPreview compose definition.
docker-compose.preview.sandbox.yml	Removes old PullPreview sandbox override.
docker-compose.preview.pr.yml	Removes old PullPreview PR override.
.gitignore	Ignores decrypted infra env files and keeps encrypted env tracked.
.github/workflows/pull-preview.yml	Removes PullPreview workflow.
.github/workflows/pull-preview-script.sh	Removes PullPreview helper script.
.github/workflows/preview.yml	Adds SSH-based PR preview deploy/teardown workflow.
.github/workflows/on_pr.yml	Switches PR pipeline to GHCR builds + new preview workflow.
.github/workflows/on_main.yml	Switches main pipeline to GHCR builds and removes AWS deploy steps.
.github/workflows/ghcr-build-template.yml	Adds reusable GHCR build/push workflow.
.github/workflows/ghcr-build-all.yml	Adds GHCR build-all aggregator workflow.
.github/workflows/ecrbuild-template.yml	Removes ECR build template workflow.
.github/workflows/ecrbuild-all.yml	Removes ECR build-all workflow.
.github/workflows/e2e.yml	Updates e2e to pull images from GHCR and adjusts permissions.
.github/workflows/deploy.yml	Adds Hetzner deploy workflow (build + SSH deploy).
.github/workflows/deploy-template.yml	Removes ECS deploy template workflow.
.github/workflows/awsdeploy.yml	Removes ECS deploy orchestrator workflow.
.env.example	Adds a base env example for local/dev/self-hosting defaults.

Comments suppressed due to low confidence (1)

infra/stack.yml:154

stack.yml configures Caddy to proxy /assets* and /assets-ui* to minio:9000/9001 (see infra/Caddyfile), but this stack doesn't define a minio service. As-is, asset routes will 502 and site serving via the fs s3 plugin will fail. Add a minio service (and any init/bucket/user bootstrap you need) or remove the MinIO proxies and the unused minio_data volume.

volumes:
  pgdata:
  minio_data:
  caddy_data:
  caddy_config:

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-02T11:27:28Z

infra/Caddyfile

@@ -0,0 +1,40 @@
+{
+    admin :2019


Caddy admin API is bound to :2019, which listens on all interfaces. That can allow any reachable peer (including other containers on the overlay network) to reconfigure Caddy. Bind it to localhost (e.g. 127.0.0.1:2019) or disable the admin endpoint unless you explicitly need remote admin access.

Suggested change

admin :2019

admin 127.0.0.1:2019

Copilot · 2026-04-02T11:27:28Z

infra/Caddyfile.preview

@@ -0,0 +1,53 @@
+{
+    admin :2019


Caddy admin API is bound to :2019 (all interfaces). For a public preview proxy this is risky; bind it to localhost or disable the admin endpoint to avoid exposing remote config mutation.

Suggested change

admin :2019

admin 127.0.0.1:2019

Copilot · 2026-04-02T11:27:28Z

infra/Caddyfile.preview

+    on_demand_tls {
+        interval 2m
+        burst 5
+    }
+}
+


on_demand_tls is enabled without an ask/permission check to restrict which hostnames can obtain certs. This can be abused to trigger certificate issuance for arbitrary domains pointing at the box (rate limits/DoS). Add an ask endpoint / permission module to only allow *.preview.pubpub.org, or switch to a wildcard cert via DNS challenge.

Suggested change

on_demand_tls {

interval 2m

burst 5

}

}

on_demand_tls {

ask http://localhost:8080/allow-domain

interval 2m

burst 5

}

}

:8080 {

@allow expression {path} == "/allow-domain" && {query.domain}.matches(`^[A-Za-z0-9-]+\.preview\.pubpub\.org$`)

handle @allow {

respond "OK" 200

}

respond "Not allowed" 403

}

Copilot · 2026-04-02T11:27:29Z

infra/stack.preview.yml

+    minio:
+        image: minio/minio:latest
+        environment:
+            MINIO_ROOT_USER: minioadmin
+            MINIO_ROOT_PASSWORD: minioadmin
+        command: server --console-address ":9001" /data


MinIO is configured with the well-known default root credentials (minioadmin/minioadmin). Since the stack also exposes the console via the proxy, this is effectively public admin access. Set strong credentials via .env/SOPS and avoid committing defaults here.

Copilot · 2026-04-02T11:27:29Z

infra/Caddyfile.preview

+    handle_path /assets-ui* {
+        reverse_proxy minio:9001
+    }
+


The MinIO console is proxied publicly at /assets-ui*. Even with strong credentials, this increases attack surface for preview environments. Consider removing this route or protecting it with auth / IP allowlisting.

Suggested change

handle_path /assets-ui* {

reverse_proxy minio:9001

}

Copilot · 2026-04-02T11:27:31Z

.github/workflows/deploy.yml

+            echo "image_tag=${{ github.sha }}" >> $GITHUB_OUTPUT
+            echo "host=${{ secrets.SSH_HOST_STAGING }}" >> $GITHUB_OUTPUT
+            echo "env_file=.env.staging.enc" >> $GITHUB_OUTPUT


In the workflow_run (staging) path, image_tag is set to ${{ github.sha }} even though checkout uses ${{ github.event.workflow_run.head_sha }}. For workflow_run events these SHAs can differ, which can deploy an image tag that wasn't built/doesn't match the checked-out code. Use github.event.workflow_run.head_sha for image_tag (and ensure the env file name exists in-repo; .env.staging.enc is currently not present).

Suggested change

echo "image_tag=${{ github.sha }}" >> $GITHUB_OUTPUT

echo "host=${{ secrets.SSH_HOST_STAGING }}" >> $GITHUB_OUTPUT

echo "env_file=.env.staging.enc" >> $GITHUB_OUTPUT

echo "image_tag=${{ github.event.workflow_run.head_sha || github.sha }}" >> $GITHUB_OUTPUT

echo "host=${{ secrets.SSH_HOST_STAGING }}" >> $GITHUB_OUTPUT

echo "env_file=.env.enc" >> $GITHUB_OUTPUT

Copilot · 2026-04-02T11:27:31Z

.github/workflows/deploy.yml

+          tags: |
+            ghcr.io/pubpub/platform:${{ steps.vars.outputs.image_tag }}
+            ${{ steps.vars.outputs.publish_latest == 'true' && 'ghcr.io/pubpub/platform:latest' || '' }}
+          platforms: linux/amd64


The tags: input includes a conditional line that becomes an empty string when publish_latest is false. docker/build-push-action can fail on empty/invalid tags. Build the tag list so it only contains valid tags (e.g., compute tags in a prior step and pass a comma/newline list without blank entries).

Copilot · 2026-04-02T11:27:32Z

.github/workflows/ghcr-build-template.yml

+                  sha_short=$(git describe --always --abbrev=40 --dirty)
+
+                  if [[ -z "${{ inputs.package }}" ]]; then
+                      echo "target=monorepo" >> $GITHUB_OUTPUT
+                  else
+                      echo "target=${{ inputs.target || format('next-app-{0}', inputs.package) }}" >> $GITHUB_OUTPUT
+                  fi
+
+                  echo "label=ghcr.io/pubpub/${{ inputs.ghcr_image_name }}:$sha_short" >> $GITHUB_OUTPUT
+
+                  TAGS="ghcr.io/pubpub/${{ inputs.ghcr_image_name }}:$sha_short"
+                  if [[ "${{ inputs.publish_latest }}" == "true" ]]; then
+                      TAGS="$TAGS,ghcr.io/pubpub/${{ inputs.ghcr_image_name }}:latest"


image-sha is described as a SHA tag, but git describe can return non-SHA strings (e.g. v1.2.3-4-g<sha>). This becomes a problem now that other workflows deploy/pull images using raw commit SHAs. Prefer tagging images with the exact commit SHA (e.g. git rev-parse HEAD or ${{ github.sha }}) and keep deploy/e2e/preview workflows consistent.

Copilot · 2026-04-02T11:27:32Z

.github/workflows/deploy.yml

+    inputs:
+      skip_ci_check:
+        description: Deploy even if CI failed
+        required: false
+        default: false
+        type: boolean


skip_ci_check is defined as an input, but it isn't referenced anywhere in the workflow. Either wire it into the if: condition (so you can deploy after a failed CI run) or remove the input to avoid misleading callers.

Suggested change

inputs:

skip_ci_check:

description: Deploy even if CI failed

required: false

default: false

type: boolean

Copilot · 2026-04-02T11:27:32Z

infra/import-backup.sh

+echo "importing $DUMP_FILE into container $DB_CONTAINER ..."
+
+if [[ "$DUMP_FILE" == *.sql ]]; then
+    sudo docker exec -i "$DB_CONTAINER" \
+        psql -U "$PGUSER" -d "$PGDATABASE" < "$DUMP_FILE"
+else
+    sudo docker exec -i "$DB_CONTAINER" \
+        pg_restore --clean --if-exists --no-owner -U "$PGUSER" -d "$PGDATABASE" < "$DUMP_FILE"
+fi


This script uses set -u but relies on PGUSER/PGDATABASE being set in the environment. If the caller hasn't exported them, the script will exit with an unbound-variable error before running psql/pg_restore. Consider sourcing infra/.env (or accepting --pguser/--pgdatabase flags / defaulting to POSTGRES_USER/POSTGRES_DB) and validating the required values up-front.

github-actions · 2026-04-02T18:56:30Z

Preview deployed at https://pr-1432.preview.pubstar.org

feat: allow deploying to hetzner

4b5f107

tefkah requested review from 3mcd, Copilot and isTravis April 2, 2026 11:19

Copilot started reviewing on behalf of tefkah April 2, 2026 11:20 View session

chore: format

5769e32

Copilot AI reviewed Apr 2, 2026

View reviewed changes

tefkah added 6 commits April 2, 2026 13:31

fix: ci issue

070aa8d

fix: use sentry auth token

d83286b

fix: use sentry auth token

69ff995

fix: don't secret it like that

d18fd3f

fix: try and pass through sentry_Auth

9b073e2

fix: try again

e894fd2

tefkah added the preview Auto-deploys a preview application label Apr 2, 2026

tefkah added 7 commits April 2, 2026 20:08

fix: also filter on label

facea38

fix: push to kf

52b38e0

fix: fix even more

fe19959

fix: fix even more

307af6b

ugh

a35f173

fix: expedite testing

e5a56cd

fix: reencrypt

fd2d9ba

fix: wait for rollouot

72fe9de

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: allow deploying to hetzner#1432

feat: allow deploying to hetzner#1432
tefkah wants to merge 16 commits intomainfrom
tfk/hetzner-host

tefkah commented Apr 2, 2026 •

edited by isTravis

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-    on_demand_tls {
-        interval 2m
-        burst 5
-    }
-}
+    on_demand_tls {
+        ask http://localhost:8080/allow-domain
+        interval 2m
+        burst 5
+    }
+}
+:8080 {
+    @allow expression {path} == "/allow-domain" && {query.domain}.matches(`^[A-Za-z0-9-]+\.preview\.pubpub\.org$`)
+    handle @allow {
+        respond "OK" 200
+    }
+    respond "Not allowed" 403
+}

Conversation

tefkah commented Apr 2, 2026 • edited by isTravis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue(s) Resolved

High-level Explanation of PR

Test Plan

Screenshots (if applicable)

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tefkah commented Apr 2, 2026 •

edited by isTravis

Loading