Skip to content

🐛 Supervise background processes in AuthBridge entrypoints#308

Merged
huang195 merged 1 commit into
rossoctl:mainfrom
huang195:fix/supervise-background-processes
Apr 14, 2026
Merged

🐛 Supervise background processes in AuthBridge entrypoints#308
huang195 merged 1 commit into
rossoctl:mainfrom
huang195:fix/supervise-background-processes

Conversation

@huang195

@huang195 huang195 commented Apr 14, 2026

Copy link
Copy Markdown
Member

Summary

  • Replace fragile exec envoy pattern with wait -n supervision loop in both entrypoint scripts
  • Shell stays PID 1 and monitors all critical background processes (go-processor, spiffe-helper, envoy)
  • If any critical process exits, the container terminates so Kubernetes restarts it
  • Client-registration remains unmonitored (one-shot, failure is non-fatal by design)

Fixes #278

Test plan

  • Build both Docker images (Dockerfile.envoy, Dockerfile.authbridge) and verify they start correctly
  • Deploy to Kind cluster, verify all processes are running
  • Kill go-processor inside the envoy-proxy container — container should restart
  • Kill go-processor inside the combined sidecar container — container should restart
  • Send SIGTERM to the container — all processes should terminate cleanly (graceful shutdown)

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

Background processes (go-processor, spiffe-helper) started with & in
container entrypoints were invisible to Kubernetes — if they died, the
container stayed running in a broken state. Replace `exec envoy` with
a wait -n supervision loop: all critical processes run in the background,
the shell stays PID 1, and if any child exits the container terminates
so Kubernetes restarts it.

Fixes rossoctl#278

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>

@cwiklik cwiklik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Great fix for process supervision — replacing exec envoy with a wait -n loop correctly addresses the invisible background process failure problem (#278). The pattern is clean: shell stays PID 1, monitors critical children, trap handles graceful shutdown.

However, entrypoint-envoy.sh will fail at runtime because its container image (Dockerfile.envoy) is based on ubi9/ubi-minimal which does not include bash.

Areas reviewed: Shell scripts, Dockerfiles (base image compatibility)
Commits: 1 commit, signed-off ✓
CI status: all passing (Shell Script Lint doesn't catch runtime availability)

@@ -1,13 +1,37 @@
#!/bin/sh
#!/bin/bash

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must-fix: #!/bin/bash + wait -n will fail at runtime. Dockerfile.envoy uses ubi9/ubi-minimal which only installs ca-certificates — no bash. The container will crash on startup.

Two options:

  1. Add microdnf install -y bash to Dockerfile.envoy
  2. Rewrite using POSIX sh (e.g., background both processes, poll with kill -0 in a loop)

Option 1 is simpler but adds ~5MB to the image. Option 2 keeps the minimal image but is more complex.

entrypoint-authbridge.sh is fine — Dockerfile.authbridge is based on envoyproxy/envoy (Debian), which includes bash.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ubi9/ubi-minimal ships with bash 5.1.8 out of the box — no microdnf install needed:

$ podman run --rm ubi9/ubi-minimal bash --version
GNU bash, version 5.1.8(1)-release (aarch64-redhat-linux-gnu)

Verified end-to-end on a Kind cluster: built the image, deployed the quickstart demo, confirmed both processes start under the supervision loop, killed go-processor inside the container, and watched Kubernetes restart it. No issues.

echo "[AuthBridge] Starting go-processor..."
/usr/local/bin/go-processor &
CRITICAL_PIDS="$CRITICAL_PIDS $!"
sleep 2

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Clean pattern — CRITICAL_PIDS accumulation, wait -n, trap with cleanup. Client-registration correctly excluded from monitoring. The ${SPIRE_ENABLED:-} default is a good defensive fix too.

@pdettori pdettori left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional issue not covered by prior reviews: the envoy entrypoint has an uninitialized-variable race in the cleanup handler.

Areas reviewed: Signal handling, variable lifecycle under set -u
Commits: 1 commit, signed-off ✓
CI status: all passing


cleanup() {
echo "Received signal, shutting down..."
kill "$GO_PROCESSOR_PID" "$ENVOY_PID" 2>/dev/null || true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must-fix: GO_PROCESSOR_PID and ENVOY_PID are referenced here but not yet defined when the trap is installed (line 15). If SIGTERM arrives before both PID assignments (lines 20 and 29), set -u triggers a fatal unbound-variable error that bypasses || true — the error occurs at variable expansion, before kill runs.

Initialize both to empty before the trap, matching the CRITICAL_PIDS="" pattern in entrypoint-authbridge.sh:

GO_PROCESSOR_PID=""
ENVOY_PID=""

cleanup() {
  ...
}
trap cleanup TERM INT

@cwiklik
cwiklik dismissed their stale review April 14, 2026 20:14

My mistake — ubi9/ubi-minimal ships with bash 5.1.8 out of the box. Verified by PR author on a Kind cluster end-to-end. Dismissing and approving.

@cwiklik cwiklik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary (updated)

My earlier REQUEST_CHANGES was incorrect — ubi9/ubi-minimal ships with bash 5.1.8 out of the box. Apologies for the false flag.

The supervision pattern is solid: shell stays PID 1, wait -n monitors critical children, trap handles graceful SIGTERM shutdown. Both entrypoint scripts are consistent in approach. Client-registration correctly excluded from monitoring (one-shot, non-fatal). The ${SPIRE_ENABLED:-} default fixes are a good defensive improvement.

Author verified end-to-end on Kind: built image, deployed quickstart, killed go-processor, confirmed Kubernetes restart.

Areas reviewed: Shell scripts, Dockerfiles
Commits: 1 commit, signed-off ✓
CI status: all passing

@huang195
huang195 merged commit bef33e2 into rossoctl:main Apr 14, 2026
15 checks passed
@huang195
huang195 deleted the fix/supervise-background-processes branch April 14, 2026 20:22
huang195 added a commit to huang195/kagenti-extensions that referenced this pull request Apr 15, 2026
The Dockerfile now produces a single image containing both Envoy and the
authbridge binary, a true drop-in replacement for envoy-with-processor.
The entrypoint uses wait -n process supervision (same pattern as PR rossoctl#308).

Key changes:
- Multi-stage build: Go builder, Envoy binary, ubi9-minimal runtime
- entrypoint.sh starts authbridge then Envoy, monitors both with wait -n
- Runs as UID 1337 (required by proxy-init iptables rules)
- Same base image (ubi9-minimal) and ports as Dockerfile.envoy

Verified E2E on Kind cluster: inbound JWT validation, outbound RFC 8693
token exchange, bypass paths, and process supervision all working.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
huang195 added a commit to huang195/kagenti-extensions that referenced this pull request Apr 15, 2026
The Dockerfile now produces a single image containing both Envoy and the
authbridge binary, a true drop-in replacement for envoy-with-processor.
The entrypoint uses wait -n process supervision (same pattern as PR rossoctl#308).

Key changes:
- Multi-stage build: Go builder, Envoy binary, ubi9-minimal runtime
- entrypoint.sh starts authbridge then Envoy, monitors both with wait -n
- Runs as UID 1337 (required by proxy-init iptables rules)
- Same base image (ubi9-minimal) and ports as Dockerfile.envoy

Verified E2E on Kind cluster: inbound JWT validation, outbound RFC 8693
token exchange, bypass paths, and process supervision all working.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
huang195 added a commit to huang195/kagenti-extensions that referenced this pull request Apr 15, 2026
The Dockerfile now produces a single image containing both Envoy and the
authbridge binary, a true drop-in replacement for envoy-with-processor.
The entrypoint uses wait -n process supervision (same pattern as PR rossoctl#308).

Key changes:
- Multi-stage build: Go builder, Envoy binary, ubi9-minimal runtime
- entrypoint.sh starts authbridge then Envoy, monitors both with wait -n
- Runs as UID 1337 (required by proxy-init iptables rules)
- Same base image (ubi9-minimal) and ports as Dockerfile.envoy

Verified E2E on Kind cluster: inbound JWT validation, outbound RFC 8693
token exchange, bypass paths, and process supervision all working.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
huang195 added a commit to huang195/kagenti-extensions that referenced this pull request Apr 15, 2026
The Dockerfile now produces a single image containing both Envoy and the
authbridge binary, a true drop-in replacement for envoy-with-processor.
The entrypoint uses wait -n process supervision (same pattern as PR rossoctl#308).

Key changes:
- Multi-stage build: Go builder, Envoy binary, ubi9-minimal runtime
- entrypoint.sh starts authbridge then Envoy, monitors both with wait -n
- Runs as UID 1337 (required by proxy-init iptables rules)
- Same base image (ubi9-minimal) and ports as Dockerfile.envoy

Verified E2E on Kind cluster: inbound JWT validation, outbound RFC 8693
token exchange, bypass paths, and process supervision all working.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
huang195 added a commit to huang195/kagenti-extensions that referenced this pull request Apr 15, 2026
The Dockerfile now produces a single image containing both Envoy and the
authbridge binary, a true drop-in replacement for envoy-with-processor.
The entrypoint uses wait -n process supervision (same pattern as PR rossoctl#308).

Key changes:
- Multi-stage build: Go builder, Envoy binary, ubi9-minimal runtime
- entrypoint.sh starts authbridge then Envoy, monitors both with wait -n
- Runs as UID 1337 (required by proxy-init iptables rules)
- Same base image (ubi9-minimal) and ports as Dockerfile.envoy

Verified E2E on Kind cluster: inbound JWT validation, outbound RFC 8693
token exchange, bypass paths, and process supervision all working.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
huang195 added a commit to huang195/kagenti-extensions that referenced this pull request Apr 15, 2026
The Dockerfile now produces a single image containing both Envoy and the
authbridge binary, a true drop-in replacement for envoy-with-processor.
The entrypoint uses wait -n process supervision (same pattern as PR rossoctl#308).

Key changes:
- Multi-stage build: Go builder, Envoy binary, ubi9-minimal runtime
- entrypoint.sh starts authbridge then Envoy, monitors both with wait -n
- Runs as UID 1337 (required by proxy-init iptables rules)
- Same base image (ubi9-minimal) and ports as Dockerfile.envoy

Verified E2E on Kind cluster: inbound JWT validation, outbound RFC 8693
token exchange, bypass paths, and process supervision all working.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
huang195 added a commit to huang195/kagenti-extensions that referenced this pull request Apr 15, 2026
The Dockerfile now produces a single image containing both Envoy and the
authbridge binary, a true drop-in replacement for envoy-with-processor.
The entrypoint uses wait -n process supervision (same pattern as PR rossoctl#308).

Key changes:
- Multi-stage build: Go builder, Envoy binary, ubi9-minimal runtime
- entrypoint.sh starts authbridge then Envoy, monitors both with wait -n
- Runs as UID 1337 (required by proxy-init iptables rules)
- Same base image (ubi9-minimal) and ports as Dockerfile.envoy

Verified E2E on Kind cluster: inbound JWT validation, outbound RFC 8693
token exchange, bypass paths, and process supervision all working.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Hai Huang <huang195@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Starting background processes in a container is fragile

3 participants