:bug: Starting background processes in a container is fragile

[grs]

Describe the bug

If the background process exits, it will not be started. At present the ext_proc server is run in the background of the envoy proxy server, meaning that if that ext_proc server process exits for any reason, the pod will remain in a broken state until manually restarted.

There is also a combined container that includes container registration and spiffe-helper processes as well, though that is only used when explicitly enabled.

Steps To Reproduce

This issue arose from an inspection of the code. (You could observe the problem artificially by killing the background process in a container and observing that the container as a whole is not restarted).

Expected Behavior

Whenever an important process exits, it should be automatically restarted.

Additional Context

No response

[huang195]

A well known pattern to deal with this type of issue in containers is to use something like systemd or supervisord to start other processes as services and manage their lifecycle. If one of the managed processes dies, the manager will restart it. If the manager dies, the pod will get restarted.

[huang195]

Analysis

Both entrypoint scripts start critical long-running processes in the background with & and no supervision:

Entrypoint	Background process	If it dies...
entrypoint-envoy.sh:5	go-processor &	ext_proc calls fail silently; Envoy stays up, pod looks healthy
entrypoint-authbridge.sh:25	spiffe-helper &	JWT SVIDs stop renewing, tokens eventually expire
entrypoint-authbridge.sh:32	go-processor &	Same as envoy case

Envoy is PID 1 via exec, so Kubernetes only detects failure if Envoy itself exits. Background processes are invisible to the kubelet.

Options

Option A: Shell-based wait loop (recommended)

Replace exec envoy with a pattern that starts all processes in the background, then uses wait -n to detect when any child exits. If a critical process dies, kill the others and exit non-zero so Kubernetes restarts the container.

envoy -c /etc/envoy/envoy.yaml ... &
ENVOY_PID=$!
go-processor &
GP_PID=$!

trap 'kill $ENVOY_PID $GP_PID 2>/dev/null; exit 0' TERM INT

# Wait for ANY child to exit
wait -n
echo "A process exited unexpectedly, terminating container"
kill $ENVOY_PID $GP_PID 2>/dev/null
exit 1

Pros: No new dependencies, small diff, works in both images.
Cons: No individual process restart — if go-processor crashes, the entire container restarts (Envoy cold-start). Requires bash (wait -n is not POSIX sh). Loses exec for Envoy, so signal forwarding needs a trap.

Option B: supervisord

Add supervisord as PID 1 with each process as a supervised program. Restarts individual processes without killing the whole container.

Pros: Individual process restart (go-processor can recover without restarting Envoy), battle-tested, configurable restart policies and backoff.
Cons: Adds a dependency (~5-10MB). The envoy-only image (Dockerfile.envoy) uses ubi-minimal which has no Python — would need a base image change or a Go-based alternative. The combined sidecar image already has Python so supervisord works there easily.

Recommendation

Going with Option A — it's the lowest overhead fix with no new dependencies. A full container restart on go-processor crash is acceptable since it should be rare, and a clean restart is a reliable recovery. The entrypoint scripts need to switch from #!/bin/sh to #!/bin/bash for wait -n support.