Skip to content

Local: analytics container boots against an unmigrated DB — entrypoint ignores failed Logflare.Release.migrate and never waits for supabase_db #6088

Description

@unhingedpanda

Describe the bug

The generated entrypoint for the supabase_analytics (Logflare) container runs migrations and then starts the app unconditionally, with no set -e, no exit-code check, and no wait for supabase_db readiness:

sh -c "cat <<'EOF' > run.sh && sh run.sh
./logflare eval Logflare.Release.migrate
./logflare start --sname logflare
EOF
"

(Verified via docker inspect --format '{{json .Config.Entrypoint}}' on the running container, and the literal ./logflare eval Logflare.Release.migrate is present in the CLI binary.)

If the migrate step loses the race against Postgres accepting connections, its failure is silently swallowed and Logflare boots against an unmigrated database. Oban's producers then die on the missing public.oban_jobs, which terminates the whole application, and the container restarts into the same state.

Because migrations only run at container start, the same end state is reached whenever the database is recreated underneath a still-running analytics container — nothing ever re-runs them.

To Reproduce

  1. supabase start with analytics enabled (the default).
  2. Cause the analytics container to (re)start while supabase_db is not yet accepting connections — or recreate/wipe the _supabase database underneath a running analytics container.
  3. docker logs supabase_analytics_<project>.

Observed, in order, in a single container's logs:

[error] #PID<0.6872.0> (Oban.Notifiers.Postgres) failed to connect to Postgres:
  ** (DBConnection.ConnectionError) tcp connect (supabase_db_<project>:5432):
     connection refused - :econnrefused

then, once it is up but unmigrated:

[error] GenServer {Oban.Registry, {Oban, {:producer, "default"}}} terminating
** (Postgrex.Error) ERROR 42P01 (undefined_table) relation "public.oban_jobs" does not exist
    (oban 2.20.3) lib/oban/engines/basic.ex:127: anonymous fn/4 in Oban.Engines.Basic.fetch_jobs/3
    (logflare 1.46.0) lib/logflare/repo.ex:2: anonymous fn/1 in Logflare.Repo.transaction/2
Last message: :dispatch

Kernel pid terminated (application_controller) ("{application_terminated,logflare,shutdown}")

Oban's two tables fail asymmetrically, which is what escalates this from degraded to fatal — oban_peers degrades gracefully:

[error] The `oban_peers` table is undefined and leadership is disabled.
Run migrations up to v11 to restore peer leadership.

while a missing oban_jobs kills the producer and takes the application down with it.

In my case the container reached RestartCount=11, with _supabase left holding zero tables in both public and _analytics.

Expected behavior

The analytics container should not start Logflare unless migrations actually succeeded, and should not attempt them before supabase_db is ready.

This is not an image or dependency regression

Worth stating explicitly, since it looks like one at first glance. With the same logflare:1.46.0 image and the same Oban 2.20.3, starting the container against an already-healthy supabase_db succeeds cleanly in ~20s and the migrations are complete and correctly placed:

  • _analytics → 37 tables, _analytics.schema_migrations head 20260618090000
  • publicoban_jobs, oban_peers

So the migration definitions are fine and the schema split is fine. The defect is purely the ordering/error-handling in the generated entrypoint.

Suggested fix

set -e
until pg_isready -h "$DB_HOSTNAME" -p "$DB_PORT" -U "$DB_USERNAME"; do sleep 1; done
./logflare eval Logflare.Release.migrate
./logflare start --sname logflare

set -e alone fixes the silent fallthrough; the readiness loop fixes the race. Beyond that:

  • A health-gated dependency on supabase_db so analytics isn't started concurrently in the first place.
  • supabase db reset should restart the analytics container, so migrations re-run against the recreated database.

Secondary observations

Orphaned vector work. Every sink in the generated vector config POSTs to http://supabase_analytics_<project>:4000/api/logs. When analytics is down, vector keeps scraping all container logs, running its full transform chain, and retrying HTTP into a void indefinitely — in my stack it was the largest container by memory (536 MiB) while its only consumer was dead. It would be reasonable for vector to back off, or for the CLI to stop it alongside analytics.

Crashloop never converges. A container failing identically many times in a row restarts hot rather than backing off, which is a noticeable battery/CPU cost on laptops. I suspect the boot survives just long enough each cycle to reset Docker's restart backoff, so the backoff never accumulates — but I want to flag that I could not verify that mechanism cleanly (the container was manually killed partway through my investigation, so my cycle-duration measurement is unreliable). The non-convergence itself is real; the explanation is a hypothesis.

System information

  • Supabase CLI: 2.111.0
  • Image: public.ecr.aws/supabase/logflare:1.46.0
  • Elixir 1.19.5, stdlib 6.2.2.2, ecto 3.13.5, ecto_sql 3.13.4, oban 2.20.3, db_connection 2.9.0, broadway 1.2.1, gen_stage 1.3.1
  • Postgres image: public.ecr.aws/supabase/postgres:17.6.1.156
  • Docker 29.6.2 (build dfc4efb), Docker Desktop 4.84.0
  • macOS 26.6 (25G72), Apple M5 Pro, arm64

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions