Scheduler becomes an un-replaced zombie after host sleep/resume; recurring jobs silently stop

[julik]

Version: solid_queue 1.3.1 · Rails 8 · Puma plugin (SOLID_QUEUE_IN_PUMA=true) · single host · SQLite queue DB

Summary

After the host sleeps and resumes, the scheduler ends up alive at the OS level but deregistered from solid_queue_processes, with its scheduling thread dead. The fork supervisor only replaces children that exit, so it never replaces it, and no recurring jobs are enqueued again until a manual restart. Dispatcher and worker recover normally, so ad-hoc jobs keep running and hide the outage.

This is the scenario PR #337 targeted ("a computer going to sleep and then coming back to life"), but the self-heal it added doesn't cover the scheduler here. No existing issue ties the sleep/resume trigger to the scheduler-zombie outcome end-to-end.

In practice this means that a Rails app running on a laptop that sleeps and then awakens stops running its scheduled tasks until restarted.

Evidence

Last calendar_sync recurring execution: 2026-06-30 19:30 UTC; nothing for days after, while ad-hoc jobs kept running and no FailedExecution rows were created.

solid_queue_processes has no Scheduler row — only supervisor/dispatcher/worker (the latter two re-forked after the sleep). But ps shows the scheduler still alive:

61362 solid-queue-fork-supervisor(1.3.1): supervising 61365, 43988, 43991
61365 solid-queue-scheduler(1.3.1): scheduling calendar_sync,...

Pid 61365 is alive and supervised but has no DB registration → zombie.

Log around the sleep boundary:

Deregister Scheduler   process_id: 5403, pid: 61365, last_heartbeat_at: "2026-06-30T19:11:08Z", pruned: true
Error enqueuing recurring task   task: "calendar_sync", at: "2026-06-30T19:12:00Z"

After this: no further Register Scheduler and no Replaced terminated Scheduler. Dispatcher/worker get Replaced terminated … repeatedly across the same period; the scheduler only ever gets Shutdown Scheduler.

Root cause

1. During sleep no process heartbeats. On resume, the dispatcher's prune sweep deletes the others' stale Process rows, including the scheduler's.
2. Sleep freezes rather than kills: the scheduler resumes with its row already pruned, so its next enqueue errors and its scheduling thread dies.
3. The supervisor decides replacement by OS process exit, not by DB registration. An alive-but-deregistered child is never replaced.

Why #337's self-heal misses it: the RecordNotFound self-heal in Registrable#heartbeat only fires if the heartbeat thread ticks. Here the scheduling thread dies first via the unrescued enqueue error (recurring_schedule.rb doesn't rescue arbitrary exceptions); if the heartbeat timer task also errors, nothing is left to trip the self-heal.

Prior art

• Handle scenarios where registered processes disappear outside regular flow #337 (merged, in 1.3.1) — names this trigger; adds the self-heal that doesn't cover the scheduler here.
• Supervisor remains alive with missing worker process; due scheduled jobs stop being dispatched until restart #751 (open, 1.4.0) — supervisor alive, a child missing, scheduled jobs stall until restart. Same shape, reproduces after Handle scenarios where registered processes disappear outside regular flow #337.
• RecurringTask#enqueue does not report rescued Job::EnqueueError to Rails.error #746 (open, 1.4.0) — scheduler swallows the enqueue error (forwards only error.message), so nothing surfaces. The "silent" half.

Reproduction

1. Run with a scheduler + a recurring task (e.g. every minute), via the Puma plugin or bin/jobs.
2. Suspend the host (or SIGSTOP the process group / pause the VM) longer than process_alive_threshold (default 5 min).
3. Resume.
4. Observe: Deregister Scheduler … pruned: true, Error enqueuing recurring task, then no Register Scheduler/Replaced terminated Scheduler. Scheduler process stays alive, solid_queue_processes has no Scheduler row, recurring jobs never enqueue again until restart.

Suggested directions

• Make any process that finds its own registration gone tear down the whole process (so the supervisor re-forks it), not just one thread.
• Don't let an enqueue error permanently kill the scheduling thread — rescue/report and keep scheduling (ties to RecurringTask#enqueue does not report rescued Job::EnqueueError to Rails.error #746).
• Optionally, let the supervisor treat a child whose registration vanished as replaceable (ties to Supervisor remains alive with missing worker process; due scheduled jobs stop being dispatched until restart #751).

Note: process_alive_threshold / heartbeat interval are global (#346), so no config reliably prevents this.