Skip to content

Local runtime / file storage hygiene: resume() never advances durable sleep; soft-deleted schedules never purged (case mismatch); .locks/ grows unbounded #557

Description

@LivinTribunal

Three small, independently-verified defects in the local runtime / file storage, bundled to keep the tracker tidy. All reproduced with executed scripts on v0.3.8.

1. Local-runtime resume() can never advance a durable sleep()

Each resume() recomputes the sleep deadline from "now" (context/local.py:783 resume_at = now + duration), so the elapsed check at :786 is always False and it re-raises SuspensionSignal (:803) — no matter how much wall-clock time has passed. The originally recorded deadline (SLEEP_STARTED resume_at, loaded into pending_sleeps at replay.py:135) is never consulted, and SLEEP_COMPLETED is never emitted, so the _completed_sleeps short-circuit (:778) never fires.

Executed: sleep("3s") workflow, resumed 5 s past the deadlinestatus=suspended, zero progress; second resume identical. The shipped example examples/local/durable/04_long_running.py run as-is prints "Workflow completed! Result: None" while Final status: suspended — the success message is misleading.

Fix sketch: on resume, honor the recorded resume_at from the event log instead of recomputing.

2. delete_old_runs never purges soft-deleted schedules (case mismatch)

storage/file.py:1046 tests status == "DELETED", but the persisted enum value is "deleted" (ScheduleStatus.DELETED = "deleted", schemas.py:61) — the branch runs but never matches, so soft-deleted schedules are immortal on disk. Terminal runs are purged correctly (correct lowercase set at file.py:975). Reachable in production via the Celery cleanup task (celery/tasks.py:2979).

Executed: completed run → purged ✓; soft-deleted schedule (status='deleted' on disk) → not purged ✗; manually flipping the file to "DELETED" → purged, proving the case mismatch is the sole cause.

3. .locks/ directories grow unbounded

With filelock ≥ 3.29.5, lock files persist after release by design (upstream #577). FileStorageBackend creates one lock per run/entity and nothing in storage/file.py ever unlinks them (grep: no cleanup path). Executed: 0 → 10 lock files after 5 sequential runs (2 per run), linear growth, never reclaimed. Harmless functionally (verified: reacquisition over stale files works, incl. after kill -9), but unbounded inode growth on long-lived storage dirs.

Fix sketch: unlink lock files in delete_old_runs, or periodic sweep of .locks/ for entries with no matching run.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions