Skip to content

Scheduler: one throwing schedule aborts the whole tick and starves all later schedules #552

Description

@LivinTribunal

Severity: P1 — one bad schedule permanently starves every other schedule.

What happens

If any schedule's cron spec throws during processing (e.g. a cron stored before validation existed, or corrupted data), the whole scheduler tick aborts — every schedule sorted after the bad one is never processed. With the bad schedule overdue (sorted first), valid schedules are starved indefinitely.

Root cause

pyworkflow/scheduler/local.py:158-167: _process_schedule's except handler re-calls calculate_next_run_time(...) on the same failing spec (line 166). That second call re-raises; the exception escapes _process_schedule and there is no per-schedule guard in the for schedule in due_schedules loops (_tick :121-122, tick_once :245-246) — only the whole-tick guard in run() (:103-106). One throwing schedule = the rest of the tick is dead.

Executed evidence (deterministic, v0.3.8; also reproduces with croniter 6.2.2)

File storage with two schedules — one corrupted on disk to cron 5-1/0 * * * * (raises at construction), one valid * * * * *, both due; single tick_once():

ordering valid schedule fired
corrupt sorted first 0 times (starved)
valid sorted first 1 time, then tick aborts

tick_once raised CroniterBadCronError out of the loop in both cases.

Suggested fix

Wrap each schedule in its own try/except inside the due-schedules loop, and don't recompute next_run_time in the error handler for a spec that just failed — mark the schedule errored/paused instead so it can't wedge the scheduler.

Related but distinct: #553 (trigger_schedule executes the workflow before stats/next-run are persisted, which turns this starvation into runaway re-execution of the bad schedule's workflow).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions