From 4ba3063e0ba60eb1f8a90abe7eb6d54d720767e8 Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Thu, 16 Jul 2026 10:54:53 -0600 Subject: [PATCH 1/9] Document the built-in scheduler component plugin New reference page for the scheduler (config surface, cron syntax, intervals, timezone/DST semantics, handler contract and idempotency, cluster leader election and catch-up behavior), a row in the built-in components table, sidebar entry, and a 5.2 release-notes entry. Companion to HarperFast/harper#1828. Co-Authored-By: Claude Fable 5 --- reference/components/overview.md | 1 + reference/components/scheduler.md | 105 ++++++++++++++++++++++++++++++ release-notes/v5-lincoln/5.2.md | 6 ++ sidebarsReference.ts | 5 ++ 4 files changed, 117 insertions(+) create mode 100644 reference/components/scheduler.md diff --git a/reference/components/overview.md b/reference/components/overview.md index d680d2a3..ffabca3f 100644 --- a/reference/components/overview.md +++ b/reference/components/overview.md @@ -131,6 +131,7 @@ Extensions require an `extensionModule` option pointing to the extension source. | [`loadEnv`](../environment-variables/overview.md) | Load environment variables from `.env` files | | [`rest`](../rest/overview.md) | Enable automatic REST endpoint generation | | [`roles`](../users-and-roles/overview.md) | Define role-based access control from YAML files | +| [`scheduler`](./scheduler.md) | Run recurring jobs on a cron or interval schedule | | [`static`](../static-files/overview.md) | Serve static files via HTTP | ## Known Custom Components diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md new file mode 100644 index 00000000..35e09110 --- /dev/null +++ b/reference/components/scheduler.md @@ -0,0 +1,105 @@ +--- +title: Scheduler +--- + +# Scheduler + + + +The scheduler is a built-in plugin that runs recurring jobs declared in a component's configuration. Harper invokes a designated export from your component on a cron or interval schedule, and in a cluster each job fires exactly once per occurrence - on a single, automatically elected node - rather than once per node or worker. + +Use it for the maintenance passes applications otherwise hand-roll with `setInterval`: nightly cleanups, periodic re-aggregation, cache refreshes, digest generation. + +## Configuration + +In your component's `config.yaml`, use the `scheduler` key to declare jobs: + +```yaml +scheduler: + jobs: + - name: nightly-cleanup + cron: '0 2 * * *' + timezone: America/New_York + handler: ./jobs.js#cleanupOldRecords + - name: refresh-summaries + interval: 90s + handler: ./jobs.js#refreshSummaries +``` + +Each job entry supports: + +### `name` + +Type: `string` (required) + +A name for the job, unique within the component. Used in logs and to key the job's run state. + +### `cron` + +Type: `string` + +A five-field cron expression: minute, hour, day-of-month, month, day-of-week. Supports `*`, lists (`1,15`), ranges (`9-17`), steps (`*/15`, `0-59/5`), month and day names (`JAN`, `MON-FRI`), and the common macros (`@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@yearly`). Day-of-week `0` and `7` both mean Sunday. Following the POSIX cron rule, when both day-of-month and day-of-week are restricted, the job runs when either matches. + +There is no seconds field. Expressions that can never fire (such as `0 0 30 2 *` - February 30th) are rejected when the component loads. + +Exactly one of `cron` or `interval` is required. + +### `interval` + +Type: `string` (duration) + +A simple cadence instead of a cron expression: a number of seconds, or a value like `90s`, `5m`, `1h`, `1d`. Minimum one second. Use this for "every N" maintenance passes that do not align to wall-clock times. + +### `timezone` + +Type: `string` + +An IANA timezone (for example `America/Chicago`) the cron expression is evaluated in. Defaults to the server's timezone. Only valid with `cron`. + +During daylight-saving transitions: a job scheduled inside the spring-forward gap (a wall-clock time that does not exist that day) runs at the shifted instant rather than being skipped, and a job scheduled inside the fall-back overlap (a wall-clock time that occurs twice) runs once, at the first occurrence. + +### `handler` + +Type: `string` (required) + +The module and export to invoke, as `#`, relative to the component directory - for example `./jobs.js#cleanupOldRecords`. Omit the `#...` suffix to use the module's default export. The module is loaded in your component's environment, so it has access to the same globals (`tables`, `databases`, `server`, and so on) as the rest of your component code. + +A bad handler reference (missing module, missing export, non-function export) fails the component load at deploy time rather than failing silently at the first scheduled fire. + +## Writing a Handler + +The handler is called with a single context argument and may return a promise: + +```javascript +export async function cleanupOldRecords(context) { + // context.jobName - the configured job name + // context.scheduledAt - the occurrence this run is for (Date) + // context.catchUp - true when this run is making up a missed occurrence + const cutoff = Date.now() - 30 * 24 * 60 * 60 * 1000; + for await (const record of tables.Events.search([{ attribute: 'timestamp', comparator: 'less', value: cutoff }])) { + await tables.Events.delete(record.id); + } +} +``` + +**Handlers should be idempotent.** Harper's clustering has no distributed lock, so leadership failover and daylight-saving fall-back can occasionally deliver the same logical occurrence twice. Design handlers so that running twice for one occurrence is harmless. + +Runs of the same job never overlap: if a run is still going when its next occurrence arrives, that occurrence is skipped (with a log entry) rather than stacked. + +## Cluster Behavior + +One node in the cluster - the scheduler leader - runs all scheduled jobs; the others watch. Leader election is automatic and requires no configuration: + +- The leader maintains a heartbeat in the replicated `system` database. If it stops heartbeating (crash, shutdown, partition) for more than five minutes, the next node in line promotes itself; with default timings, failover completes within about six and a half minutes. +- A restarting node defers to an actively heartbeating leader, so leadership is stable across deploys and restarts. +- On a single-node instance, that node simply runs the jobs. + +### Missed Occurrences + +If the most recent occurrence of a cron job was missed - the leader was down, a failover was in progress, or daylight-saving time skipped the slot - the scheduler fires one catch-up run for it (with `context.catchUp` set to `true`). Only the single most recent missed occurrence is made up, not every occurrence missed during an outage. Interval jobs resume their cadence from their last recorded run. + +A newly deployed job waits for its first scheduled time; it does not fire immediately on deploy. + +### Run State + +Each job's last run time, status, duration, and last error (if any) are recorded in the `hdb_scheduler_state` system table, alongside the leader lease. This state replicates across the cluster so a newly promoted leader knows what has already run. diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 8cea990a..6405d6f9 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -14,6 +14,12 @@ All patch release notes for 5.2.x are available on the [releases page](https://g Vector searches combined with filters now evaluate the filter during HNSW graph traversal, so the query keeps exploring until it has enough matching nearest neighbors instead of post-filtering a fixed candidate set (which under-filled results under selective filters). Filters can come from query conditions, a JS-API `vectorFilter` function, or a record-scoped `allowRead` override — overriding `allowRead` on a table now makes it a row-level access check, evaluated per record with `this` bound to the record (closing the gap where a collection scan could return rows a single-record GET would deny). With it, a restricted user's vector search returns the k nearest records they are allowed to see. Very selective conditions automatically use an exact scan instead of graph traversal, and a `filterExpansion` visit budget bounds traversal cost. See [Vector Indexing](/reference/v5/database/schema#vector-indexing). +## Components + +### Scheduler: Recurring Jobs from Component Config + +Components can now declare recurring jobs in their configuration with a new built-in `scheduler` plugin. Jobs run on a five-field cron expression or a simple interval (`90s`, `5m`, `1h`), invoking a designated export from the component. In a cluster, each job fires exactly once per occurrence on an automatically elected leader node, with heartbeat-based failover, catch-up for missed occurrences, and per-job run state recorded in a replicated system table. See [Scheduler](/reference/v5/components/scheduler). + ## Configuration ### Replicated `set_configuration` diff --git a/sidebarsReference.ts b/sidebarsReference.ts index 63957800..b17f475e 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -158,6 +158,11 @@ const sidebars: SidebarsConfig = { id: 'components/javascript-environment', label: 'JavaScript Environment', }, + { + type: 'doc', + id: 'components/scheduler', + label: 'Scheduler', + }, { type: 'doc', id: 'components/nextjs', From 0ded9872f985d5498b31de44696227827bbc2e3c Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Thu, 16 Jul 2026 11:03:00 -0600 Subject: [PATCH 2/9] Use snapshot and external-API examples instead of record cleanup Record cleanup is built into Harper (table expiration/eviction), so it was a misleading flagship example; point readers there instead and demonstrate the scheduler with jobs only it can do: a daily dataset snapshot (which also demonstrates idempotent handler design) and a scheduled external-API pull. Co-Authored-By: Claude Fable 5 --- reference/components/scheduler.md | 37 ++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md index 35e09110..6592df20 100644 --- a/reference/components/scheduler.md +++ b/reference/components/scheduler.md @@ -8,7 +8,9 @@ title: Scheduler The scheduler is a built-in plugin that runs recurring jobs declared in a component's configuration. Harper invokes a designated export from your component on a cron or interval schedule, and in a cluster each job fires exactly once per occurrence - on a single, automatically elected node - rather than once per node or worker. -Use it for the maintenance passes applications otherwise hand-roll with `setInterval`: nightly cleanups, periodic re-aggregation, cache refreshes, digest generation. +Use it for the recurring work applications otherwise hand-roll with `setInterval`: taking daily snapshots of a dataset, pulling from an external API on a schedule, re-aggregating summary tables, generating digests or reports. + +For expiring old records, prefer the table-level [`expiration` and `eviction` options](../database/schema.md#table) - record cleanup is built into Harper and does not need a scheduled job. ## Configuration @@ -17,13 +19,13 @@ In your component's `config.yaml`, use the `scheduler` key to declare jobs: ```yaml scheduler: jobs: - - name: nightly-cleanup + - name: daily-metrics-snapshot cron: '0 2 * * *' timezone: America/New_York - handler: ./jobs.js#cleanupOldRecords - - name: refresh-summaries - interval: 90s - handler: ./jobs.js#refreshSummaries + handler: ./jobs.js#snapshotMetrics + - name: sync-exchange-rates + interval: 15m + handler: ./jobs.js#syncExchangeRates ``` Each job entry supports: @@ -71,14 +73,29 @@ A bad handler reference (missing module, missing export, non-function export) fa The handler is called with a single context argument and may return a promise: ```javascript -export async function cleanupOldRecords(context) { +// Nightly snapshot: roll the current state of a table into a dated snapshot record +export async function snapshotMetrics(context) { // context.jobName - the configured job name // context.scheduledAt - the occurrence this run is for (Date) // context.catchUp - true when this run is making up a missed occurrence - const cutoff = Date.now() - 30 * 24 * 60 * 60 * 1000; - for await (const record of tables.Events.search([{ attribute: 'timestamp', comparator: 'less', value: cutoff }])) { - await tables.Events.delete(record.id); + const day = context.scheduledAt.toISOString().slice(0, 10); + let total = 0; + let active = 0; + for await (const device of tables.Devices.search([])) { + total++; + if (device.status === 'active') active++; } + // Keyed by day, so a duplicate delivery of the same occurrence just rewrites + // the same record - this is what makes the handler idempotent + await tables.DailyDeviceSnapshot.put({ id: day, total, active }); +} + +// Scheduled pull from an external API into a Harper table +export async function syncExchangeRates(context) { + const response = await fetch('https://api.example.com/rates'); + if (!response.ok) throw new Error(`rates API responded ${response.status}`); + const rates = await response.json(); + await tables.ExchangeRates.put({ id: 'latest', fetchedAt: context.scheduledAt, ...rates }); } ``` From 0279a40c8bd6afe4e299efc792d0e814876bed9a Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Fri, 17 Jul 2026 09:29:52 -0600 Subject: [PATCH 3/9] Add YAML quoting guidance for cron expressions and handler references Unquoted leading-asterisk crons are YAML aliases and @macros are reserved characters; a space before # in a handler reference silently drops the export name. Surfaced by audit of the feature PR. Also fixes a leftover cleanupOldRecords reference from the example rework. Co-Authored-By: Claude Fable 5 --- reference/components/scheduler.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md index 6592df20..e10b8d1f 100644 --- a/reference/components/scheduler.md +++ b/reference/components/scheduler.md @@ -44,6 +44,8 @@ A five-field cron expression: minute, hour, day-of-month, month, day-of-week. Su There is no seconds field. Expressions that can never fire (such as `0 0 30 2 *` - February 30th) are rejected when the component loads. +Always quote the expression in YAML: a leading `*` (as in `*/5 * * * *`) is YAML alias syntax and `@daily` starts with a reserved character - unquoted, both fail with confusing YAML parse errors rather than anything about cron. + Exactly one of `cron` or `interval` is required. ### `interval` @@ -64,7 +66,7 @@ During daylight-saving transitions: a job scheduled inside the spring-forward ga Type: `string` (required) -The module and export to invoke, as `#`, relative to the component directory - for example `./jobs.js#cleanupOldRecords`. Omit the `#...` suffix to use the module's default export. The module is loaded in your component's environment, so it has access to the same globals (`tables`, `databases`, `server`, and so on) as the rest of your component code. +The module and export to invoke, as `#`, relative to the component directory - for example `./jobs.js#snapshotMetrics`. Omit the `#...` suffix to use the module's default export. Do not put a space before the `#` (YAML would treat the rest of the line as a comment and silently drop the export name). The module is loaded in your component's environment, so it has access to the same globals (`tables`, `databases`, `server`, and so on) as the rest of your component code. A bad handler reference (missing module, missing export, non-function export) fails the component load at deploy time rather than failing silently at the first scheduled fire. From af1a83782ce8d78b5e5612540a00c073c66c25a9 Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Fri, 17 Jul 2026 11:11:38 -0600 Subject: [PATCH 4/9] Align scheduler docs with the implementation contract (review findings) - 'exactly once per occurrence' -> leader-coordinated single execution under normal operation, with possible duplicate delivery during failover/DST fall-back/split-brain recovery (consistent with the idempotency requirement); release-notes entry aligned too - overlap semantics corrected: runs never overlap, but an overdue interval job runs back-to-back after a slow handler, and cron makes up at most its most recent missed occurrence - interval documents both accepted types (duration string or number of seconds) and both bounds (1s to 365 days) Co-Authored-By: Claude Fable 5 --- reference/components/scheduler.md | 8 ++++---- release-notes/v5-lincoln/5.2.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md index e10b8d1f..2b6c8905 100644 --- a/reference/components/scheduler.md +++ b/reference/components/scheduler.md @@ -6,7 +6,7 @@ title: Scheduler -The scheduler is a built-in plugin that runs recurring jobs declared in a component's configuration. Harper invokes a designated export from your component on a cron or interval schedule, and in a cluster each job fires exactly once per occurrence - on a single, automatically elected node - rather than once per node or worker. +The scheduler is a built-in plugin that runs recurring jobs declared in a component's configuration. Harper invokes a designated export from your component on a cron or interval schedule. In a cluster, execution is leader-coordinated: under normal operation each occurrence runs once, on a single automatically elected node, rather than once per node or worker. During leadership failover, daylight-saving fall-back, or split-brain recovery an occurrence can occasionally be delivered more than once - which is why handlers must be idempotent (see below). Use it for the recurring work applications otherwise hand-roll with `setInterval`: taking daily snapshots of a dataset, pulling from an external API on a schedule, re-aggregating summary tables, generating digests or reports. @@ -50,9 +50,9 @@ Exactly one of `cron` or `interval` is required. ### `interval` -Type: `string` (duration) +Type: `string` (duration) or `number` (seconds) -A simple cadence instead of a cron expression: a number of seconds, or a value like `90s`, `5m`, `1h`, `1d`. Minimum one second. Use this for "every N" maintenance passes that do not align to wall-clock times. +A simple cadence instead of a cron expression: a number of seconds, or a duration string like `90s`, `5m`, `1h`, `1d`. Must be between one second and 365 days. Use this for "every N" maintenance passes that do not align to wall-clock times. ### `timezone` @@ -103,7 +103,7 @@ export async function syncExchangeRates(context) { **Handlers should be idempotent.** Harper's clustering has no distributed lock, so leadership failover and daylight-saving fall-back can occasionally deliver the same logical occurrence twice. Design handlers so that running twice for one occurrence is harmless. -Runs of the same job never overlap: if a run is still going when its next occurrence arrives, that occurrence is skipped (with a log entry) rather than stacked. +Runs of the same job never overlap. Missed time is not queued up, but it is also not always skipped outright: when a run outlasts its cadence, an interval job's next run starts promptly after the previous one finishes (a slow handler can therefore run back-to-back), and a cron job makes up at most its single most recent missed occurrence (delivered with `context.catchUp` set to `true`). ## Cluster Behavior diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 6405d6f9..4889005b 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -18,7 +18,7 @@ Vector searches combined with filters now evaluate the filter during HNSW graph ### Scheduler: Recurring Jobs from Component Config -Components can now declare recurring jobs in their configuration with a new built-in `scheduler` plugin. Jobs run on a five-field cron expression or a simple interval (`90s`, `5m`, `1h`), invoking a designated export from the component. In a cluster, each job fires exactly once per occurrence on an automatically elected leader node, with heartbeat-based failover, catch-up for missed occurrences, and per-job run state recorded in a replicated system table. See [Scheduler](/reference/v5/components/scheduler). +Components can now declare recurring jobs in their configuration with a new built-in `scheduler` plugin. Jobs run on a five-field cron expression or a simple interval (`90s`, `5m`, `1h`), invoking a designated export from the component. In a cluster, execution is leader-coordinated - under normal operation each occurrence runs once, on an automatically elected leader node - with heartbeat-based failover, catch-up for missed occurrences, and per-job run state recorded in a replicated system table (handlers should be idempotent, as failover can occasionally deliver an occurrence twice). See [Scheduler](/reference/v5/components/scheduler). ## Configuration From caf03967bb025cf6685672ce8f2b385fb059cbb7 Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Fri, 17 Jul 2026 14:41:09 -0600 Subject: [PATCH 5/9] Spread untrusted API payload before explicit keys in scheduler example Review nit (kriszyp): as written, a rogue `id` in the external API response would override the explicit 'latest' id and redirect the write. Co-Authored-By: Claude Fable 5 --- reference/components/scheduler.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md index 2b6c8905..6a7de3f7 100644 --- a/reference/components/scheduler.md +++ b/reference/components/scheduler.md @@ -97,7 +97,9 @@ export async function syncExchangeRates(context) { const response = await fetch('https://api.example.com/rates'); if (!response.ok) throw new Error(`rates API responded ${response.status}`); const rates = await response.json(); - await tables.ExchangeRates.put({ id: 'latest', fetchedAt: context.scheduledAt, ...rates }); + // Spread the untrusted API payload first so our explicit keys win - otherwise + // a rogue `id` in the response could redirect the write to another record + await tables.ExchangeRates.put({ ...rates, id: 'latest', fetchedAt: context.scheduledAt }); } ``` From 20b6a4b576f162e3fc0ac303bd508a0d12291d1c Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Mon, 20 Jul 2026 09:09:22 -0600 Subject: [PATCH 6/9] docs(scheduler): use documented search({}) form in handler example Matches the unconditioned-scan form documented in resource-api.md (review nit from kriszyp). Co-Authored-By: Claude Opus 4.8 --- reference/components/scheduler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md index 6a7de3f7..1002dba8 100644 --- a/reference/components/scheduler.md +++ b/reference/components/scheduler.md @@ -83,7 +83,7 @@ export async function snapshotMetrics(context) { const day = context.scheduledAt.toISOString().slice(0, 10); let total = 0; let active = 0; - for await (const device of tables.Devices.search([])) { + for await (const device of tables.Devices.search({})) { total++; if (device.status === 'active') active++; } From 004cec8427a6b750dbfe4e33d372c2303a459a86 Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Mon, 20 Jul 2026 12:55:01 -0600 Subject: [PATCH 7/9] =?UTF-8?q?docs(scheduler):=20address=20round-2=20revi?= =?UTF-8?q?ew=20=E2=80=94=20topology=20qualification,=20bounded=20external?= =?UTF-8?q?=20calls,=20config-replacement=20warning,=20leader-local=20time?= =?UTF-8?q?zone=20caveat,=20payload=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Qualify the cluster-once guarantee: it requires hdb_scheduler_state to reach every participating node; new Replication Topology section covers the constrained/directional-topology failure mode - Bound the example's external fetch with AbortSignal.timeout and document failure semantics (failed runs are recorded, not retried) - Warn that creating config.yaml replaces the default component config - Recommend explicit timezone for clusters (default resolves on the current leader's host) - Validate/destructure the API payload in the example instead of spreading an untrusted shape - Release notes: note catch-up backfills only the most recent miss Co-Authored-By: Claude Fable 5 --- reference/components/scheduler.md | 30 ++++++++++++++++++++++-------- release-notes/v5-lincoln/5.2.md | 2 +- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md index 1002dba8..4787de65 100644 --- a/reference/components/scheduler.md +++ b/reference/components/scheduler.md @@ -6,7 +6,7 @@ title: Scheduler -The scheduler is a built-in plugin that runs recurring jobs declared in a component's configuration. Harper invokes a designated export from your component on a cron or interval schedule. In a cluster, execution is leader-coordinated: under normal operation each occurrence runs once, on a single automatically elected node, rather than once per node or worker. During leadership failover, daylight-saving fall-back, or split-brain recovery an occurrence can occasionally be delivered more than once - which is why handlers must be idempotent (see below). +The scheduler is a built-in plugin that runs recurring jobs declared in a component's configuration. Harper invokes a designated export from your component on a cron or interval schedule. In a cluster, execution is leader-coordinated: under normal operation each occurrence runs once, on a single automatically elected node, rather than once per node or worker. This coordination relies on the `hdb_scheduler_state` system table replicating to every participating node (see [Cluster Behavior](#cluster-behavior)). During leadership failover, daylight-saving fall-back, or split-brain recovery an occurrence can occasionally be delivered more than once - which is why handlers must be idempotent (see below). Use it for the recurring work applications otherwise hand-roll with `setInterval`: taking daily snapshots of a dataset, pulling from an external API on a schedule, re-aggregating summary tables, generating digests or reports. @@ -14,7 +14,7 @@ For expiring old records, prefer the table-level [`expiration` and `eviction` op ## Configuration -In your component's `config.yaml`, use the `scheduler` key to declare jobs: +In your component's `config.yaml`, use the `scheduler` key to declare jobs. If your application does not yet have a `config.yaml`, note that creating one **replaces** Harper's [default component configuration](./overview.md#default-configuration) rather than merging with it - keep the plugins your app relies on (`rest`, `graphqlSchema`, `jsResource`, and so on) declared alongside the `scheduler` block, or your endpoints and resources will stop loading: ```yaml scheduler: @@ -58,7 +58,7 @@ A simple cadence instead of a cron expression: a number of seconds, or a duratio Type: `string` -An IANA timezone (for example `America/Chicago`) the cron expression is evaluated in. Defaults to the server's timezone. Only valid with `cron`. +An IANA timezone (for example `America/Chicago`) the cron expression is evaluated in. Defaults to the server's timezone - specifically, the local timezone of whichever node is currently the scheduler leader. In a cluster whose nodes may not share a host timezone, set an explicit `timezone`: otherwise a failover between differently-configured nodes can shift a job like `0 2 * * *` by hours, producing an extra or missing wall-clock run. Only valid with `cron`. During daylight-saving transitions: a job scheduled inside the spring-forward gap (a wall-clock time that does not exist that day) runs at the shifted instant rather than being skipped, and a job scheduled inside the fall-back overlap (a wall-clock time that occurs twice) runs once, at the first occurrence. @@ -94,17 +94,27 @@ export async function snapshotMetrics(context) { // Scheduled pull from an external API into a Harper table export async function syncExchangeRates(context) { - const response = await fetch('https://api.example.com/rates'); + // Always bound external calls: a request that never settles would leave this + // single-flight job stuck (runs never overlap, so nothing reschedules while + // a run is in flight) + const response = await fetch('https://api.example.com/rates', { signal: AbortSignal.timeout(10_000) }); if (!response.ok) throw new Error(`rates API responded ${response.status}`); - const rates = await response.json(); - // Spread the untrusted API payload first so our explicit keys win - otherwise - // a rogue `id` in the response could redirect the write to another record - await tables.ExchangeRates.put({ ...rates, id: 'latest', fetchedAt: context.scheduledAt }); + const payload = await response.json(); + // Persist only the fields you expect - Harper schemas are flexible by + // default, so an unvalidated spread would store whatever shape the + // upstream happened to send + const { usd, eur, gbp } = payload; + if (typeof usd !== 'number' || typeof eur !== 'number' || typeof gbp !== 'number') { + throw new Error('rates API returned an unexpected shape'); + } + await tables.ExchangeRates.put({ usd, eur, gbp, id: 'latest', fetchedAt: context.scheduledAt }); } ``` **Handlers should be idempotent.** Harper's clustering has no distributed lock, so leadership failover and daylight-saving fall-back can occasionally deliver the same logical occurrence twice. Design handlers so that running twice for one occurrence is harmless. +**A failed run is not retried.** If a handler throws (or its outbound request times out), the run is recorded with `lastStatus: error` and the occurrence is not made up - the job simply fires again at its next scheduled time. Handlers that need delivery guarantees should implement their own bounded retry/backoff inside the handler. + Runs of the same job never overlap. Missed time is not queued up, but it is also not always skipped outright: when a run outlasts its cadence, an interval job's next run starts promptly after the previous one finishes (a slow handler can therefore run back-to-back), and a cron job makes up at most its single most recent missed occurrence (delivered with `context.catchUp` set to `true`). ## Cluster Behavior @@ -121,6 +131,10 @@ If the most recent occurrence of a cron job was missed - the leader was down, a A newly deployed job waits for its first scheduled time; it does not fire immediately on deploy. +### Replication Topology + +The cluster-once guarantee holds only where the `hdb_scheduler_state` system table actually replicates: leader election and run-state coordination happen through its rows. On constrained or directional replication topologies where the `system` database does not reach every node, nodes that cannot see the leader's heartbeat will each elect themselves and run every job continuously - not just the occasional failover duplicate that idempotency covers. If you scope replication (for example with `replication.databases`), ensure the `system` database replicates among all scheduler-participating nodes. + ### Run State Each job's last run time, status, duration, and last error (if any) are recorded in the `hdb_scheduler_state` system table, alongside the leader lease. This state replicates across the cluster so a newly promoted leader knows what has already run. diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 4889005b..8e230419 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -18,7 +18,7 @@ Vector searches combined with filters now evaluate the filter during HNSW graph ### Scheduler: Recurring Jobs from Component Config -Components can now declare recurring jobs in their configuration with a new built-in `scheduler` plugin. Jobs run on a five-field cron expression or a simple interval (`90s`, `5m`, `1h`), invoking a designated export from the component. In a cluster, execution is leader-coordinated - under normal operation each occurrence runs once, on an automatically elected leader node - with heartbeat-based failover, catch-up for missed occurrences, and per-job run state recorded in a replicated system table (handlers should be idempotent, as failover can occasionally deliver an occurrence twice). See [Scheduler](/reference/v5/components/scheduler). +Components can now declare recurring jobs in their configuration with a new built-in `scheduler` plugin. Jobs run on a five-field cron expression or a simple interval (`90s`, `5m`, `1h`), invoking a designated export from the component. In a cluster, execution is leader-coordinated - under normal operation each occurrence runs once, on an automatically elected leader node - with heartbeat-based failover, catch-up for missed occurrences, and per-job run state recorded in a replicated system table (handlers should be idempotent, as failover can occasionally deliver an occurrence twice; conversely, catch-up only backfills the single most recent missed occurrence, not a full backlog). See [Scheduler](/reference/v5/components/scheduler). ## Configuration From c01c5d65aedd7418f9932465f564eb3db55b376b Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Mon, 20 Jul 2026 13:13:00 -0600 Subject: [PATCH 8/9] docs(scheduler): overlap guard is per-leader; scope failover estimate to first-successor-alive Co-Authored-By: Claude Fable 5 --- reference/components/scheduler.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md index 4787de65..e48169c6 100644 --- a/reference/components/scheduler.md +++ b/reference/components/scheduler.md @@ -115,13 +115,13 @@ export async function syncExchangeRates(context) { **A failed run is not retried.** If a handler throws (or its outbound request times out), the run is recorded with `lastStatus: error` and the occurrence is not made up - the job simply fires again at its next scheduled time. Handlers that need delivery guarantees should implement their own bounded retry/backoff inside the handler. -Runs of the same job never overlap. Missed time is not queued up, but it is also not always skipped outright: when a run outlasts its cadence, an interval job's next run starts promptly after the previous one finishes (a slow handler can therefore run back-to-back), and a cron job makes up at most its single most recent missed occurrence (delivered with `context.catchUp` set to `true`). +On a single scheduler leader, runs of the same job never overlap. Note this guard is per-leader: in the rare duplicate-delivery cases above (a partitioned leader with a slow handler still in flight while its successor promotes), the two deliveries of one occurrence run on different nodes and CAN overlap in time - so idempotency must cover concurrent execution, not just repeated execution. Missed time is not queued up, but it is also not always skipped outright: when a run outlasts its cadence, an interval job's next run starts promptly after the previous one finishes (a slow handler can therefore run back-to-back), and a cron job makes up at most its single most recent missed occurrence (delivered with `context.catchUp` set to `true`). ## Cluster Behavior One node in the cluster - the scheduler leader - runs all scheduled jobs; the others watch. Leader election is automatic and requires no configuration: -- The leader maintains a heartbeat in the replicated `system` database. If it stops heartbeating (crash, shutdown, partition) for more than five minutes, the next node in line promotes itself; with default timings, failover completes within about six and a half minutes. +- The leader maintains a heartbeat in the replicated `system` database. If it stops heartbeating (crash, shutdown, partition) for more than five minutes, the next node in line promotes itself; with default timings, failover completes within about six and a half minutes when the first eligible successor is up. Each earlier successor that is also down adds a further escalation delay (two and a half minutes per node) before the next one in line claims leadership. - A restarting node defers to an actively heartbeating leader, so leadership is stable across deploys and restarts. - On a single-node instance, that node simply runs the jobs. From 7031a07fbcc2bbb8659eea975c7ac9b361edb33e Mon Sep 17 00:00:00 2001 From: jcohen-hdb Date: Mon, 20 Jul 2026 14:01:00 -0600 Subject: [PATCH 9/9] docs(scheduler): non-overlap guard also has a reload/redeploy boundary Co-Authored-By: Claude Fable 5 --- reference/components/scheduler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/components/scheduler.md b/reference/components/scheduler.md index e48169c6..1b3832b1 100644 --- a/reference/components/scheduler.md +++ b/reference/components/scheduler.md @@ -115,7 +115,7 @@ export async function syncExchangeRates(context) { **A failed run is not retried.** If a handler throws (or its outbound request times out), the run is recorded with `lastStatus: error` and the occurrence is not made up - the job simply fires again at its next scheduled time. Handlers that need delivery guarantees should implement their own bounded retry/backoff inside the handler. -On a single scheduler leader, runs of the same job never overlap. Note this guard is per-leader: in the rare duplicate-delivery cases above (a partitioned leader with a slow handler still in flight while its successor promotes), the two deliveries of one occurrence run on different nodes and CAN overlap in time - so idempotency must cover concurrent execution, not just repeated execution. Missed time is not queued up, but it is also not always skipped outright: when a run outlasts its cadence, an interval job's next run starts promptly after the previous one finishes (a slow handler can therefore run back-to-back), and a cron job makes up at most its single most recent missed occurrence (delivered with `context.catchUp` set to `true`). +For a single, unchanged job registration on one scheduler leader, runs never overlap. Two boundaries pierce that guard, and both mean idempotency must cover concurrent execution, not just repeated execution: across nodes, the rare duplicate-delivery cases above (a partitioned leader with a slow handler still in flight while its successor promotes) run one occurrence on two nodes at once; and across reloads, a redeploy cannot cancel a handler that is already executing, so the redeployed job can fire while the previous registration's run is still in flight on the same node. Missed time is not queued up, but it is also not always skipped outright: when a run outlasts its cadence, an interval job's next run starts promptly after the previous one finishes (a slow handler can therefore run back-to-back), and a cron job makes up at most its single most recent missed occurrence (delivered with `context.catchUp` set to `true`). ## Cluster Behavior