Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,46 @@ All notable changes to taskito are documented here. The format is based on
[Semantic Versioning](https://semver.org/spec/v2.0.0.html). All SDKs (Python, Node, Java) and the
underlying Rust crates are released together, in lock-step.

## 0.20.0

Scaling release: the queue survives higher task ingest/dispatch and pub/sub fan-out rates, and
schema management moves to code-first migrations. Worker dispatch now respects each pool's
execution capacity, so a worker no longer claims more than it can run and starve its peers.

### Added

- **Code-first migrations.** All schema DDL is built programmatically with `sea-query` (no raw
SQL) and applied by a versioned migrator with path-based auto-discovery — dropping a
`migrations/mXXXX_*.rs` file registers it automatically. Migration recording is atomic and
idempotent, so concurrent startups and interruptions are safe. New scaling indexes ship as
part of the baseline.
- **Batching.** Keyed pub/sub fan-out is committed in a single transaction, Redis subscription
reads are pipelined to constant round trips, execution claims are taken per batch, and
successful results are finalized together — cutting round trips under load across all SDKs.
- **Bounded in-flight dispatch.** A scheduler now caps how many jobs it keeps in flight at the
worker pool's execution parallelism, freeing a slot (and refilling immediately) as each job
finishes.

### Changed

- **Drain-until-empty polling.** Each poll wake dispatches all ready work instead of one job per
tick, removing the throughput ceiling imposed by the poll interval.
- **Jittered backoff.** Retry backoff uses full jitter, and the poller's gate-reschedule delays
are jittered, to avoid retry stampedes.
- Topic fan-out releases the GIL during publish/reap/backlog work (Python).

### Fixed

- **Worker fairness.** Under drain-until-empty, one worker could claim a whole backlog and
strand it `Running` on itself while peers sharing the database sat idle; in-flight dispatch is
now bounded to the pool size.
- **Workflow step double-execution.** A step now honors the task's `max_retries`, and the
tracker reacts only to a node's terminal job failure — closing an exactly-once violation under
load.
- Jobs stranded by a transient claim error return to `Pending` immediately rather than waiting
out the stale-job reaper.
- Postgres batch claims run in one transaction (no partially-committed claims on error).

## 0.19.0

Parity release: the Java SDK reaches feature parity with the other SDKs (and its baseline moves
Expand Down
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion crates/taskito-core/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "taskito-core"
version = "0.19.0"
version = "0.20.0"
edition = "2021"

[features]
Expand Down
19 changes: 11 additions & 8 deletions crates/taskito-core/src/scheduler/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -274,15 +274,18 @@ impl Scheduler {
/// refill. Only ids this scheduler dispatched are tracked, so results for
/// recovered or foreign jobs (e.g. from maintenance) are a harmless no-op.
fn release_in_flight(&self, job_id: &str) {
if self.config.max_in_flight.is_none() {
let Some(max) = self.config.max_in_flight else {
return;
}
let freed = self
.in_flight
.lock()
.unwrap_or_else(|p| p.into_inner())
.remove(job_id);
if freed {
};
let mut in_flight = self.in_flight.lock().unwrap_or_else(|p| p.into_inner());
// Only wake the poller when the pool was saturated — that's the only time
// it's parked on the cap. Below the cap it's already dispatching on its
// normal cadence, so an extra wake per completion just adds needless
// dequeue contention with everything else touching the database.
let was_full = in_flight.len() >= max;
let freed = in_flight.remove(job_id);
drop(in_flight);
if freed && was_full {
self.dispatch_wake.notify_one();
}
}
Expand Down
2 changes: 1 addition & 1 deletion crates/taskito-java/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "taskito-java"
version = "0.19.0"
version = "0.20.0"
edition = "2021"
description = "Java (JNI) bindings for the Taskito task-queue core"
license = "MIT"
Expand Down
2 changes: 1 addition & 1 deletion crates/taskito-mesh/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "taskito-mesh"
version = "0.19.0"
version = "0.20.0"
edition = "2021"

[dependencies]
Expand Down
2 changes: 1 addition & 1 deletion crates/taskito-node/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "taskito-node"
version = "0.19.0"
version = "0.20.0"
edition = "2021"
description = "Node.js (napi-rs) bindings for the Taskito task-queue core"
license = "MIT"
Expand Down
2 changes: 1 addition & 1 deletion crates/taskito-python/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "taskito-python"
version = "0.19.0"
version = "0.20.0"
edition = "2021"

[features]
Expand Down
2 changes: 1 addition & 1 deletion crates/taskito-workflows/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "taskito-workflows"
version = "0.19.0"
version = "0.20.0"
edition = "2021"

[features]
Expand Down
2 changes: 1 addition & 1 deletion docs/app/lib/version.ts
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
// AUTO-GENERATED from /CHANGELOG.md by scripts/sync-changelog.mjs — do not edit directly.
export const VERSION = "0.19.0";
export const VERSION = "0.20.0";
40 changes: 40 additions & 0 deletions docs/content/docs/resources/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,46 @@ All notable changes to taskito are documented here. The format is based on
[Semantic Versioning](https://semver.org/spec/v2.0.0.html). All SDKs (Python, Node, Java) and the
underlying Rust crates are released together, in lock-step.

## 0.20.0

Scaling release: the queue survives higher task ingest/dispatch and pub/sub fan-out rates, and
schema management moves to code-first migrations. Worker dispatch now respects each pool's
execution capacity, so a worker no longer claims more than it can run and starve its peers.

### Added

- **Code-first migrations.** All schema DDL is built programmatically with `sea-query` (no raw
SQL) and applied by a versioned migrator with path-based auto-discovery — dropping a
`migrations/mXXXX_*.rs` file registers it automatically. Migration recording is atomic and
idempotent, so concurrent startups and interruptions are safe. New scaling indexes ship as
part of the baseline.
- **Batching.** Keyed pub/sub fan-out is committed in a single transaction, Redis subscription
reads are pipelined to constant round trips, execution claims are taken per batch, and
successful results are finalized together — cutting round trips under load across all SDKs.
- **Bounded in-flight dispatch.** A scheduler now caps how many jobs it keeps in flight at the
worker pool's execution parallelism, freeing a slot (and refilling immediately) as each job
finishes.

### Changed

- **Drain-until-empty polling.** Each poll wake dispatches all ready work instead of one job per
tick, removing the throughput ceiling imposed by the poll interval.
- **Jittered backoff.** Retry backoff uses full jitter, and the poller's gate-reschedule delays
are jittered, to avoid retry stampedes.
- Topic fan-out releases the GIL during publish/reap/backlog work (Python).

### Fixed

- **Worker fairness.** Under drain-until-empty, one worker could claim a whole backlog and
strand it `Running` on itself while peers sharing the database sat idle; in-flight dispatch is
now bounded to the pool size.
- **Workflow step double-execution.** A step now honors the task's `max_retries`, and the
tracker reacts only to a node's terminal job failure — closing an exactly-once violation under
load.
- Jobs stranded by a transient claim error return to `Pending` immediately rather than waiting
out the stale-job reaper.
- Postgres batch claims run in one transaction (no partially-committed claims on error).

## 0.19.0

Parity release: the Java SDK reaches feature parity with the other SDKs (and its baseline moves
Expand Down
2 changes: 1 addition & 1 deletion sdks/java/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ plugins {
}

group = "org.byteveda"
version = "0.19.0"
version = "0.20.0"

java {
// Sources + javadoc jars are added by the maven-publish plugin below.
Expand Down
2 changes: 1 addition & 1 deletion sdks/java/graalvm-smoke/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ plugins {
}

group = "org.byteveda"
version = "0.19.0"
version = "0.20.0"

repositories {
mavenCentral()
Expand Down
2 changes: 1 addition & 1 deletion sdks/java/processor/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ plugins {
}

group = "org.byteveda"
version = "0.19.0"
version = "0.20.0"

mavenPublishing {
publishToMavenCentral()
Expand Down
2 changes: 1 addition & 1 deletion sdks/java/spring/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ plugins {

group = "org.byteveda"

version = "0.18.0"
version = "0.20.0"

mavenPublishing {
publishToMavenCentral()
Expand Down
2 changes: 1 addition & 1 deletion sdks/java/test-support/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ plugins {
}

group = "org.byteveda"
version = "0.19.0"
version = "0.20.0"

mavenPublishing {
publishToMavenCentral()
Expand Down
2 changes: 1 addition & 1 deletion sdks/node/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@byteveda/taskito",
"version": "0.19.0",
"version": "0.20.0",
"description": "Rust-powered task queue for Node.js — no broker required.",
"license": "MIT",
"type": "module",
Expand Down
2 changes: 1 addition & 1 deletion sdks/python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "maturin"

[project]
name = "taskito"
version = "0.19.0"
version = "0.20.0"
description = "Rust-powered task queue for Python. No broker required."
requires-python = ">=3.10"
license = { file = "LICENSE" }
Expand Down
2 changes: 1 addition & 1 deletion sdks/python/taskito/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,4 +138,4 @@

__version__ = _get_version("taskito")
except PackageNotFoundError:
__version__ = "0.19.0"
__version__ = "0.20.0"