diff --git a/docs/content/docs/architecture/index.mdx b/docs/content/docs/architecture/index.mdx index f0cd98be..8447cf63 100644 --- a/docs/content/docs/architecture/index.mdx +++ b/docs/content/docs/architecture/index.mdx @@ -53,7 +53,7 @@ high-level model, then drill into the layer that interests you. icon={} title="Resources" href="/architecture/resources" - description="The 3-layer DI pipeline — argument interception, worker injection, proxy reconstruction." + description="The dependency-injection pipeline — worker injection, scopes, and per-SDK argument handling." /> } @@ -65,6 +65,6 @@ high-level model, then drill into the layer that interests you. icon={} title="Serialization" href="/architecture/serialization" - description="Cloudpickle, JSON, MsgPack — how arguments and results cross the worker boundary." + description="JSON, MsgPack, CBOR, and language-native codecs — how arguments and results cross the worker boundary." /> diff --git a/docs/content/docs/architecture/mesh.mdx b/docs/content/docs/architecture/mesh.mdx index 16e594fb..11bd74e3 100644 --- a/docs/content/docs/architecture/mesh.mdx +++ b/docs/content/docs/architecture/mesh.mdx @@ -249,8 +249,8 @@ after a period of queue emptiness. The mesh does **not** modify the [`Scheduler`](/architecture/scheduler) struct or the [`WorkerDispatcher`](/architecture/worker-pool) trait. -Instead, `run_worker` (in `crates/taskito-python/src/py_queue/worker.rs`) -spawns a **mesh bridge** — an intermediate `tokio::sync::mpsc` channel +Instead, the per-SDK worker entrypoint spawns a **mesh bridge** — an +intermediate `tokio::sync::mpsc` channel between the scheduler and dispatcher: The `run_mesh_bridge()` function: diff --git a/docs/content/docs/java/api-reference/queue/events.mdx b/docs/content/docs/java/api-reference/queue/events.mdx index b81d61ae..f2f7aa48 100644 --- a/docs/content/docs/java/api-reference/queue/events.mdx +++ b/docs/content/docs/java/api-reference/queue/events.mdx @@ -5,9 +5,17 @@ description: "Structured task logs, webhooks, and the autoscaler." ## Logs +Durable, per-job structured log rows in shared storage. Beyond diagnostics they +carry **streaming partial results**: a running task writes partials at level +`"result"` and a consumer polls them live (see the +[streaming guide](/java/guides/core/streaming)). + | Method | Description | |---|---| -| `writeTaskLog(...)` / `getTaskLogs(id)` | Structured task-log lines for a job. | +| `writeTaskLog(jobId, taskName, level, message)` | Append a log row. An overload adds a String `extra` payload. | +| `getTaskLogs(jobId)` → `List` | Every row for a job, oldest first. | +| `getTaskLogsAfter(jobId, afterId)` → `List` | Rows after a cursor id — poll this to stream new partials without rescanning. | +| `queryTaskLogs(taskName, level, sinceMs, limit)` → `List` | Cross-job query by task and level (e.g. every `"result"` line since a timestamp). | ## Webhooks & autoscaler diff --git a/docs/content/docs/java/api-reference/queue/workers.mdx b/docs/content/docs/java/api-reference/queue/workers.mdx index e5828a75..3284f858 100644 --- a/docs/content/docs/java/api-reference/queue/workers.mdx +++ b/docs/content/docs/java/api-reference/queue/workers.mdx @@ -16,6 +16,12 @@ description: "Worker builder, middleware, and workflow control." |---|---| | `use(Middleware)` | Cross-cutting hooks — `onEnqueue`, `before`/`after`/`onError`, and the outcome hooks. | +## Circuit breakers + +| Method | Description | +|---|---| +| `listCircuitBreakers()` → `List` | Live state of every registered [circuit breaker](/java/api-reference/task#circuitbreakerconfig) — open/closed/half-open, failure counts, next-probe time. | + ## Workflows | Method | Description | diff --git a/docs/content/docs/java/api-reference/task.mdx b/docs/content/docs/java/api-reference/task.mdx index b4129b61..f81627d8 100644 --- a/docs/content/docs/java/api-reference/task.mdx +++ b/docs/content/docs/java/api-reference/task.mdx @@ -30,6 +30,7 @@ enqueue options. The fluent option methods each return a new descriptor. | `delayMs(long)` / `delay(Duration)` | Schedule after a delay. | | `retryPolicy(RetryPolicy)` | Backoff curve — registered with the worker on `start()`. | | `codecs(String...)` | Named [payload codecs](/java/api-reference/serializers) applied to this task. | +| `circuitBreaker(CircuitBreakerConfig)` | Trip the task after repeated failures; the worker registers the breaker on `start()`. | | `withOptions(EnqueueOptions)` | Replace the default options wholesale. | ```java @@ -45,7 +46,10 @@ Task sendEmail = Task.of("send_email", EmailPayload.class) Per-enqueue overrides, built with `EnqueueOptions.builder()`; unset fields take core defaults. Fields: `queue`, `priority`, `maxRetries`, `timeoutMs` / `timeout(Duration)`, `delayMs` / `delay(Duration)`, `uniqueKey` (alias -`jobId`) for idempotent enqueues, `metadata`, `namespace`. +`jobId`) for idempotent enqueues, `metadata`, `namespace`, and +`dependsOn(String... jobIds)` (this job stays pending until the listed jobs +complete successfully; a failed or cancelled dependency cancels it — see +[enqueue options](/java/guides/core/enqueue-options#job-dependencies)). ```java taskito.enqueue(sendEmail, payload, EnqueueOptions.builder() @@ -64,6 +68,32 @@ taskito.enqueue(sendEmail, payload, EnqueueOptions.builder() The retry *budget* comes from `maxRetries`; the policy only supplies the timing, which the core scheduler applies durably. +## `CircuitBreakerConfig` + +Guards a task: after `threshold` failures inside the rolling `window`, the +breaker opens and the task is skipped for `cooldown` before probing again. +Attach with `task.circuitBreaker(config)`; the worker registers it on `start()`. + +| Factory / method | Description | +|---|---| +| `CircuitBreakerConfig.of(int threshold)` | Open after `threshold` failures; other fields default. | +| `CircuitBreakerConfig.builder(int threshold)` | Builder for the full config. | +| `window(Duration)` / `windowSeconds(long)` | Rolling failure window. | +| `cooldown(Duration)` / `cooldownSeconds(long)` | How long the breaker stays open. | +| `halfOpenProbes(int)` | Trial jobs admitted while half-open. | +| `halfOpenSuccessRate(double)` | Success fraction among probes needed to close. | + +```java +Task charge = Task.of("charge", ChargePayload.class) + .circuitBreaker(CircuitBreakerConfig.builder(5) + .window(Duration.ofMinutes(1)) + .cooldown(Duration.ofSeconds(30)) + .build()); +``` + +Inspect live breaker state with `taskito.listCircuitBreakers()` +([workers & hooks](/java/api-reference/queue/workers#circuit-breakers)). + ## Annotations `@TaskHandler` marks a method as a task handler. A compile-time processor diff --git a/docs/content/docs/java/guides/core/enqueue-options.mdx b/docs/content/docs/java/guides/core/enqueue-options.mdx index 08229156..9f5b8f9c 100644 --- a/docs/content/docs/java/guides/core/enqueue-options.mdx +++ b/docs/content/docs/java/guides/core/enqueue-options.mdx @@ -26,6 +26,7 @@ taskito.enqueue(sendEmail, payload, EnqueueOptions.builder() | `uniqueKey` | Idempotency key — a duplicate enqueue is a no-op while the first job is pending/running. | | `metadata` | Free-form string stored with the job (surfaces on the dashboard / inspection). | | `namespace` | Partition the store — jobs are only visible to clients/workers in the same namespace. | +| `dependsOn(String... jobIds)` | Gate this job on other jobs — it stays pending until they all finish ([below](#job-dependencies)). | `toBuilder()` derives a modified copy from an existing instance, and the fluent `Task` methods ([tasks](/java/guides/core/tasks#per-task-defaults)) cover the @@ -57,3 +58,20 @@ List ids = taskito.enqueueMany(resize, List.of(a, b, c), A `uniqueKey` on the batch dedupes there too: a payload whose key already has an active job resolves to the existing job's id instead of failing the batch. `enqueueAll` is an alias of `enqueueMany`. + +## Job dependencies + +`dependsOn(jobId...)` makes a job wait for other jobs before it becomes eligible +to run — a lightweight per-job DAG, distinct from the +[workflow builder](/java/guides/workflows). The job stays pending until every +listed job completes successfully. + +```java +String extract = taskito.enqueue(extractTask, source); +String transform = taskito.enqueue(transformTask, config, + EnqueueOptions.builder().dependsOn(extract).build()); +``` + +Enqueue is rejected if a listed job id is missing or already dead, cancelled, or +failed. If a dependency later fails or is cancelled, the dependent is cancelled +too — the cancellation cascades down the chain. diff --git a/docs/content/docs/java/guides/integrations/meta.json b/docs/content/docs/java/guides/integrations/meta.json index 995a6027..883db2eb 100644 --- a/docs/content/docs/java/guides/integrations/meta.json +++ b/docs/content/docs/java/guides/integrations/meta.json @@ -1,4 +1,4 @@ { "title": "Integrations", - "pages": ["index", "spring", "micrometer", "sentry"] + "pages": ["index", "spring", "micrometer", "prometheus", "sentry"] } diff --git a/docs/content/docs/java/guides/operations/meta.json b/docs/content/docs/java/guides/operations/meta.json index 40bb2fc1..66d2416b 100644 --- a/docs/content/docs/java/guides/operations/meta.json +++ b/docs/content/docs/java/guides/operations/meta.json @@ -1 +1 @@ -{ "title": "Operations", "pages": ["backends", "inspection", "dashboard", "sso", "mesh", "autoscaling", "cli", "testing", "security", "troubleshooting", "deployment", "graalvm"] } +{ "title": "Operations", "pages": ["backends", "inspection", "dashboard", "sso", "mesh", "autoscaling", "keda", "cli", "testing", "security", "troubleshooting", "deployment", "graalvm"] } diff --git a/docs/content/docs/node/api-reference/index.mdx b/docs/content/docs/node/api-reference/index.mdx index 0258b130..3ac3ecba 100644 --- a/docs/content/docs/node/api-reference/index.mdx +++ b/docs/content/docs/node/api-reference/index.mdx @@ -15,5 +15,6 @@ The public API exported from the `taskito` barrel. Contrib helpers live under | [Context](/node/api-reference/context) | `currentJob()` — signal, progress, ids. | | [Serializers](/node/api-reference/serializers) | `JsonSerializer`, `MsgpackSerializer`, `CborSerializer`, the `Serializer` interface. | | [Workflows](/node/api-reference/workflows) | The workflow builder + run handle. | +| [Testing](/node/api-reference/testing) | `mockResource` — stub an injected resource in tests. | | [Errors](/node/api-reference/errors) | The `TaskitoError` hierarchy. | | [CLI](/node/api-reference/cli) | The `taskito` command. | diff --git a/docs/content/docs/node/api-reference/meta.json b/docs/content/docs/node/api-reference/meta.json index a8692ac6..9a42889d 100644 --- a/docs/content/docs/node/api-reference/meta.json +++ b/docs/content/docs/node/api-reference/meta.json @@ -1 +1 @@ -{ "title": "API Reference", "root": true, "pages": ["index", "queue", "task", "worker", "result", "context", "serializers", "workflows", "errors", "cli"] } +{ "title": "API Reference", "root": true, "pages": ["index", "queue", "task", "worker", "result", "context", "serializers", "workflows", "testing", "errors", "cli"] } diff --git a/docs/content/docs/node/api-reference/testing.mdx b/docs/content/docs/node/api-reference/testing.mdx new file mode 100644 index 00000000..ca6f40ef --- /dev/null +++ b/docs/content/docs/node/api-reference/testing.mdx @@ -0,0 +1,52 @@ +--- +title: Testing +description: "mockResource — swap a real dependency for a stub in tests." +--- + +```ts +import { mockResource } from "@byteveda/taskito"; +import type { MockResource } from "@byteveda/taskito"; +``` + +The Node SDK's testing model is: run a real in-process worker against a +throwaway queue (a temp SQLite file), enqueue, and await the result — the same +API you run in production. See the [testing guide](/node/guides/operations/testing) +for the full worker-in-a-test harness. This page covers the one dedicated test +utility: `mockResource`, for swapping an injected [resource](/node/api-reference/queue/resources) +for a stub. + +## `mockResource` + +```ts +function mockResource(value: T): MockResource; +``` + +Builds a [`MockResource`](#mockresource-1) wrapping `value`. Register its +`factory` with `queue.resource(name, mock.factory)` in place of the real +factory, then assert on `mock.resolutions` to confirm the resource was built. + +```ts +import { mockResource } from "@byteveda/taskito"; + +const db = mockResource({ query: async () => [{ id: 1 }] }); +queue.resource("db", db.factory); + +// ...run the task under test... + +expect(db.resolutions).toBe(1); // the worker built the resource exactly once +``` + +## `MockResource` + +```ts +interface MockResource { + value: T; // the value the factory returns + factory: () => T; // pass to queue.resource(name, mock.factory) + resolutions: number; // how many times the factory was invoked +} +``` + +`resolutions` increments each time the injected factory runs, so a test can +assert a resource was (or wasn't) constructed — useful for verifying scope +behavior, e.g. a `worker`-scoped resource builds once while a `task`-scoped one +builds per job. diff --git a/docs/content/docs/python/api-reference/errors.mdx b/docs/content/docs/python/api-reference/errors.mdx new file mode 100644 index 00000000..3bf7bde3 --- /dev/null +++ b/docs/content/docs/python/api-reference/errors.mdx @@ -0,0 +1,63 @@ +--- +title: Errors +description: "The taskito exception hierarchy — catch by specific type or by the TaskitoError base." +--- + +Almost every exception taskito raises extends `TaskitoError`, so a single +`except` can scope them; the specific subclasses let you branch on what went +wrong. Two exceptions sit outside the tree (noted below) — a blanket +`except TaskitoError` will not catch them. + +```python +from taskito import TaskitoError, ResourceNotFoundError + +try: + result = job.result(timeout=5) +except ResourceNotFoundError as exc: + # the message names the missing resource, e.g. "Resource 'db' is not registered" + print(exc) +except TaskitoError as exc: + report(exc) +``` + +## Hierarchy + +| Class | Extends | Raised when | +|---|---|---| +| `TaskitoError` | `Exception` | Base — never raised directly. | +| `TaskTimeoutError` | `TaskitoError` | A task exceeds its hard timeout. | +| `SoftTimeoutError` | `TaskitoError` | A task exceeds its soft timeout (checked cooperatively via the context). | +| `TaskCancelledError` | `TaskitoError` | A running task detects it has been cancelled. | +| `TaskFailedError` | `TaskitoError` | Awaiting a job that failed or dead-lettered. Carries `errtype`, `traceback`, `job_id`, `raw_error`. | +| `MaxRetriesExceededError` | `TaskitoError` | A task exhausts all retry attempts. Same failure details as `TaskFailedError`. | +| `SerializationError` | `TaskitoError` | Serialization / deserialization or payload-integrity failure (e.g. a bad `SignedSerializer` signature). | +| `CryptoError` | `SerializationError` | A [payload codec](/python/api-reference/serializers) (`HmacCodec`, `AesGcmCodec`) fails to decrypt or verify. | +| `InterceptionError` | `SerializationError` | An [argument interceptor](/python/guides/resources/proxies) rejects or misbehaves. | +| `CircuitBreakerOpenError` | `TaskitoError` | A task's circuit breaker is open. | +| `RateLimitExceededError` | `TaskitoError` | A task's rate limit is exceeded. | +| `JobNotFoundError` | `TaskitoError`, `KeyError` | A job ID isn't found in storage. | +| `QueueError` | `TaskitoError` | A queue-level operational error. | +| `NotesValidationError` | `TaskitoError`, `ValueError` | A `notes` dict breaks the contract (>15 fields, >4 KiB, …). | +| `PredicateRejectedError` | `TaskitoError` | An enqueue-time [predicate](/python/guides/core/predicates) cancelled the submission. Carries `task_name`, `reason`. | +| `BatchPartialFailureError` | `TaskFailedError` | A batch task where some items failed. | +| `ResourceError` | `TaskitoError` | Base for resource dependency-injection errors. | +| `ResourceInitError` | `ResourceError` | A resource factory fails during initialization. | +| `ResourceUnavailableError` | `ResourceError` | A resource is permanently unhealthy and can't be resolved. | +| `CircularDependencyError` | `ResourceError` | Resource dependencies form a cycle. | +| `ResourceNotFoundError` | `ResourceError`, `KeyError` | Resolving a resource name that was never registered. | +| `ProxyReconstructionError` | `ResourceError` | A proxy handler fails to reconstruct an object from its recipe. | +| `ProxyCleanupError` | `ResourceError` | A proxy handler fails during cleanup. | + +## Outside the `TaskitoError` tree + +| Class | Extends | Import from | Raised when | +|---|---|---|---| +| `LockNotAcquiredError` | `Exception` | `taskito.locks` | `queue.lock(...)` can't acquire a held lock. | +| `BatchResultTypeError` | `TypeError` | `taskito` | A batch result is read as the wrong type. | + +The `KeyError` / `ValueError` / `TypeError` mix-ins keep existing +`except KeyError` / `except ValueError` clauses working when these are raised +inside enqueue or lookup paths. + +See [error handling](/python/guides/reliability/error-handling) for retry / +timeout / dead-letter behavior around a failing task. diff --git a/docs/content/docs/python/api-reference/index.mdx b/docs/content/docs/python/api-reference/index.mdx index 5b006c2b..200cbd6d 100644 --- a/docs/content/docs/python/api-reference/index.mdx +++ b/docs/content/docs/python/api-reference/index.mdx @@ -10,9 +10,13 @@ import { ListTodo, CheckCircle2, Variable, + Binary, GitFork, Workflow, + Undo2, + Boxes, TestTube, + AlertTriangle, Terminal, } from "lucide-react"; @@ -50,6 +54,12 @@ signature, parameters, return values, and a short example. href="/python/api-reference/context" description="current_job — progress, logging, cancellation, timeouts inside a task." /> + } + title="Serializers" + href="/python/api-reference/serializers" + description="SmartSerializer, JSON/CBOR, payload codecs, signing and encryption." + /> } title="Canvas" @@ -62,12 +72,30 @@ signature, parameters, return values, and a short example. href="/python/api-reference/workflows" description="The DAG builder — fan-out, fan-in, gates, conditions, sub-workflows." /> + } + title="Saga" + href="/python/api-reference/saga" + description="Compensating workflows — rollback steps that undo completed work on failure." + /> + } + title="Batching" + href="/python/api-reference/batching" + description="Coalesce many enqueues into a single batched job." + /> } title="Testing" href="/python/api-reference/testing" description="test_mode, MockResource, fixtures for synchronous in-process verification." /> + } + title="Errors" + href="/python/api-reference/errors" + description="The TaskitoError exception hierarchy — catch by type or by the base." + /> } title="CLI" diff --git a/docs/content/docs/python/api-reference/meta.json b/docs/content/docs/python/api-reference/meta.json index a27967f8..a2b2f0b7 100644 --- a/docs/content/docs/python/api-reference/meta.json +++ b/docs/content/docs/python/api-reference/meta.json @@ -8,11 +8,13 @@ "task", "result", "context", + "serializers", "canvas", "workflows", "saga", "batching", "testing", + "errors", "cli" ] } diff --git a/docs/content/docs/python/api-reference/serializers.mdx b/docs/content/docs/python/api-reference/serializers.mdx new file mode 100644 index 00000000..8e88dd45 --- /dev/null +++ b/docs/content/docs/python/api-reference/serializers.mdx @@ -0,0 +1,99 @@ +--- +title: Serializers +description: "SmartSerializer, JsonSerializer, CborSerializer, payload codecs, and the Serializer protocol." +--- + +```python +from taskito import Queue, JsonSerializer, CborSerializer +``` + +The Rust core treats payloads as opaque bytes — serialization happens entirely +Python-side. Producers and workers must use a compatible serializer. + +## Built-ins + +| Class | Description | +|---|---| +| `SmartSerializer` | **Default.** MsgPack for plain data, transparent cloudpickle fallback for anything MsgPack can't encode (lambdas, closures, class instances). A one-byte tag records which codec produced each payload; also reads CBOR (`0x02`) wire payloads from another SDK. | +| `CloudpickleSerializer` | Pure cloudpickle — handles any picklable Python object. Same-language only (not cross-SDK). | +| `JsonSerializer` | Human-readable JSON bytes. Simple, cross-language, MsgPack-native types only. | +| `MsgPackSerializer` | Compact binary (MsgPack). MsgPack-native types only. | +| `CborSerializer` | Binary CBOR (RFC 8949), tagged with the `0x02` wire-envelope byte — the format for tasks produced or consumed by another Taskito SDK. Round-trips big integers, `datetime` (tz-aware), `bytes`, and `Decimal` losslessly. | +| `SignedSerializer` | Wraps another serializer with an HMAC-SHA256 tag; the worker refuses to deserialize bytes not produced with the shared key. | +| `EncryptedSerializer` | Wraps another serializer with AES-256-GCM; confidentiality **and** integrity. | + +Set one queue-wide, or override per task: + +```python +queue = Queue(serializer=JsonSerializer()) + +@queue.task(serializer=CborSerializer()) +def cross_sdk_job(payload: dict) -> None: ... +``` + + + `SmartSerializer` is the default because it is fast for the common case yet + never fails on an exotic Python object. Reach for `CborSerializer` only when a + task is produced or consumed by another SDK. + + +## Signing and encryption + +`SignedSerializer(inner, key)` and `EncryptedSerializer(inner, key)` both wrap a +delegate serializer. `key` must be at least 32 bytes of CSPRNG output, shared by +every producer and worker. Signing authenticates (prevents a storage writer from +smuggling code into a cloudpickle payload); encryption also gives +confidentiality. + +```python +import os +from taskito import Queue, SignedSerializer, SmartSerializer + +key = os.urandom(32) # share across producers and workers +queue = Queue(serializer=SignedSerializer(SmartSerializer(), key)) +``` + +## Payload codecs + +A `PayloadCodec` is a reversible byte transform layered *around* a serializer — +compression, encryption, signing — rather than a replacement for it. Codecs +compose: a chain encodes in list order on the producer and decodes in reverse on +the worker. Wire formats are part of the cross-SDK contract, so a codec-framed +payload decodes from any Taskito SDK. + +| Class | Wire format | Description | +|---|---|---| +| `GzipCodec(max_decompressed_bytes=...)` | standard gzip stream | Compresses; decompression capped at 64 MiB by default (zip-bomb guard). | +| `HmacCodec(key)` | `[32-byte mac][body]` | HMAC-SHA256 signs; rejects tampered or wrong-key payloads. | +| `AesGcmCodec(key)` | `[12-byte IV][ciphertext \|\| 16-byte tag]` | AES-128/192/256-GCM (by key length); confidentiality + integrity. | + +Apply a chain queue-wide with `Queue(codec=...)` (wraps the serializer, covers +every payload and result), or register named codecs via `Queue(codecs=...)` and +opt individual tasks in with `@queue.task(codecs=[...])` (payload only — results +still use the plain serializer): + +```python +from taskito import Queue, GzipCodec, HmacCodec + +queue = Queue(codec=[GzipCodec(), HmacCodec(hmac_key)]) # gzip, then hmac +``` + + + `CodecSerializer(delegate, codecs)` is the internal `Serializer` that layers a + codec chain around a delegate — `Queue(codec=...)` builds one for you, so you + rarely construct it directly. Nondeterministic codecs (e.g. `AesGcmCodec`) make + a payload's bytes non-reproducible, so avoid pairing them with content-hash + idempotency. + + +## `Serializer` protocol + +```python +class Serializer(Protocol): + def dumps(self, obj: Any) -> bytes: ... + def loads(self, data: bytes) -> Any: ... +``` + +Any object with `dumps`/`loads` is a valid serializer. See the +[Serializers guide](/python/guides/extensibility/serializers) for a custom +implementation and the format-selection tradeoffs. diff --git a/docs/content/docs/resources/changelog.mdx b/docs/content/docs/resources/changelog.mdx index de352637..54ebe274 100644 --- a/docs/content/docs/resources/changelog.mdx +++ b/docs/content/docs/resources/changelog.mdx @@ -5,6 +5,8 @@ description: "Release history for taskito — every notable change, fix, and fea {/* AUTO-GENERATED from /CHANGELOG.md by scripts/sync-changelog.mjs — do not edit directly. */} +Code snippets below use the Python API for illustration. Every release ships all three SDKs — Python, Node, and Java — in lock-step; see each SDK's API reference for its exact syntax. + All notable changes to taskito are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); the project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). All SDKs (Python, Node, Java) and the diff --git a/docs/content/docs/resources/comparison.mdx b/docs/content/docs/resources/comparison.mdx index d974de5d..870e68c7 100644 --- a/docs/content/docs/resources/comparison.mdx +++ b/docs/content/docs/resources/comparison.mdx @@ -1,18 +1,39 @@ --- title: Comparison -description: "Taskito vs Celery, RQ, Dramatiq, Huey, TaskIQ — feature matrix and decision guide." +description: "Taskito vs other task queues — feature matrix and decision guide." --- **TL;DR**: Taskito is without the -broker (a separate server such as Redis or RabbitMQ that holds pending jobs). Rust scheduler, no Redis/RabbitMQ, lower latency, better concurrency. -Start with SQLite, scale to Postgres when needed. +broker (a separate server such as Redis or RabbitMQ that holds pending jobs). Rust scheduler, no broker required, lower latency, better concurrency. +Start with SQLite; add Postgres or Redis as a backend when you scale out. + +## When to use taskito + +taskito is ideal when: + +- **Single-machine deployments** — no need for distributed workers across multiple servers +- **Zero infrastructure** — you don't want to install, configure, or manage Redis or RabbitMQ +- **Embedded applications** — CLI tools, desktop apps, or services where simplicity matters +- **Prototyping** — get a task queue running in 5 lines, iterate fast +- **Low-to-medium throughput** — hundreds to thousands of jobs per second is plenty + +## When NOT to use taskito + +Consider alternatives when: + +- **Multi-server workers** — you need workers on separate machines (taskito supports this with Postgres/Redis backends, but a broker-based queue has more mature distributed tooling) +- **Very high throughput** — millions of jobs/sec across a cluster (use a dedicated broker such as RabbitMQ or Kafka) +- **Existing broker infrastructure** — if Redis/RabbitMQ is already in your stack, a broker-native queue may be simpler +- **Complex routing** — you need topic exchanges, message filtering, or fan-out patterns beyond taskito's pub/sub + + ## Feature matrix | Feature | taskito | Celery | RQ | Dramatiq | Huey | TaskIQ | |---|---|---|---|---|---|---| | Broker required | **No** | Redis / RabbitMQ | Redis | Redis / RabbitMQ | Redis | Redis / RabbitMQ / Nats | -| Core language | **Rust + ** | Python | Python | Python | Python | Python | +| Core language | **Rust + Python** | Python | Python | Python | Python | Python | | Priority queues | **Yes** | Yes | No | No | Yes | Yes | | Rate limiting | **Yes** | Yes | No | Yes | No | No | | Dead letter queue | **Yes** | No | Yes | No | No | No | @@ -39,25 +60,6 @@ Start with SQLite, scale to Postgres when needed. | Result backend (where task return values are stored) | **Built-in** (SQLite) | Redis / DB / custom | Redis | Redis / custom | Redis / SQLite | Redis / custom | | Setup complexity | **`pip install`** | Broker + backend | Redis server | Broker | Redis server | Broker + backend | -## When to use taskito - -taskito is ideal when: - -- **Single-machine deployments** — no need for distributed workers across multiple servers -- **Zero infrastructure** — you don't want to install, configure, or manage Redis or RabbitMQ -- **Embedded applications** — CLI tools, desktop apps, or services where simplicity matters -- **Prototyping** — get a task queue running in 5 lines, iterate fast -- **Low-to-medium throughput** — hundreds to thousands of jobs per second is plenty - -## When NOT to use taskito - -Consider alternatives when: - -- **Multi-server workers** — you need workers on separate machines (taskito supports this with Postgres/Redis backends, but Celery has more mature distributed tooling) -- **Very high throughput** — millions of jobs/sec across a cluster (use Celery + RabbitMQ) -- **Existing Redis infrastructure** — if Redis is already in your stack, RQ or Huey are simple choices -- **Complex routing** — you need topic exchanges, message filtering, or pub/sub patterns (use Celery + RabbitMQ) - ## Detailed comparison ### vs Celery @@ -144,3 +146,54 @@ async and already have a broker. Choose taskito if you want zero infrastructure. Choose TaskIQ if you're fully async and already have Redis/Nats. + + + + + +## How taskito compares + +Most Node task queues — **BullMQ**, **Bee-Queue**, **Bull** — require a running +Redis instance as the broker, and **Agenda** requires MongoDB. taskito embeds +the queue in a Rust core with a pluggable backend (SQLite, Postgres, or Redis), +so you can start with **zero infrastructure** and add Redis only when you scale +out. + +| | taskito | BullMQ / Bee-Queue | Agenda | +|---|---|---|---| +| **Broker** | None (SQLite) — optional Postgres/Redis | Redis required | MongoDB required | +| **Core** | Rust scheduler | JS + Redis Lua | JS + Mongo | +| **Priorities / rate limits / DLQ** | Built-in | Priorities + rate limits (BullMQ); no built-in DLQ | Limited | +| **Workflows** | Builder DAG, fan-out, sub-workflows, sagas | Flows (BullMQ) | No | +| **Result streaming** | Yes (publish/stream) | No | No | + +**Choose taskito** for a broker-optional queue with a workflow engine built in. +**Choose BullMQ** if Redis is already central to your stack and you want its +mature ecosystem. + + + + + +## How taskito compares + +JVM background-job tools each specialize: **Quartz** for cron scheduling, +**JobRunr** for background jobs (RDBMS or Redis), **Spring Batch** for +batch/ETL pipelines. taskito is a general-purpose task queue with a Rust core: +priorities, rate limits, dead-letter queues, workflows, and result streaming, +backed by SQLite, Postgres, or Redis, usable standalone or via the Spring Boot +starter. + +| | taskito | Quartz | JobRunr | Spring Batch | +|---|---|---|---|---| +| **Focus** | Task queue + workflows | Cron scheduling | Background jobs | Batch/ETL pipelines | +| **Backend** | SQLite / Postgres / Redis | Your RDBMS | Your RDBMS / Redis | Your RDBMS | +| **Priorities / rate limits / DLQ** | Built-in | No | Limited | No | +| **Workflows** | Builder DAG, fan-out, sub-workflows, sagas | No | No | Step/chunk model | +| **Result streaming** | Yes (publish/stream) | No | No | No | + +**Choose taskito** for a broker-optional queue with priorities, DLQ, and a +workflow engine. **Choose Quartz/Spring Batch** for pure cron scheduling or +chunk-oriented batch processing tightly coupled to Spring. + + diff --git a/scripts/sync-changelog.mjs b/scripts/sync-changelog.mjs index fe7611a8..86a45420 100644 --- a/scripts/sync-changelog.mjs +++ b/scripts/sync-changelog.mjs @@ -43,6 +43,8 @@ const frontmatter = [ "", "{/* AUTO-GENERATED from /CHANGELOG.md by scripts/sync-changelog.mjs — do not edit directly. */}", "", + 'Code snippets below use the Python API for illustration. Every release ships all three SDKs — Python, Node, and Java — in lock-step; see each SDK\'s API reference for its exact syntax.', + "", ].join("\n"); writeFileSync(target, `${frontmatter}\n${body}\n`);