Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/content/docs/architecture/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ high-level model, then drill into the layer that interests you.
icon={<Boxes />}
title="Resources"
href="/architecture/resources"
description="The 3-layer DI pipeline — argument interception, worker injection, proxy reconstruction."
description="The dependency-injection pipeline — worker injection, scopes, and per-SDK argument handling."
/>
<Card
icon={<ShieldAlert />}
Expand All @@ -65,6 +65,6 @@ high-level model, then drill into the layer that interests you.
icon={<FileCode2 />}
title="Serialization"
href="/architecture/serialization"
description="Cloudpickle, JSON, MsgPack — how arguments and results cross the worker boundary."
description="JSON, MsgPack, CBOR, and language-native codecs — how arguments and results cross the worker boundary."
/>
</Cards>
4 changes: 2 additions & 2 deletions docs/content/docs/architecture/mesh.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -249,8 +249,8 @@ after a period of queue emptiness.

The mesh does **not** modify the [`Scheduler`](/architecture/scheduler)
struct or the [`WorkerDispatcher`](/architecture/worker-pool) trait.
Instead, `run_worker` (in `crates/taskito-python/src/py_queue/worker.rs`)
spawns a **mesh bridge** — an intermediate `tokio::sync::mpsc` channel
Instead, the per-SDK worker entrypoint spawns a **mesh bridge** — an
intermediate `tokio::sync::mpsc` channel
between the scheduler and dispatcher:

The `run_mesh_bridge()` function:
Expand Down
10 changes: 9 additions & 1 deletion docs/content/docs/java/api-reference/queue/events.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,17 @@ description: "Structured task logs, webhooks, and the autoscaler."

## Logs

Durable, per-job structured log rows in shared storage. Beyond diagnostics they
carry **streaming partial results**: a running task writes partials at level
`"result"` and a consumer polls them live (see the
[streaming guide](/java/guides/core/streaming)).

| Method | Description |
|---|---|
| `writeTaskLog(...)` / `getTaskLogs(id)` | Structured task-log lines for a job. |
| `writeTaskLog(jobId, taskName, level, message)` | Append a log row. An overload adds a String `extra` payload. |
| `getTaskLogs(jobId)` → `List<TaskLog>` | Every row for a job, oldest first. |
| `getTaskLogsAfter(jobId, afterId)` → `List<TaskLog>` | Rows after a cursor id — poll this to stream new partials without rescanning. |
| `queryTaskLogs(taskName, level, sinceMs, limit)` → `List<TaskLog>` | Cross-job query by task and level (e.g. every `"result"` line since a timestamp). |

## Webhooks & autoscaler

Expand Down
6 changes: 6 additions & 0 deletions docs/content/docs/java/api-reference/queue/workers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ description: "Worker builder, middleware, and workflow control."
|---|---|
| `use(Middleware)` | Cross-cutting hooks — `onEnqueue`, `before`/`after`/`onError`, and the outcome hooks. |

## Circuit breakers

| Method | Description |
|---|---|
| `listCircuitBreakers()` → `List<CircuitBreakerState>` | Live state of every registered [circuit breaker](/java/api-reference/task#circuitbreakerconfig) — open/closed/half-open, failure counts, next-probe time. |

Comment thread
pratyush618 marked this conversation as resolved.
## Workflows

| Method | Description |
Expand Down
32 changes: 31 additions & 1 deletion docs/content/docs/java/api-reference/task.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ enqueue options. The fluent option methods each return a new descriptor.
| `delayMs(long)` / `delay(Duration)` | Schedule after a delay. |
| `retryPolicy(RetryPolicy)` | Backoff curve — registered with the worker on `start()`. |
| `codecs(String...)` | Named [payload codecs](/java/api-reference/serializers) applied to this task. |
| `circuitBreaker(CircuitBreakerConfig)` | Trip the task after repeated failures; the worker registers the breaker on `start()`. |
| `withOptions(EnqueueOptions)` | Replace the default options wholesale. |

```java
Expand All @@ -45,7 +46,10 @@ Task<EmailPayload> sendEmail = Task.of("send_email", EmailPayload.class)
Per-enqueue overrides, built with `EnqueueOptions.builder()`; unset fields take
core defaults. Fields: `queue`, `priority`, `maxRetries`, `timeoutMs` /
`timeout(Duration)`, `delayMs` / `delay(Duration)`, `uniqueKey` (alias
`jobId`) for idempotent enqueues, `metadata`, `namespace`.
`jobId`) for idempotent enqueues, `metadata`, `namespace`, and
`dependsOn(String... jobIds)` (this job stays pending until the listed jobs
complete successfully; a failed or cancelled dependency cancels it — see
[enqueue options](/java/guides/core/enqueue-options#job-dependencies)).

```java
taskito.enqueue(sendEmail, payload, EnqueueOptions.builder()
Expand All @@ -64,6 +68,32 @@ taskito.enqueue(sendEmail, payload, EnqueueOptions.builder()
The retry *budget* comes from `maxRetries`; the policy only supplies the
timing, which the core scheduler applies durably.

## `CircuitBreakerConfig`

Guards a task: after `threshold` failures inside the rolling `window`, the
breaker opens and the task is skipped for `cooldown` before probing again.
Attach with `task.circuitBreaker(config)`; the worker registers it on `start()`.

| Factory / method | Description |
|---|---|
| `CircuitBreakerConfig.of(int threshold)` | Open after `threshold` failures; other fields default. |
| `CircuitBreakerConfig.builder(int threshold)` | Builder for the full config. |
| `window(Duration)` / `windowSeconds(long)` | Rolling failure window. |
| `cooldown(Duration)` / `cooldownSeconds(long)` | How long the breaker stays open. |
| `halfOpenProbes(int)` | Trial jobs admitted while half-open. |
| `halfOpenSuccessRate(double)` | Success fraction among probes needed to close. |

```java
Task<ChargePayload> charge = Task.of("charge", ChargePayload.class)
.circuitBreaker(CircuitBreakerConfig.builder(5)
.window(Duration.ofMinutes(1))
.cooldown(Duration.ofSeconds(30))
.build());
```

Inspect live breaker state with `taskito.listCircuitBreakers()`
([workers & hooks](/java/api-reference/queue/workers#circuit-breakers)).

## Annotations

`@TaskHandler` marks a method as a task handler. A compile-time processor
Expand Down
18 changes: 18 additions & 0 deletions docs/content/docs/java/guides/core/enqueue-options.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ taskito.enqueue(sendEmail, payload, EnqueueOptions.builder()
| `uniqueKey` | Idempotency key — a duplicate enqueue is a no-op while the first job is pending/running. |
| `metadata` | Free-form string stored with the job (surfaces on the dashboard / inspection). |
| `namespace` | Partition the store — jobs are only visible to clients/workers in the same namespace. |
| `dependsOn(String... jobIds)` | Gate this job on other jobs — it stays pending until they all finish ([below](#job-dependencies)). |

`toBuilder()` derives a modified copy from an existing instance, and the fluent
`Task` methods ([tasks](/java/guides/core/tasks#per-task-defaults)) cover the
Expand Down Expand Up @@ -57,3 +58,20 @@ List<String> ids = taskito.enqueueMany(resize, List.of(a, b, c),
A `uniqueKey` on the batch dedupes there too: a payload whose key already has
an active job resolves to the existing job's id instead of failing the batch.
`enqueueAll` is an alias of `enqueueMany`.

## Job dependencies

`dependsOn(jobId...)` makes a job wait for other jobs before it becomes eligible
to run — a lightweight per-job DAG, distinct from the
[workflow builder](/java/guides/workflows). The job stays pending until every
listed job completes successfully.

```java
String extract = taskito.enqueue(extractTask, source);
String transform = taskito.enqueue(transformTask, config,
EnqueueOptions.builder().dependsOn(extract).build());
```

Enqueue is rejected if a listed job id is missing or already dead, cancelled, or
failed. If a dependency later fails or is cancelled, the dependent is cancelled
too — the cancellation cascades down the chain.
2 changes: 1 addition & 1 deletion docs/content/docs/java/guides/integrations/meta.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"title": "Integrations",
"pages": ["index", "spring", "micrometer", "sentry"]
"pages": ["index", "spring", "micrometer", "prometheus", "sentry"]
}
2 changes: 1 addition & 1 deletion docs/content/docs/java/guides/operations/meta.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{ "title": "Operations", "pages": ["backends", "inspection", "dashboard", "sso", "mesh", "autoscaling", "cli", "testing", "security", "troubleshooting", "deployment", "graalvm"] }
{ "title": "Operations", "pages": ["backends", "inspection", "dashboard", "sso", "mesh", "autoscaling", "keda", "cli", "testing", "security", "troubleshooting", "deployment", "graalvm"] }
1 change: 1 addition & 0 deletions docs/content/docs/node/api-reference/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,6 @@ The public API exported from the `taskito` barrel. Contrib helpers live under
| [Context](/node/api-reference/context) | `currentJob()` — signal, progress, ids. |
| [Serializers](/node/api-reference/serializers) | `JsonSerializer`, `MsgpackSerializer`, `CborSerializer`, the `Serializer` interface. |
| [Workflows](/node/api-reference/workflows) | The workflow builder + run handle. |
| [Testing](/node/api-reference/testing) | `mockResource` — stub an injected resource in tests. |
| [Errors](/node/api-reference/errors) | The `TaskitoError` hierarchy. |
| [CLI](/node/api-reference/cli) | The `taskito` command. |
2 changes: 1 addition & 1 deletion docs/content/docs/node/api-reference/meta.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{ "title": "API Reference", "root": true, "pages": ["index", "queue", "task", "worker", "result", "context", "serializers", "workflows", "errors", "cli"] }
{ "title": "API Reference", "root": true, "pages": ["index", "queue", "task", "worker", "result", "context", "serializers", "workflows", "testing", "errors", "cli"] }
52 changes: 52 additions & 0 deletions docs/content/docs/node/api-reference/testing.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
---
title: Testing
description: "mockResource — swap a real dependency for a stub in tests."
---

```ts
import { mockResource } from "@byteveda/taskito";
import type { MockResource } from "@byteveda/taskito";
```

The Node SDK's testing model is: run a real in-process worker against a
throwaway queue (a temp SQLite file), enqueue, and await the result — the same
API you run in production. See the [testing guide](/node/guides/operations/testing)
for the full worker-in-a-test harness. This page covers the one dedicated test
utility: `mockResource`, for swapping an injected [resource](/node/api-reference/queue/resources)
for a stub.

## `mockResource`

```ts
function mockResource<T>(value: T): MockResource<T>;
```

Builds a [`MockResource`](#mockresource-1) wrapping `value`. Register its
`factory` with `queue.resource(name, mock.factory)` in place of the real
factory, then assert on `mock.resolutions` to confirm the resource was built.

```ts
import { mockResource } from "@byteveda/taskito";

const db = mockResource({ query: async () => [{ id: 1 }] });
queue.resource("db", db.factory);

// ...run the task under test...

expect(db.resolutions).toBe(1); // the worker built the resource exactly once
```

## `MockResource`

```ts
interface MockResource<T> {
value: T; // the value the factory returns
factory: () => T; // pass to queue.resource(name, mock.factory)
resolutions: number; // how many times the factory was invoked
}
```

`resolutions` increments each time the injected factory runs, so a test can
assert a resource was (or wasn't) constructed — useful for verifying scope
behavior, e.g. a `worker`-scoped resource builds once while a `task`-scoped one
builds per job.
63 changes: 63 additions & 0 deletions docs/content/docs/python/api-reference/errors.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
title: Errors
description: "The taskito exception hierarchy — catch by specific type or by the TaskitoError base."
---

Almost every exception taskito raises extends `TaskitoError`, so a single
`except` can scope them; the specific subclasses let you branch on what went
wrong. Two exceptions sit outside the tree (noted below) — a blanket
`except TaskitoError` will not catch them.

```python
from taskito import TaskitoError, ResourceNotFoundError

try:
result = job.result(timeout=5)
except ResourceNotFoundError as exc:
# the message names the missing resource, e.g. "Resource 'db' is not registered"
print(exc)
except TaskitoError as exc:
report(exc)
```
Comment thread
pratyush618 marked this conversation as resolved.

## Hierarchy

| Class | Extends | Raised when |
|---|---|---|
| `TaskitoError` | `Exception` | Base — never raised directly. |
| `TaskTimeoutError` | `TaskitoError` | A task exceeds its hard timeout. |
| `SoftTimeoutError` | `TaskitoError` | A task exceeds its soft timeout (checked cooperatively via the context). |
| `TaskCancelledError` | `TaskitoError` | A running task detects it has been cancelled. |
| `TaskFailedError` | `TaskitoError` | Awaiting a job that failed or dead-lettered. Carries `errtype`, `traceback`, `job_id`, `raw_error`. |
| `MaxRetriesExceededError` | `TaskitoError` | A task exhausts all retry attempts. Same failure details as `TaskFailedError`. |
| `SerializationError` | `TaskitoError` | Serialization / deserialization or payload-integrity failure (e.g. a bad `SignedSerializer` signature). |
| `CryptoError` | `SerializationError` | A [payload codec](/python/api-reference/serializers) (`HmacCodec`, `AesGcmCodec`) fails to decrypt or verify. |
| `InterceptionError` | `SerializationError` | An [argument interceptor](/python/guides/resources/proxies) rejects or misbehaves. |
| `CircuitBreakerOpenError` | `TaskitoError` | A task's circuit breaker is open. |
| `RateLimitExceededError` | `TaskitoError` | A task's rate limit is exceeded. |
| `JobNotFoundError` | `TaskitoError`, `KeyError` | A job ID isn't found in storage. |
| `QueueError` | `TaskitoError` | A queue-level operational error. |
| `NotesValidationError` | `TaskitoError`, `ValueError` | A `notes` dict breaks the contract (>15 fields, >4 KiB, …). |
| `PredicateRejectedError` | `TaskitoError` | An enqueue-time [predicate](/python/guides/core/predicates) cancelled the submission. Carries `task_name`, `reason`. |
| `BatchPartialFailureError` | `TaskFailedError` | A batch task where some items failed. |
| `ResourceError` | `TaskitoError` | Base for resource dependency-injection errors. |
| `ResourceInitError` | `ResourceError` | A resource factory fails during initialization. |
| `ResourceUnavailableError` | `ResourceError` | A resource is permanently unhealthy and can't be resolved. |
| `CircularDependencyError` | `ResourceError` | Resource dependencies form a cycle. |
| `ResourceNotFoundError` | `ResourceError`, `KeyError` | Resolving a resource name that was never registered. |
| `ProxyReconstructionError` | `ResourceError` | A proxy handler fails to reconstruct an object from its recipe. |
| `ProxyCleanupError` | `ResourceError` | A proxy handler fails during cleanup. |

## Outside the `TaskitoError` tree

| Class | Extends | Import from | Raised when |
|---|---|---|---|
| `LockNotAcquiredError` | `Exception` | `taskito.locks` | `queue.lock(...)` can't acquire a held lock. |
| `BatchResultTypeError` | `TypeError` | `taskito` | A batch result is read as the wrong type. |

The `KeyError` / `ValueError` / `TypeError` mix-ins keep existing
`except KeyError` / `except ValueError` clauses working when these are raised
inside enqueue or lookup paths.

See [error handling](/python/guides/reliability/error-handling) for retry /
timeout / dead-letter behavior around a failing task.
28 changes: 28 additions & 0 deletions docs/content/docs/python/api-reference/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,13 @@ import {
ListTodo,
CheckCircle2,
Variable,
Binary,
GitFork,
Workflow,
Undo2,
Boxes,
TestTube,
AlertTriangle,
Terminal,
} from "lucide-react";

Expand Down Expand Up @@ -50,6 +54,12 @@ signature, parameters, return values, and a short example.
href="/python/api-reference/context"
description="current_job — progress, logging, cancellation, timeouts inside a task."
/>
<Card
icon={<Binary />}
title="Serializers"
href="/python/api-reference/serializers"
description="SmartSerializer, JSON/CBOR, payload codecs, signing and encryption."
/>
<Card
icon={<GitFork />}
title="Canvas"
Expand All @@ -62,12 +72,30 @@ signature, parameters, return values, and a short example.
href="/python/api-reference/workflows"
description="The DAG builder — fan-out, fan-in, gates, conditions, sub-workflows."
/>
<Card
icon={<Undo2 />}
title="Saga"
href="/python/api-reference/saga"
description="Compensating workflows — rollback steps that undo completed work on failure."
/>
<Card
icon={<Boxes />}
title="Batching"
href="/python/api-reference/batching"
description="Coalesce many enqueues into a single batched job."
/>
<Card
icon={<TestTube />}
title="Testing"
href="/python/api-reference/testing"
description="test_mode, MockResource, fixtures for synchronous in-process verification."
/>
<Card
icon={<AlertTriangle />}
title="Errors"
href="/python/api-reference/errors"
description="The TaskitoError exception hierarchy — catch by type or by the base."
/>
<Card
icon={<Terminal />}
title="CLI"
Expand Down
2 changes: 2 additions & 0 deletions docs/content/docs/python/api-reference/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@
"task",
"result",
"context",
"serializers",
"canvas",
"workflows",
"saga",
"batching",
"testing",
"errors",
"cli"
]
}
Loading