Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 47 additions & 2 deletions crates/taskito-core/BINDING_CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,53 @@ depending on it). Keep Python/Node/Java specifics in the shell.
them. Each shell serializes args/kwargs at enqueue and deserializes them in the
worker — using whatever serializer it wants. The Python shell defaults to cloudpickle
(Python-only). **Cross-language constraint:** a job enqueued by one language and run
by another requires both to agree on a neutral serializer (JSON or msgpack);
cloudpickle is Python-internal only.
by another requires both sides to use the wire envelope below.

## Wire envelope (cross-SDK payloads)
A wire payload is one tag byte followed by the codec body. The tag records which
codec produced the body, so any shell can dispatch a decoder (or reject clearly)
without out-of-band configuration:

| Tag | Body | Cross-SDK | Notes |
|--------|-------------|-----------|-------|
| `0x00` | native | **never** — reject with a clear error | Language-native codec (e.g. pickle). Same-language producer/consumer only. |
| `0x01` | msgpack | optional | Legacy tagged format; shells MAY read it, SHOULD NOT write it cross-SDK. |
| `0x02` | CBOR (RFC 8949) | **default** | The cross-SDK wire format. |
| `0x03` | reserved | — | Tagged JSON (not yet specified). |
| `0x04+`| reserved | — | Future (protobuf, …). |

Untagged payloads predate the envelope and are same-SDK legacy; a shell MUST NOT
assume any tag discipline unless the task is configured with a tagged serializer
on both sides. (Sniffing is unsafe: raw msgpack/CBOR bodies can begin with any
byte value.)

**Why CBOR over JSON**: integers survive — JS `Number` is exact only to 2^53−1
while other languages carry 64-bit/unbounded ints; CBOR bignums round-trip them
losslessly. IANA tags also round-trip datetimes (tag 0/1), decimals (tag 4), and
byte strings without a hand-rolled registry. Mature codecs exist everywhere
(`cbor2`, `cbor-x`, `jackson-dataformat-cbor`, `fxamacker/cbor`, …).

**Call body** (`Job.payload`): a 2-element CBOR array `[args, kwargs]` — `args`
an array, `kwargs` a map (empty map when the language has no keyword arguments).
Job-scoped extras belong in the `metadata`/`notes` columns, not the payload.
Convention for cross-SDK tasks: prefer a single object argument
(`args = [ {…} ]`, `kwargs = {}`) — it maps cleanly onto every language's
handler-binding model.

**Result body** (`Job.result`): a bare CBOR value (no array wrapper).

**Cross-SDK rules**:
- A shell reading tag `0x00` on a payload it did not produce MUST fail with an
error naming the tag, not a generic decode error.
- Producer and consumer of a task MUST be configured with the same wire
serializer; the tag is a self-check, not a negotiation mechanism.
- Delivery-side semantics (retries, DLQ, acks) are unaffected — the envelope is
purely a payload contract.

**Test vectors** (hex, `0x02`-tagged CBOR):
- call `f(1, "a")`, no kwargs → `02 82 82 01 61 61 a0` — `[ [1, "a"], {} ]`
- result `true` → `02 f5`
- big int `2^53` → `02 1b 00 20 00 00 00 00 00 00`

## Dispatch call sequence
1. Shell constructs `Storage` (SQLite default; `postgres`/`redis` features) — `storage/traits.rs`.
Expand Down
3 changes: 3 additions & 0 deletions docs/content/docs/architecture/serialization.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ Taskito queue = Taskito.builder()
| `SmartSerializer` (default) | MessagePack + cloudpickle fallback | Mixed payloads; falls back to pickle for complex objects |
| `CloudpickleSerializer` | Binary (pickle) | Complex Python objects, lambdas, closures |
| `JsonSerializer` | JSON | Simple types, cross-language interop, debugging |
| `CborSerializer` | CBOR (wire-envelope tag `0x02`) | Cross-SDK tasks — big integers, datetimes, bytes round-trip losslessly |

</SdkOnly>

Expand All @@ -71,6 +72,7 @@ Taskito queue = Taskito.builder()
|---|---|---|
| `JsonSerializer` (default) | JSON | Simple types, cross-language interop, debugging |
| `MsgpackSerializer` | Binary (msgpack) | Compact payloads, smaller storage |
| `CborSerializer` | Binary (CBOR, wire-envelope tag `0x02`) | Cross-SDK tasks — big integers, `Date`, bytes round-trip losslessly |
| `SignedSerializer` | Wraps a codec + HMAC-SHA256 | Tamper detection |
| `EncryptedSerializer` | Wraps a codec + AES-256-GCM | Encrypted, authenticated payloads |

Expand All @@ -82,6 +84,7 @@ Taskito queue = Taskito.builder()
|---|---|---|
| `JsonSerializer` (default) | JSON (Jackson) | Simple types, cross-language interop, debugging |
| `MsgpackSerializer` | Binary (msgpack) | Compact payloads, smaller storage |
| `CborSerializer` | Binary (CBOR, wire-envelope tag `0x02`) | Cross-SDK tasks — big integers, decimals, bytes round-trip losslessly |
| `SignedSerializer` | Wraps a serializer + HMAC-SHA256 | Tamper detection |
| `EncryptedSerializer` | Wraps a serializer + AES-GCM | Encrypted, authenticated payloads |

Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/java/api-reference/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The public API lives under `org.byteveda.taskito` (Maven coordinates
| [Result](/java/api-reference/result) | Awaiting jobs and reading typed results. |
| [Context](/java/api-reference/context) | `Resources.use` and the middleware `TaskContext`. |
| [Resources](/java/api-reference/resources) | Worker dependency injection — scopes, pools, disposal. |
| [Serializers](/java/api-reference/serializers) | `JsonSerializer`, `MsgpackSerializer`, payload codecs. |
| [Serializers](/java/api-reference/serializers) | `JsonSerializer`, `MsgpackSerializer`, `CborSerializer`, payload codecs. |
| [Batching](/java/api-reference/batching) | `Batcher` — buffer payloads and flush them as one `enqueueMany` call. |
| [Workflows](/java/api-reference/workflows) | The `Workflow` builder, `WorkflowRun`, gates, sagas, analysis. |
| [Canvas](/java/api-reference/canvas) | `Canvas.link`, `chain`, `group`, and `chord` shortcuts. |
Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/java/api-reference/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Every package lives under `org.byteveda.taskito`:
| `worker` | `Worker`, `Worker.Builder`, `Handler`, `HandlerRegistry` |
| `workflows` | `Workflow`, `Step`, `Canvas`, `WorkflowRun`, gates, sagas, analysis |
| `resources` | Worker dependency injection — scopes, pools, `ResourceContext` |
| `serialization` | `Serializer`, `JsonSerializer`, `MsgpackSerializer`, payload codecs |
| `serialization` | `Serializer`, `JsonSerializer`, `MsgpackSerializer`, `CborSerializer`, payload codecs |
| `locks` | `Lock`, `LockInfo` |
| `predicates` | Enqueue-time gates — `Predicate`, `EnqueueGate`, `Recipes` |
| `interception` | Enqueue interceptors — `Interceptor`, `Interception` |
Expand Down
12 changes: 11 additions & 1 deletion docs/content/docs/java/api-reference/serializers.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Serializers
description: "JsonSerializer, MsgpackSerializer, payload codecs, and the Serializer interface."
description: "JsonSerializer, MsgpackSerializer, CborSerializer, payload codecs, and the Serializer interface."
---

```java
Expand All @@ -19,12 +19,22 @@ Taskito taskito = Taskito.builder()
|---|---|
| `JsonSerializer` | Default. JSON via Jackson; accepts a custom `ObjectMapper`. |
| `MsgpackSerializer` | Compact binary (MessagePack via Jackson). Its `org.msgpack:jackson-dataformat-msgpack` dependency is `compileOnly` — add it to your build. |
| `CborSerializer` | Binary (CBOR via Jackson), tagged with the `0x02` wire-envelope byte — the default format for tasks produced or consumed by another Taskito SDK. Its `com.fasterxml.jackson.dataformat:jackson-dataformat-cbor` dependency is `compileOnly` — add it to your build. |
| `SignedSerializer(delegate, key)` | Prefixes each payload with an HMAC-SHA256 tag; verifies it (constant-time) on read — authentication, not encryption. |
| `EncryptedSerializer(delegate, key)` | AES-GCM with a fresh 12-byte IV per payload — confidentiality **and** integrity. Key must be 16/24/32 bytes. |

Producers and workers must configure a compatible serializer — the Rust core
treats payloads as opaque bytes.

```java
import org.byteveda.taskito.serialization.CborSerializer;

Taskito taskito = Taskito.builder()
.sqlite("taskito.db")
.serializer(new CborSerializer())
.open();
```

## Payload codecs

A `PayloadCodec` is a byte-to-byte transform (`encode`/`decode`) applied after
Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/node/api-reference/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The public API exported from the `taskito` barrel. Contrib helpers live under
| [Worker](/node/api-reference/worker) | `runWorker` options and the worker handle. |
| [Result](/node/api-reference/result) | Awaiting results and the result waiter. |
| [Context](/node/api-reference/context) | `currentJob()` — signal, progress, ids. |
| [Serializers](/node/api-reference/serializers) | `JsonSerializer`, `MsgpackSerializer`, the `Serializer` interface. |
| [Serializers](/node/api-reference/serializers) | `JsonSerializer`, `MsgpackSerializer`, `CborSerializer`, the `Serializer` interface. |
| [Workflows](/node/api-reference/workflows) | The workflow builder + run handle. |
| [Errors](/node/api-reference/errors) | The `TaskitoError` hierarchy. |
| [CLI](/node/api-reference/cli) | The `taskito` command. |
9 changes: 8 additions & 1 deletion docs/content/docs/node/api-reference/serializers.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Serializers
description: "JsonSerializer, MsgpackSerializer, and the Serializer interface."
description: "JsonSerializer, MsgpackSerializer, CborSerializer, and the Serializer interface."
---

```ts
Expand All @@ -14,13 +14,20 @@ import type { Serializer } from "@byteveda/taskito";
|---|---|
| `JsonSerializer` | Default. Human-readable JSON bytes. |
| `MsgpackSerializer` | Compact binary (msgpack). |
| `CborSerializer` | Binary (CBOR, RFC 8949), tagged with the `0x02` wire-envelope byte — the default format for tasks produced or consumed by another Taskito SDK. Integers beyond `Number.MAX_SAFE_INTEGER` round-trip as `BigInt`. |
| `SignedSerializer` | Wraps another codec with an HMAC-SHA256 tag; rejects tampered payloads. |
| `EncryptedSerializer` | Wraps another codec with AES-256-GCM; encrypts + authenticates. |

```ts
new Queue({ dbPath: "taskito.db", serializer: new MsgpackSerializer() });
```

```ts
import { Queue, CborSerializer } from "@byteveda/taskito";

new Queue({ dbPath: "taskito.db", serializer: new CborSerializer() });
```

`SignedSerializer(secret, inner?)` prefixes each payload with a 32-byte HMAC tag
and verifies it (timing-safe) on read — authentication, not encryption.
`EncryptedSerializer(secret, inner?)` goes further: AES-256-GCM gives
Expand Down
104 changes: 89 additions & 15 deletions docs/content/docs/shared/guides/extensibility/serializers.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Pluggable Serializers
description: "Built-in JSON, MessagePack, signed, and encrypted serializers, plus the cross-SDK payload-codec chain for compression and encryption."
description: "Built-in JSON, MessagePack, CBOR, signed, and encrypted serializers, plus the cross-SDK payload-codec chain for compression and encryption."
---

Task arguments and results are serialized with a pluggable `Serializer`. The
Expand All @@ -26,15 +26,16 @@ strings, numbers, booleans, `None`, tuples), with an automatic
<SdkOnly sdk="node">

The default is `JsonSerializer` — human-readable and easy to debug. Other
built-ins: `MsgpackSerializer`, `SignedSerializer`, `EncryptedSerializer`.
built-ins: `MsgpackSerializer`, `CborSerializer`, `SignedSerializer`,
`EncryptedSerializer`.

</SdkOnly>

<SdkOnly sdk="java">

The default is `JsonSerializer` — Jackson-backed, human-readable and easy to
debug. Other built-ins: `MsgpackSerializer`, `SignedSerializer`,
`EncryptedSerializer`.
debug. Other built-ins: `MsgpackSerializer`, `CborSerializer`,
`SignedSerializer`, `EncryptedSerializer`.

</SdkOnly>

Expand Down Expand Up @@ -176,6 +177,77 @@ you want to spell it out in your own requirements file.

</SdkOnly>

### CborSerializer

Binary serialization writing the `0x02` wire-envelope tag — the default
format for tasks produced or consumed by another Taskito SDK. Unlike JSON,
big integers, `datetime`/`Date`, `bytes`, and decimals round-trip losslessly
across languages.

<CodeTabs>
<Tab sdk="python">

```python
from taskito import CborSerializer, Queue

queue = Queue(serializer=CborSerializer())
```

</Tab>
<Tab sdk="node">

```ts
import { Queue, CborSerializer } from "@byteveda/taskito";

new Queue({ dbPath: "taskito.db", serializer: new CborSerializer() });
```

</Tab>
<Tab sdk="java">

```java
import org.byteveda.taskito.Taskito;
import org.byteveda.taskito.serialization.CborSerializer;

Taskito taskito = Taskito.builder()
.sqlite("taskito.db")
.serializer(new CborSerializer())
.open();
```

</Tab>
</CodeTabs>

<SdkOnly sdk="python">

`cbor2` ships as a required taskito dependency — no extra install needed. The
default `SmartSerializer` can already *read* `0x02`-tagged CBOR payloads (for
example one written by another Taskito SDK) without extra configuration; use
`CborSerializer` explicitly to *write* cross-SDK payloads.

</SdkOnly>

<SdkOnly sdk="node">

`cbor-x` ships as a core dependency — no extra install required. Integers
beyond `Number.MAX_SAFE_INTEGER` round-trip as `BigInt`.

</SdkOnly>

<SdkOnly sdk="java">

`CborSerializer`'s `com.fasterxml.jackson.dataformat:jackson-dataformat-cbor`
dependency is `compileOnly` — add it to your build explicitly, same as
`MsgpackSerializer`.

</SdkOnly>

<Callout type="info" title="Cross-SDK payload contract">
Producer and consumer of a task must configure the same wire serializer.
For cross-SDK tasks, prefer a single object/dict argument — it maps
cleanly onto every language's handler-binding model.
</Callout>

### SignedSerializer

HMAC-SHA256 integrity tag. Unlike `EncryptedSerializer`, it does **not** hide
Expand Down Expand Up @@ -411,21 +483,23 @@ override it in a generics-aware implementation, as `JsonSerializer` does.

## Choosing a serializer

| | SmartSerializer | CloudpickleSerializer | JsonSerializer | MsgPackSerializer | EncryptedSerializer |
|---|---|---|---|---|---|
| **Complex objects** | Yes (cloudpickle fallback) | Yes | No | No | Depends on inner serializer |
| **Debugging** | Binary payloads (opaque) | Binary payloads (opaque) | Human-readable JSON | Binary (opaque) | Ciphertext (opaque) |
| **Cross-language** | Python only | Python only | Any language | Any language | Python only (by default) |
| **Performance** | Best for plain data | Good | Good for simple types | Best | Adds encryption overhead |
| **Security** | None | None | None | None | AES-256-GCM |
| **Extra dependency** | No | No | No | No | `cryptography` (`taskito[encryption]`) |
| **Default** | Yes | No | No | No | No |
| | SmartSerializer | CloudpickleSerializer | JsonSerializer | MsgPackSerializer | CborSerializer | EncryptedSerializer |
|---|---|---|---|---|---|---|
| **Complex objects** | Yes (cloudpickle fallback) | Yes | No | No | No (CBOR-representable types only) | Depends on inner serializer |
| **Debugging** | Binary payloads (opaque) | Binary payloads (opaque) | Human-readable JSON | Binary (opaque) | Binary (opaque) | Ciphertext (opaque) |
| **Cross-language** | Python only | Python only | Portable format, same-SDK payloads | Portable format, same-SDK payloads | Yes (cross-SDK wire format) | Python only (by default) |
| **Performance** | Best for plain data | Good | Good for simple types | Best | Good | Adds encryption overhead |
| **Security** | None | None | None | None | None | AES-256-GCM |
| **Extra dependency** | No | No | No | No | No | `cryptography` (`taskito[encryption]`) |
| **Default** | Yes | No | No | No | No | No |

**Rule of thumb**: use the default `SmartSerializer` unless you have a
specific reason to switch — it gets you compact MessagePack payloads for
plain data with an automatic cloudpickle fallback for anything MessagePack
can't encode. Use `EncryptedSerializer` when tasks carry sensitive data that
must not be readable in the database. See
can't encode. Use `CborSerializer` when a task is produced or consumed by
another Taskito SDK — it's the only serializer here whose wire format is
part of the cross-SDK contract. Use `EncryptedSerializer` when tasks carry
sensitive data that must not be readable in the database. See
Comment thread
coderabbitai[bot] marked this conversation as resolved.
[Architecture: Serialization](/architecture/serialization) for a shorter
overview and what gets serialized where.

Expand Down
5 changes: 5 additions & 0 deletions sdks/java/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,11 @@ dependencies {
compileOnly("org.msgpack:jackson-dataformat-msgpack:0.9.8")
testImplementation("org.msgpack:jackson-dataformat-msgpack:0.9.8")

// Optional: the CBOR wire serializer (cross-SDK payloads). Same model —
// consumers that use CborSerializer add this dependency themselves.
compileOnly("com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.17.2")
testImplementation("com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.17.2")

// Optional: observability contrib middleware. Consumers that use them add the
// matching runtime dependency themselves.
compileOnly("io.micrometer:micrometer-observation:1.13.6")
Expand Down
10 changes: 5 additions & 5 deletions sdks/java/src/main/java/org/byteveda/taskito/DefaultTaskito.java
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ private Optional<String> dispatchEnqueue(
// Serialize before codec-encoding so the idempotency key hashes the deterministic
// pre-codec payload — a non-deterministic codec (e.g. an AES-GCM nonce) must not
// change the dedup key.
byte[] payloadBytes = serializer.serialize(context.payload());
byte[] payloadBytes = serializer.serializeCall(context.payload());
String uniqueKey = resolveUniqueKey(taskName, payloadBytes, finalOptions, taskIdempotentDefault);
if (uniqueKey != null && !uniqueKey.equals(finalOptions.uniqueKey())) {
finalOptions = finalOptions.toBuilder().uniqueKey(uniqueKey).build();
Expand Down Expand Up @@ -359,7 +359,7 @@ public <T> List<String> enqueueMany(Task<T> task, List<T> payloads, EnqueueOptio
// auto-derive, mirroring the single-enqueue path (explicit
// uniqueKey/idempotencyKey precedence preserved). Key hashes the
// deterministic pre-codec bytes.
byte[] payloadBytes = serializer.serialize(payload);
byte[] payloadBytes = serializer.serializeCall(payload);
EnqueueOptions jobOptions = options;
String uniqueKey = resolveUniqueKey(task.name(), payloadBytes, jobOptions, task.idempotent());
if (uniqueKey != null && !uniqueKey.equals(jobOptions.uniqueKey())) {
Expand Down Expand Up @@ -634,7 +634,7 @@ public Optional<LockInfo> getLockInfo(String name) {

@Override
public long registerPeriodic(PeriodicTask task) {
byte[] payload = task.payload == null ? null : serializer.serialize(task.payload);
byte[] payload = task.payload == null ? null : serializer.serializeCall(task.payload);
return backend.registerPeriodic(
task.name, task.taskName, task.cron, payload, task.queue, task.timezone, task.enabled);
}
Expand Down Expand Up @@ -688,7 +688,7 @@ public WorkflowRun submitWorkflow(Workflow workflow, Map<String, Object> supplie
+ "' has no payload; supply one via submitWorkflow(wf, payloads)");
}
payloadNames.add(step.name);
payloads.add(serializer.serialize(payload));
payloads.add(serializer.serializeCall(payload));
}
}
String runId = backend.submitWorkflow(
Expand Down Expand Up @@ -869,7 +869,7 @@ private String encodeChild(Workflow child) {
}
specs.add(stepSpec(step));
if (step.payload != null) {
payloads.put(step.name, Base64.getEncoder().encodeToString(serializer.serialize(step.payload)));
payloads.put(step.name, Base64.getEncoder().encodeToString(serializer.serializeCall(step.payload)));
}
}
Map<String, Object> blob = new LinkedHashMap<>();
Expand Down
Loading