From c338683748b5e389228ea1889bf68a045096a4e3 Mon Sep 17 00:00:00 2001
From: Quantum Explorer <quantum@dash.org>
Date: Fri, 20 Mar 2026 11:25:24 +0300
Subject: [PATCH 1/4] docs(book): add document serialization wire format
 chapter

Describes the binary wire format for serialized documents, including
the version varint, field layout, time fields bitfield, property
encoding rules, and common pitfalls for third-party deserializers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 book/src/SUMMARY.md                           |   1 +
 .../serialization/document-serialization.md   | 231 ++++++++++++++++++
 2 files changed, 232 insertions(+)
 create mode 100644 book/src/serialization/document-serialization.md

diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md
index d547b218514..f8859d679ae 100644
--- a/book/src/SUMMARY.md
+++ b/book/src/SUMMARY.md
@@ -40,6 +40,7 @@
 # Serialization
 
 - [Platform Serialization](serialization/platform-serialization.md)
+- [Document Serialization](serialization/document-serialization.md)
 - [Derive Macros](serialization/derive-macros.md)
 
 # Addresses
diff --git a/book/src/serialization/document-serialization.md b/book/src/serialization/document-serialization.md
new file mode 100644
index 00000000000..6d860436f74
--- /dev/null
+++ b/book/src/serialization/document-serialization.md
@@ -0,0 +1,231 @@
+# Document Serialization
+
+This chapter describes the binary wire format used to serialize and deserialize documents on Dash Platform. If you are building a tool that reads documents from gRPC responses, GroveDB storage, or any other source of raw document bytes, this is the specification you need.
+
+Documents are **not self-describing**. The binary format does not include field names or type tags. To interpret the bytes, you must know which data contract and document type the document belongs to. The document type's schema defines the field order, types, and sizes that determine how the bytes map to values.
+
+## High-level structure
+
+Every serialized document follows this layout:
+
+```
+┌──────────────────────┐
+│  Serialization       │  varint (1-2 bytes)
+│  Version             │  Currently: 0, 1, or 2
+├──────────────────────┤
+│  $id                 │  32 bytes
+├──────────────────────┤
+│  $ownerId            │  32 bytes
+├──────────────────────┤
+│  $creatorId          │  V2 only, if document type supports
+│  (optional)          │  transfers/trading: 1 byte flag + 32 bytes
+├──────────────────────┤
+│  $revision           │  varint (if document type is mutable)
+├──────────────────────┤
+│  Time fields         │  2-byte bitfield + variable-length data
+│  bitfield + data     │
+├──────────────────────┤
+│  Price               │  Only if document type supports trading
+│  (optional)          │  1 byte flag + 8 bytes
+├──────────────────────┤
+│  User properties     │  Variable length, schema-dependent
+│  (in schema order)   │
+└──────────────────────┘
+```
+
+Source: `packages/rs-dpp/src/document/v0/serialize.rs`
+
+## Serialization versions
+
+The first bytes of a serialized document are a **varint** encoding the serialization version. This is not the protocol version or the document struct version — it is the version of the serialization format itself.
+
+| Serialization Version | Description |
+|----|-------------|
+| 0 | Original format. All integers encoded as **i64** (8 bytes big-endian) regardless of their schema type. |
+| 1 | Integers encoded at their **native size** (u8 = 1 byte, u16 = 2 bytes, u32 = 4 bytes, etc.). Otherwise identical to v0. |
+| 2 | Same as v1, but adds **`$creatorId`** field after `$ownerId` for document types that support transfers or trading. |
+
+The varint encoding uses the [`integer-encoding`](https://docs.rs/integer-encoding) crate's `VarInt` format. For values 0, 1, and 2, the varint is a single byte: `0x00`, `0x01`, or `0x02`.
+
+```rust
+// Serialization version is written first
+let mut buffer: Vec<u8> = 2u64.encode_var_vec(); // version 2 → byte 0x02
+```
+
+When deserializing, the version varint is read first, then the appropriate deserialization logic is dispatched:
+
+```rust
+let serialized_version: u64 = serialized_document.read_varint()?;
+match serialized_version {
+    0 => DocumentV0::from_bytes_v0(serialized_document, document_type, platform_version),
+    1 => DocumentV0::from_bytes_v1(serialized_document, document_type, platform_version),
+    2 => DocumentV0::from_bytes_v2(serialized_document, document_type, platform_version),
+    _ => Err(/* unknown version */),
+}
+```
+
+Note: version 0 has a fallback — if deserialization as v0 (all i64) fails, it retries as v1 (native integer types). This handles edge cases from protocol versions 1–8 where the version byte was 0 but non-i64 integer types may have been used.
+
+## Field-by-field breakdown
+
+### `$id` (32 bytes)
+
+The document's unique identifier, written as raw bytes. This is a 256-bit value derived from the contract ID, owner ID, document type name, and entropy via double SHA-256.
+
+### `$ownerId` (32 bytes)
+
+The identity that currently owns the document, written as raw bytes.
+
+### `$creatorId` (v2 only, conditional)
+
+Present only in serialization version 2, and only if the document type supports transfers (`documents_transferable`) or trading (`trade_mode != None`).
+
+```
+0x01  [32 bytes creatorId]    — creator ID present
+0x00                           — creator ID absent
+```
+
+### `$revision` (varint, conditional)
+
+Present only if the document type requires revisions (mutable documents). Encoded as a varint (`u64`). New documents start at revision 1 (`INITIAL_REVISION`).
+
+### Time fields (2-byte bitfield + data)
+
+Time-related fields use a compact encoding with a **bitfield** to indicate which fields are present, followed by the data for each present field.
+
+**Bitfield** (2 bytes, big-endian `u16`):
+
+| Bit | Field |
+|-----|-------|
+| 0 (0x0001) | `$createdAt` |
+| 1 (0x0002) | `$updatedAt` |
+| 2 (0x0004) | `$transferredAt` |
+| 3 (0x0008) | `$createdAtBlockHeight` |
+| 4 (0x0010) | `$updatedAtBlockHeight` |
+| 5 (0x0020) | `$transferredAtBlockHeight` |
+| 6 (0x0040) | `$createdAtCoreBlockHeight` |
+| 7 (0x0080) | `$updatedAtCoreBlockHeight` |
+| 8 (0x0100) | `$transferredAtCoreBlockHeight` |
+
+**Data**: For each bit that is set (in the order above), the corresponding value is appended:
+
+- `$createdAt`, `$updatedAt`, `$transferredAt`: **8 bytes big-endian u64** — milliseconds since Unix epoch
+- `$createdAtBlockHeight`, `$updatedAtBlockHeight`, `$transferredAtBlockHeight`: **8 bytes big-endian u64** — platform block height
+- `$createdAtCoreBlockHeight`, `$updatedAtCoreBlockHeight`, `$transferredAtCoreBlockHeight`: **4 bytes big-endian u32** — core chain block height
+
+For example, if a document has `$createdAt` and `$updatedAt` set, the bitfield would be `0x0003`, followed by 16 bytes (8 for each timestamp).
+
+### Price (conditional)
+
+If the document type's `trade_mode` allows seller-set pricing:
+
+```
+0x01  [8 bytes big-endian u64]   — price in credits
+0x00                              — no price set
+```
+
+### User-defined properties
+
+Properties are serialized **in the order defined by the document type's schema** (`document_type.properties()` returns a `BTreeMap`, so properties are in alphabetical order by field name).
+
+Each property is encoded based on its type and whether it is required:
+
+**Required fields**: The value is written directly with no prefix byte.
+
+**Optional fields**: A 1-byte presence flag is written first:
+- `0x01` followed by the encoded value — field is present
+- `0x00` — field is absent
+
+#### Value encoding by type
+
+All numeric values use **big-endian** byte order.
+
+| Type | Encoding |
+|------|----------|
+| `u8` / `i8` | 1 byte |
+| `u16` / `i16` | 2 bytes big-endian |
+| `u32` / `i32` | 4 bytes big-endian |
+| `u64` / `i64` | 8 bytes big-endian |
+| `u128` / `i128` | 16 bytes big-endian |
+| `f64` | 8 bytes big-endian IEEE 754 |
+| `boolean` | 1 byte: `0x01` = true, `0x00` = false |
+| `string` | varint length prefix + UTF-8 bytes |
+| `byteArray` (fixed size) | raw bytes (no length prefix if min_size == max_size) |
+| `byteArray` (variable size) | varint length prefix + raw bytes |
+| `identifier` | 32 bytes raw |
+| `date` | 8 bytes big-endian f64 |
+| `object` | Nested fields serialized recursively in their schema order |
+
+**Important**: In serialization version 0, all integer types are encoded as **i64** (8 bytes big-endian), regardless of the actual type in the schema. This means a `u8` field that should be 1 byte is encoded as 8 bytes in v0.
+
+## Worked example: withdrawal document
+
+The withdrawals contract defines a `withdrawal` document type with these properties (in alphabetical/BTreeMap order):
+
+| Property | Type | Required |
+|----------|------|----------|
+| `amount` | integer (i64) | yes |
+| `coreFeePerByte` | integer (i64) | yes |
+| `outputScript` | byteArray (23–25 bytes, variable) | yes |
+| `pooling` | integer (i64) | yes |
+| `status` | integer (i64) | yes |
+| `transactionIndex` | integer (i64) | no |
+| `transactionSignHeight` | integer (i64) | no |
+
+The document type requires `$createdAt`, `$updatedAt`, and `$revision`.
+
+Here is a real serialized withdrawal document (hex), broken down byte by byte:
+
+```
+02                                      ← serialization version (varint: 2)
+02                                      ← (continuation of varint, actual value = 2)
+229eda94b35be55ac222ca8cc4631c0717      ← $id (32 bytes)
+c9ee4a223f2a269e06c9a1be7c5436
+b3e63ba54aba9b75994128d124e9e1ce      ← $ownerId (32 bytes)
+be348cd30415b5098c60526de0157ec5
+01                                      ← (v2: no $creatorId since withdrawals
+00                                         are not transferable — this is part
+03                                         of revision + flags region)
+0000019cd70f3323                        ← $createdAt: 1773134623523 ms
+                                          (2026-03-10 09:23:43 UTC)
+0000019d05406f0c                        ← $updatedAt: 1773909602060 ms
+                                          (2026-03-19 08:40:02 UTC)
+0100000000000026                        ← amount: 9815... wait, this needs
+                                          the schema to parse correctly
+```
+
+To properly decode the properties section, you need the document type schema — field names, types, required flags, and order. This is why the `decode-document` CLI tool (in `packages/rs-scripts`) requires the contract and document type to be specified.
+
+## The `decode-document` CLI tool
+
+For convenience, the `rs-scripts` crate provides a `decode-document` binary that handles all of this deserialization automatically:
+
+```bash
+# Install
+cargo install --path packages/rs-scripts
+
+# Decode a withdrawal document from base64
+decode-document -c withdrawals -d withdrawal "AgIintqUs1vl..."
+
+# Decode from hex
+decode-document -c withdrawals -d withdrawal "0202229eda94b35b..."
+
+# Use a contract ID instead of a name
+decode-document -c 4fJLR2GYTPFdomuTVvNy3VRrvWgvkKPzqehEBpNf2nk6 -d withdrawal "..."
+```
+
+See `packages/rs-scripts/README.md` for full usage details.
+
+## Common pitfalls for third-party deserializers
+
+1. **The serialization version varint changed the layout.** If your code was written for version 0 or 1, version 2 documents will have different field offsets due to the `$creatorId` field. Always read the version varint first and branch accordingly.
+
+2. **Integer encoding differs between v0 and v1+.** In version 0, a `u8` field occupies 8 bytes (encoded as i64). In version 1+, it occupies 1 byte. Parsing with the wrong version assumption will shift every subsequent field.
+
+3. **The time fields bitfield is variable-length data.** The 2-byte bitfield tells you how many time fields follow. If you assume a fixed number of time fields, any document with a different set of time fields (e.g., one that includes `$transferredAt` or block heights) will be misaligned.
+
+4. **Property order is alphabetical (BTreeMap order), not declaration order.** If you hard-code field offsets based on the JSON schema declaration order rather than alphabetical order, fields will be read from the wrong positions.
+
+5. **Optional fields have a presence byte.** If you forget to read the `0x00`/`0x01` prefix for optional fields, every subsequent field will be shifted by one byte.
+
+6. **ByteArray encoding depends on size constraints.** Fixed-size byte arrays (where `minItems == maxItems` in the schema) have no length prefix. Variable-size byte arrays have a varint length prefix. Check the schema to know which encoding is used.

From d61d2fb73a3bdb4309eadbf79728bb7252b1edf3 Mon Sep 17 00:00:00 2001
From: Quantum Explorer <quantum@dash.org>
Date: Fri, 20 Mar 2026 11:46:20 +0300
Subject: [PATCH 2/4] docs(book): fix document serialization code blocks and
 varint example
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Address CodeRabbit review feedback:
- Add `text` language tags to fenced code blocks (MD040)
- Fix incorrect varint continuation line — version 2 is a single-byte
  varint, the second 0x02 was the first byte of $id
- Rewrite worked example with correct byte offsets and add decoded
  tool output for clarity

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 .../serialization/document-serialization.md   | 47 +++++++++++++------
 1 file changed, 33 insertions(+), 14 deletions(-)

diff --git a/book/src/serialization/document-serialization.md b/book/src/serialization/document-serialization.md
index 6d860436f74..6b5c1d56287 100644
--- a/book/src/serialization/document-serialization.md
+++ b/book/src/serialization/document-serialization.md
@@ -8,7 +8,7 @@ Documents are **not self-describing**. The binary format does not include field
 
 Every serialized document follows this layout:
 
-```
+```text
 ┌──────────────────────┐
 │  Serialization       │  varint (1-2 bytes)
 │  Version             │  Currently: 0, 1, or 2
@@ -80,7 +80,7 @@ The identity that currently owns the document, written as raw bytes.
 
 Present only in serialization version 2, and only if the document type supports transfers (`documents_transferable`) or trading (`trade_mode != None`).
 
-```
+```text
 0x01  [32 bytes creatorId]    — creator ID present
 0x00                           — creator ID absent
 ```
@@ -119,7 +119,7 @@ For example, if a document has `$createdAt` and `$updatedAt` set, the bitfield w
 
 If the document type's `trade_mode` allows seller-set pricing:
 
-```
+```text
 0x01  [8 bytes big-endian u64]   — price in credits
 0x00                              — no price set
 ```
@@ -176,26 +176,45 @@ The document type requires `$createdAt`, `$updatedAt`, and `$revision`.
 
 Here is a real serialized withdrawal document (hex), broken down byte by byte:
 
-```
+```text
 02                                      ← serialization version (varint: 2)
-02                                      ← (continuation of varint, actual value = 2)
-229eda94b35be55ac222ca8cc4631c0717      ← $id (32 bytes)
-c9ee4a223f2a269e06c9a1be7c5436
-b3e63ba54aba9b75994128d124e9e1ce      ← $ownerId (32 bytes)
-be348cd30415b5098c60526de0157ec5
-01                                      ← (v2: no $creatorId since withdrawals
-00                                         are not transferable — this is part
-03                                         of revision + flags region)
+0222 9eda 94b3 5be5 5ac2 22ca 8cc4      ← $id (32 bytes)
+631c 0717 c9ee 4a22 3f2a 269e 06c9
+a1be 7c54
+36b3 e63b a54a ba9b 7599 4128 d124      ← $ownerId (32 bytes)
+e9e1 cebe 348c d304 15b5 098c 6052
+6de0 157e
+c501                                    ← $revision (varint: 197)
+0003                                    ← time bitfield: bits 0,1 set
+                                          ($createdAt + $updatedAt)
 0000019cd70f3323                        ← $createdAt: 1773134623523 ms
                                           (2026-03-10 09:23:43 UTC)
 0000019d05406f0c                        ← $updatedAt: 1773909602060 ms
                                           (2026-03-19 08:40:02 UTC)
-0100000000000026                        ← amount: 9815... wait, this needs
-                                          the schema to parse correctly
+                                        ← user properties follow (schema-dependent)
 ```
 
 To properly decode the properties section, you need the document type schema — field names, types, required flags, and order. This is why the `decode-document` CLI tool (in `packages/rs-scripts`) requires the contract and document type to be specified.
 
+Using the tool on this document produces:
+
+```text
+id:         9LSAr59Fw7A1PHvX9WV1RWHCjL4PrijrHZpwhYDPkMq
+owner_id:   4gY7wFM4o53jc8PJZ9KNzqzaJhXhPVMivJREKVwihKVF
+created_at: 2026-03-10 09:23:43 UTC
+updated_at: 2026-03-19 08:40:02 UTC
+revision:   197
+
+properties:
+  amount: (i64)191000
+  coreFeePerByte: (i64)1
+  outputScript: bytes 76a914...88ac
+  pooling: (i64)0
+  status: (i64)2
+  transactionIndex: (i64)9815
+  transactionSignHeight: (i64)2440497
+```
+
 ## The `decode-document` CLI tool
 
 For convenience, the `rs-scripts` crate provides a `decode-document` binary that handles all of this deserialization automatically:

From 1425adfcbb186698fe8c53470a614253e672e076 Mon Sep 17 00:00:00 2001
From: Quantum Explorer <quantum@dash.org>
Date: Fri, 20 Mar 2026 13:57:16 +0300
Subject: [PATCH 3/4] fix(docs): correct document serialization order and add
 missing type details

- Properties use schema position order (IndexMap), not alphabetical (BTreeMap)
- Document transient fields always get a presence byte even when required
- Add array type encoding to the type table
- Note 0xff prefix for optional date fields
- Clarify user-property date (f64) vs system timestamp (u64) encoding
- Note why $creatorId is absent in the v2 worked example

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 book/src/serialization/document-serialization.md | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/book/src/serialization/document-serialization.md b/book/src/serialization/document-serialization.md
index 6b5c1d56287..62643fe7ef1 100644
--- a/book/src/serialization/document-serialization.md
+++ b/book/src/serialization/document-serialization.md
@@ -126,7 +126,7 @@ If the document type's `trade_mode` allows seller-set pricing:
 
 ### User-defined properties
 
-Properties are serialized **in the order defined by the document type's schema** (`document_type.properties()` returns a `BTreeMap`, so properties are in alphabetical order by field name).
+Properties are serialized **in schema position order** — each property in the data contract schema has a `position` field, and `document_type.properties()` returns an `IndexMap` sorted by that position. This is *not* alphabetical order.
 
 Each property is encoded based on its type and whether it is required:
 
@@ -136,6 +136,8 @@ Each property is encoded based on its type and whether it is required:
 - `0x01` followed by the encoded value — field is present
 - `0x00` — field is absent
 
+**Transient fields**: Always get a presence byte, even if marked as required. The serializer checks `if !property.required || property.transient` to decide whether to write the flag.
+
 #### Value encoding by type
 
 All numeric values use **big-endian** byte order.
@@ -153,14 +155,17 @@ All numeric values use **big-endian** byte order.
 | `byteArray` (fixed size) | raw bytes (no length prefix if min_size == max_size) |
 | `byteArray` (variable size) | varint length prefix + raw bytes |
 | `identifier` | 32 bytes raw |
-| `date` | 8 bytes big-endian f64 |
-| `object` | Nested fields serialized recursively in their schema order |
+| `date` | 8 bytes big-endian f64 (when optional: `0xff` prefix + 8 bytes) |
+| `array` | varint element count + each element encoded in sequence |
+| `object` | Nested fields serialized recursively in their schema position order |
+
+**Note on date types**: User-property `date` fields are encoded as **f64** (8 bytes). System timestamps (`$createdAt`, `$updatedAt`, `$transferredAt`) are **u64** milliseconds. Both are 8 bytes big-endian but use different numeric representations.
 
 **Important**: In serialization version 0, all integer types are encoded as **i64** (8 bytes big-endian), regardless of the actual type in the schema. This means a `u8` field that should be 1 byte is encoded as 8 bytes in v0.
 
 ## Worked example: withdrawal document
 
-The withdrawals contract defines a `withdrawal` document type with these properties (in alphabetical/BTreeMap order):
+The withdrawals contract defines a `withdrawal` document type with these properties (listed here in schema position order):
 
 | Property | Type | Required |
 |----------|------|----------|
@@ -184,6 +189,8 @@ a1be 7c54
 36b3 e63b a54a ba9b 7599 4128 d124      ← $ownerId (32 bytes)
 e9e1 cebe 348c d304 15b5 098c 6052
 6de0 157e
+                                        ← no $creatorId: withdrawal document type
+                                          does not support transfers/trading
 c501                                    ← $revision (varint: 197)
 0003                                    ← time bitfield: bits 0,1 set
                                           ($createdAt + $updatedAt)

From 6355f3580adc44f63beee2455814de2582e1b879 Mon Sep 17 00:00:00 2001
From: Quantum Explorer <quantum@dash.org>
Date: Fri, 20 Mar 2026 19:59:24 +0300
Subject: [PATCH 4/4] fix(docs): correct property table order and pitfall #4
 for schema position order

- Reorder worked example property table to match actual schema positions
- Fix pitfall #4 to say schema position order, not alphabetical/BTreeMap

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 .../serialization/document-serialization.md   | 24 +++++++++----------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/book/src/serialization/document-serialization.md b/book/src/serialization/document-serialization.md
index 62643fe7ef1..a0b68c76b9e 100644
--- a/book/src/serialization/document-serialization.md
+++ b/book/src/serialization/document-serialization.md
@@ -165,17 +165,17 @@ All numeric values use **big-endian** byte order.
 
 ## Worked example: withdrawal document
 
-The withdrawals contract defines a `withdrawal` document type with these properties (listed here in schema position order):
-
-| Property | Type | Required |
-|----------|------|----------|
-| `amount` | integer (i64) | yes |
-| `coreFeePerByte` | integer (i64) | yes |
-| `outputScript` | byteArray (23–25 bytes, variable) | yes |
-| `pooling` | integer (i64) | yes |
-| `status` | integer (i64) | yes |
-| `transactionIndex` | integer (i64) | no |
-| `transactionSignHeight` | integer (i64) | no |
+The withdrawals contract defines a `withdrawal` document type with these properties (in schema position order):
+
+| Position | Property | Type | Required |
+|----------|----------|------|----------|
+| 0 | `transactionIndex` | integer (i64) | no |
+| 1 | `transactionSignHeight` | integer (i64) | no |
+| 2 | `amount` | integer (i64) | yes |
+| 3 | `coreFeePerByte` | integer (i64) | yes |
+| 4 | `pooling` | integer (i64) | yes |
+| 5 | `outputScript` | byteArray (23–25 bytes, variable) | yes |
+| 6 | `status` | integer (i64) | yes |
 
 The document type requires `$createdAt`, `$updatedAt`, and `$revision`.
 
@@ -250,7 +250,7 @@ See `packages/rs-scripts/README.md` for full usage details.
 
 3. **The time fields bitfield is variable-length data.** The 2-byte bitfield tells you how many time fields follow. If you assume a fixed number of time fields, any document with a different set of time fields (e.g., one that includes `$transferredAt` or block heights) will be misaligned.
 
-4. **Property order is alphabetical (BTreeMap order), not declaration order.** If you hard-code field offsets based on the JSON schema declaration order rather than alphabetical order, fields will be read from the wrong positions.
+4. **Property order is schema position order, not alphabetical.** Each property in the data contract schema has a `position` field. Properties are serialized in ascending position order (stored in an `IndexMap`). If you assume alphabetical order or JSON declaration order, fields will be read from the wrong positions.
 
 5. **Optional fields have a presence byte.** If you forget to read the `0x00`/`0x01` prefix for optional fields, every subsequent field will be shifted by one byte.