From 24019f66b85edeb849aeb3c6cf7392313c1ee96a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 08:48:11 +0900 Subject: [PATCH 01/97] docs: define incremental thread index design --- ...6-08-05-incremental-thread-index-design.md | 200 ++++++++++++++++++ 1 file changed, 200 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-05-incremental-thread-index-design.md diff --git a/docs/superpowers/specs/2026-08-05-incremental-thread-index-design.md b/docs/superpowers/specs/2026-08-05-incremental-thread-index-design.md new file mode 100644 index 0000000..380f2cd --- /dev/null +++ b/docs/superpowers/specs/2026-08-05-incremental-thread-index-design.md @@ -0,0 +1,200 @@ +# Incremental Thread Index Design + +## Status + +Approved product direction for issue #19. This design is implemented on a feature +branch while release blocker #17 keeps the `0.2.0` source on `main` frozen. The +feature must remain draft and must not merge until the `0.2.0` release succeeds. + +## Product problem + +`thread_messages` is a deterministic and standards-grounded batch operation, but +mail servers, migration products, archive viewers, and naruon-style control planes +receive mailbox deltas. Rebuilding an unrelated million-message mailbox after every +arrival, expunge, or metadata correction wastes CPU and memory and gives integrators +no explicit stable-identity handoff. + +## Goals + +1. Accept additions, replacements, and removals as one atomic change set. +2. Use an immutable caller-owned message key that is independent of IMAP sequence + numbers and UIDs. +3. Recompute only the old components and newly connected buckets touched by a + change while preserving exact parity with a full `thread_messages` rebuild. +4. Expose deterministic thread projections and explicit merge/split transitions. +5. Carry optional caller-owned RFC 8474 `EMAILID` and `THREADID` metadata without + silently changing a reported identifier. +6. Snapshot and restore versioned JSON-safe state without serializing payloads. +7. Preserve zero runtime dependencies, Python 3.10-3.13 support, iterative graph + processing, complete docstrings, and 100% statement and branch coverage. + +## Non-goals for this slice + +- Implementing an IMAP, JMAP, database, socket, authentication, or tenant layer. +- Inventing a server-owned stable `THREADID` policy. +- Persisting arbitrary caller payloads. +- Replacing `thread_messages`; the batch function remains the correctness oracle. +- Claiming sublinear behavior for an initial build or snapshot restore. The bounded + update path, not construction or restore, carries the incremental requirement. + +## Public API + +### `IndexedMessage` + +A frozen record containing: + +- `message_key: str`: immutable caller-owned key. +- `message: Message`: message metadata; the index copies structural fields on entry. +- `email_id: str | None`: optional caller-owned RFC 8474 immutable content ID. +- `thread_id: str | None`: optional caller-owned RFC 8474 thread correlator. + +Payload objects remain caller-owned and are retained only in memory. They never enter +snapshots or equality/delta calculations. + +### `MailboxChangeSet` + +A frozen atomic request containing: + +- `expected_version: int` +- `additions: tuple[IndexedMessage, ...]` +- `replacements: tuple[IndexedMessage, ...]` +- `removals: tuple[str, ...]` + +The three key sets must be disjoint. Additions require absent keys; replacements and +removals require existing keys. Any failure leaves the index unchanged. + +### `ThreadProjection` + +A frozen, JSON-safe description of one returned thread root: + +- `message_keys`: traversal-ordered caller keys. +- `thread_ids`: sorted distinct external thread IDs represented by the root. + +### `ThreadDelta` + +A deterministic result containing previous/new versions, affected message keys, +added/removed/updated projections, and explicit merge/split external-ID groups. +Structural overlap is computed from caller keys; mutable IMAP sequence numbers never +serve as identity. + +### `IncrementalThreadIndex` + +Constructor options mirror the batch API: + +- `group_by_subject: bool = False` +- `sort_by_sent_date: bool = False` +- `max_snapshot_records: int = 1_000_000` +- `max_snapshot_bytes: int = 256 * 1024 * 1024` + +Methods and properties: + +- `apply(change_set) -> ThreadDelta` +- `roots -> tuple[Container, ...]` +- `projections -> tuple[ThreadProjection, ...]` +- `version -> int` +- `snapshot() -> dict[str, object]` +- `restore(snapshot, *, max_snapshot_records=..., max_snapshot_bytes=...)` + +## Incremental algorithm + +The index keeps four kinds of state: + +1. Ordered records and stable insertion positions. +2. Per-message connectivity tokens. +3. Reverse token buckets. +4. Component membership and component-local `thread_messages` output. + +A record contributes tokens for its normalized `Message-ID`, every effective RFC +5256 reference identifier, and—when subject grouping is enabled—its RFC 5051 base +subject key. Token connectivity deliberately over-approximates the final root graph; +that can recompute a larger component but cannot omit a dependency. + +For one change set: + +1. Validate version, key ownership, immutable external IDs, numeric metadata, and + JSON-safe identifier bounds without mutating state. +2. Seed the affected set with every old component containing a replaced/removed key. +3. Remove old token memberships and insert new memberships on copied indexes. +4. Add every component touching an old or new token, including bridge additions. +5. Expand only that candidate region through current token buckets. +6. Repartition the candidate region iteratively and invoke `thread_messages` for + each new component in global insertion order. +7. Reuse untouched component roots and compose the global root list. Default output + uses first insertion position; RFC sent-date mode uses the same UTC/date, + sequence-number, and input-position ordering contract as the batch API. +8. Compare before/after projections and commit copied state only after all steps + succeed. + +Removing an internal or root message seeds its entire old component, so a split is +fully rediscovered. Adding a bridge message includes every touched old component, so +merges are explicit. Unrelated components are neither passed to `thread_messages` +nor traversed during the update. + +## External identity rules + +- Message keys never change; replacement uses the same key. +- A non-null `email_id` or `thread_id` cannot be removed or changed by replacement. +- All records sharing one non-null `email_id` must expose the same non-null + `thread_id`, matching RFC 8474. +- Structurally merged roots retain every caller thread ID and emit a merge group. +- Structural splits emit a split group rather than silently rewriting IDs. +- The core does not choose a canonical external ID. A server-specific policy may + consume the explicit transition later. + +## Snapshot contract + +Snapshot schema version `1` contains options, optimistic version, ordered records, +message metadata, and optional external IDs. It excludes payloads and derived graph +state. References are stored as normalized identifier lists. Datetimes are encoded +as tagged ISO 8601 strings; textual date inputs remain textual. + +Restore rejects unknown schema versions, duplicate or malformed keys, unexpected +fields, non-finite/boolean numeric metadata, oversized records/documents, and invalid +external-ID invariants. It rebuilds derived indexes through the same public +validation path and compares no untrusted serialized graph pointers. + +## Error model + +- `IncrementalThreadError`: invalid change or snapshot. +- `VersionConflictError`: optimistic version mismatch. +- `ExternalIdentityError`: immutable/cross-record EMAILID or THREADID violation. + +Messages are bounded and do not include payload representations, local paths, or +secrets. + +## Verification + +Required tests include: + +- additions, replacement, removal, delayed ancestor, bridge merge, and component + split against a complete batch rebuild; +- duplicate/missing Message-ID, raw headers, Unicode subjects, date sorting, UID + output, and RFC 5256 serialization parity; +- explicit merge/split external-ID transitions and RFC 8474 invariants; +- atomic rollback and optimistic version conflict; +- deterministic snapshot round trip, payload omission, malformed/oversized input, + and idempotency through version conflicts; +- a spy proving an unrelated component is not passed to `thread_messages` during a + bounded update; +- deep and hostile metadata cases without recursion; +- installed-wheel public API smoke coverage. + +A scheduled/manual benchmark follows in a separate bounded PR with at least 100,000 +messages, bounded deltas, wall time, peak RSS, affected-message count, and full +rebuild comparison. + +## References + +Gondwana, B. (2018). *IMAP extension for object identifiers* (RFC 8474). RFC +Editor. https://doi.org/10.17487/RFC8474 + +Jenkins, N., & Newman, C. (2019). *The JSON Meta Application Protocol (JMAP) for +mail* (RFC 8621). RFC Editor. https://doi.org/10.17487/RFC8621 + +Melnikov, A., & Cridland, D. (2014). *IMAP extensions: Quick flag changes +resynchronization (CONDSTORE) and quick mailbox resynchronization (QRESYNC)* +(RFC 7162). RFC Editor. https://doi.org/10.17487/RFC7162 + +Melnikov, A., & Leiba, B. (Eds.). (2021). *Internet Message Access Protocol +(IMAP)—Version 4rev2* (RFC 9051). RFC Editor. +https://doi.org/10.17487/RFC9051 From 21aa7ba542981192c67552055af9c58dd2432b95 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 08:49:55 +0900 Subject: [PATCH 02/97] docs: plan incremental thread index implementation --- .../2026-08-05-incremental-thread-index.md | 149 ++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-05-incremental-thread-index.md diff --git a/docs/superpowers/plans/2026-08-05-incremental-thread-index.md b/docs/superpowers/plans/2026-08-05-incremental-thread-index.md new file mode 100644 index 0000000..846adec --- /dev/null +++ b/docs/superpowers/plans/2026-08-05-incremental-thread-index.md @@ -0,0 +1,149 @@ +# Incremental Thread Index Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add an atomic, snapshot-capable incremental mailbox index that recomputes only affected reference/subject components while remaining exactly equivalent to the existing batch threader. + +**Architecture:** A new `threadweave.incremental` module owns caller keys, copied message metadata, reverse connectivity buckets, component-local batch results, projections, deltas, and snapshot validation. Existing batch/thread/container/IMAP modules remain transport-neutral and authoritative. The release-frozen `main` branch is not modified until release blocker #17 closes. + +**Tech Stack:** Python 3.10-3.13 standard library, existing `Message`, `Container`, `thread_messages`, RFC parsing/collation/date helpers, pytest, coverage, Ruff, Hatchling. + +## Global Constraints + +- Runtime dependencies remain empty. +- Production statement and branch coverage remain 100%. +- Every authored production module, class, function, method, and property has a beginner-readable docstring. +- Graph processing is iterative and identity-safe. +- Caller payloads never enter snapshots, delta equality, logs, or error messages. +- RFC 7162, RFC 8474, RFC 8621, RFC 9051, and existing RFC 5256 behavior remain traceable in APA 7th documentation. +- The PR remains draft and unmerged while issue #17 is open. + +--- + +### Task 1: Public change, projection, and error contracts + +**Files:** +- Create: `src/threadweave/incremental.py` +- Create: `tests/test_incremental_contract.py` +- Modify: `src/threadweave/__init__.py` +- Modify: `tests/test_documentation.py` + +**Interfaces:** +- Produces: `IndexedMessage`, `MailboxChangeSet`, `ThreadProjection`, `ThreadDelta`, `IncrementalThreadError`, `VersionConflictError`, `ExternalIdentityError`, `IncrementalThreadIndex`. + +- [ ] Write failing import, constructor-default, immutable-record, invalid-key, disjoint-change-set, and docstring tests. +- [ ] Run the focused tests and confirm missing symbols fail. +- [ ] Implement frozen public records, bounded key validation, exception hierarchy, and empty index properties. +- [ ] Export the new API and include the module in documentation inspection. +- [ ] Run focused tests and commit. + +### Task 2: Atomic record validation and copied metadata + +**Files:** +- Modify: `src/threadweave/incremental.py` +- Create: `tests/test_incremental_atomicity.py` + +**Interfaces:** +- Consumes: Task 1 public records. +- Produces: `IncrementalThreadIndex.apply(change_set) -> ThreadDelta` for record ownership and versioning before graph behavior. + +- [ ] Write failing tests for additions, replacement-in-place, removal, disjoint keys, missing/existing ownership errors, boolean/negative version values, and optimistic conflicts. +- [ ] Add tests proving payload identity is retained in memory while reference sequences are copied and later caller mutation cannot change indexed metadata. +- [ ] Run focused tests and confirm failures. +- [ ] Implement validation on copied state and commit only after every input passes. +- [ ] Verify failures leave version, records, roots, and projections unchanged. +- [ ] Run focused tests and commit. + +### Task 3: Connectivity indexes and bounded component recomputation + +**Files:** +- Modify: `src/threadweave/incremental.py` +- Create: `tests/test_incremental_components.py` + +**Interfaces:** +- Produces: token extraction, reverse buckets, component membership, component-local roots, affected-message reporting. + +- [ ] Write failing batch-parity tests for independent roots, linear references, shared missing roots, delayed ancestors, duplicate Message-IDs, replacement, root/internal/leaf removal, and bridge merges. +- [ ] Add a spy around the module-level batch delegate proving an unrelated component is not passed to the delegate during a one-component update. +- [ ] Run focused tests and confirm failures. +- [ ] Implement normalized ID/reference/subject tokens, copied reverse buckets, old-component seeding, touched-token expansion, iterative repartition, and component-local `thread_messages` calls. +- [ ] Compose roots in global insertion order and produce deterministic projections. +- [ ] Run focused tests and commit. + +### Task 4: Sent-date, subject, IMAP, and deep-tree parity + +**Files:** +- Modify: `src/threadweave/incremental.py` +- Create: `tests/test_incremental_parity.py` + +**Interfaces:** +- Consumes: Task 3 component state. +- Produces: exact output parity for both batch options and protocol serialization. + +- [ ] Write failing tests for RFC 5051 subject grouping, RFC 5256 sent-date ordering, explicit/implicit sequence collisions, Unicode subjects, raw headers, UID THREAD output, and one-shot source construction. +- [ ] Add deep-chain, split-tree, and malformed cycle-oriented metadata cases without recursive index traversal. +- [ ] Run focused tests and confirm failures. +- [ ] Implement global effective-sequence validation, root ordering from the first concrete node, and deterministic root composition. +- [ ] Compare projections and RFC 5256 serialization with complete batch rebuilds. +- [ ] Run focused tests and commit. + +### Task 5: RFC 8474 external identity transitions + +**Files:** +- Modify: `src/threadweave/incremental.py` +- Create: `tests/test_incremental_identity.py` + +**Interfaces:** +- Produces: immutable EMAILID/THREADID validation and explicit merge/split groups in `ThreadDelta`. + +- [ ] Write failing tests for replacement changes/removal of reported IDs, same EMAILID with inconsistent/missing THREADID, structurally merged distinct IDs, and structural splits. +- [ ] Run focused tests and confirm failures. +- [ ] Implement cross-record identity validation and deterministic overlap-based transition classification. +- [ ] Ensure no canonical thread ID is invented and sequence numbers never participate in identity. +- [ ] Run focused tests and commit. + +### Task 6: JSON-safe snapshot and restore + +**Files:** +- Modify: `src/threadweave/incremental.py` +- Create: `tests/test_incremental_snapshot.py` + +**Interfaces:** +- Produces: `snapshot()` and `IncrementalThreadIndex.restore(...)` schema version 1. + +- [ ] Write failing deterministic round-trip tests covering strings, aware/naive datetimes, references, options, external IDs, and payload omission. +- [ ] Write failing tests for unknown/missing/extra fields, duplicate keys, invalid types, noncanonical records, oversized record counts/bytes, and unsupported schema versions. +- [ ] Run focused tests and confirm failures. +- [ ] Implement tagged date encoding, strict field readers, JSON-size checks, restore through validated records, and derived-state rebuild. +- [ ] Run focused tests and commit. + +### Task 7: Buyer and research documentation, changelog, package smoke + +**Files:** +- Modify: `README.md` +- Modify: `CHANGELOG.md` +- Modify: `docs/research/README.md` +- Modify: `.github/workflows/ci.yml` +- Modify: `AGENTS.md` +- Modify: PR body + +**Interfaces:** +- Documents public behavior, complexity boundary, release freeze, and standards. + +- [ ] Add a complete incremental example with atomic changes, delta inspection, snapshot/restore, and RFC 8474 identity notes. +- [ ] Add APA 7th references and explicitly separate caller IDs from managed policy. +- [ ] Add `[Unreleased]` notes without changing version `0.2.0`. +- [ ] Extend the installed-wheel smoke test to construct and update an index. +- [ ] Add invariants to `AGENTS.md`. +- [ ] Run all repository checks and commit. + +### Task 8: Exact-head review and merge gating + +**Files:** +- No product files unless a review or check identifies a defect. + +- [ ] Run lock regeneration, Ruff, compileall, doctests, full pytest/coverage, autonomous/release-boundary coverage, build, hash-installed wheel smoke, SAST, and Security Scan. +- [ ] Confirm all production and trusted-boundary statements/branches are 100% covered. +- [ ] Review every unresolved thread and apply focused fixes. +- [ ] Keep the PR draft while issue #17 is open; do not enable auto-merge. +- [ ] After the verified `0.2.0` release closes #17, rebase/update the exact head, rerun all gates, request independent review, and merge only when policy is satisfied. From b0623b509b7782277107811754995679b5b6a0b5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 08:51:58 +0900 Subject: [PATCH 03/97] test: define incremental index public contract --- tests/test_incremental_contract.py | 234 +++++++++++++++++++++++++++++ 1 file changed, 234 insertions(+) create mode 100644 tests/test_incremental_contract.py diff --git a/tests/test_incremental_contract.py b/tests/test_incremental_contract.py new file mode 100644 index 0000000..dba9e71 --- /dev/null +++ b/tests/test_incremental_contract.py @@ -0,0 +1,234 @@ +"""Public and atomicity contracts for incremental mailbox threading.""" + +from __future__ import annotations + +from dataclasses import FrozenInstanceError + +import pytest + +from threadweave import ( + ExternalIdentityError, + IncrementalThreadError, + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + VersionConflictError, +) + + +def _record( + key: str, + *, + message_id: str | None = None, + references: tuple[str, ...] = (), + subject: str | None = None, + payload: object | None = None, + sequence_number: int | None = None, + uid: int | None = None, + email_id: str | None = None, + thread_id: str | None = None, +) -> IndexedMessage: + """Build one indexed message with explicit caller identity.""" + return IndexedMessage( + message_key=key, + message=Message( + message_id=message_id or key, + references=references, + subject=subject, + payload=key if payload is None else payload, + sequence_number=sequence_number, + uid=uid, + ), + email_id=email_id, + thread_id=thread_id, + ) + + +def test_empty_index_exposes_immutable_empty_state(): + """A new index starts at version zero with no roots or projections.""" + index = IncrementalThreadIndex() + + assert index.version == 0 + assert index.message_keys == () + assert index.roots == () + assert index.projections == () + assert len(index) == 0 + + +def test_public_records_are_frozen_and_sequences_are_normalized(): + """Change records cannot be mutated after validation or retain caller lists.""" + additions = [_record("a")] + changes = MailboxChangeSet(expected_version=0, additions=additions) + + assert changes.additions == tuple(additions) + assert changes.replacements == () + assert changes.removals == () + with pytest.raises(FrozenInstanceError): + changes.expected_version = 1 # type: ignore[misc] + + +def test_indexed_message_rejects_unsafe_keys_and_external_identifiers(): + """Caller identity values remain bounded, printable, and non-empty.""" + for key in ("", "bad\nkey", "x" * 513): + with pytest.raises(IncrementalThreadError, match="message_key"): + _record(key) + + with pytest.raises(IncrementalThreadError, match="email_id"): + _record("a", email_id="bad\x00id") + with pytest.raises(IncrementalThreadError, match="thread_id"): + _record("a", thread_id="") + + +def test_change_set_rejects_invalid_versions_duplicate_and_overlapping_keys(): + """An atomic request has one non-negative version and disjoint unique keys.""" + with pytest.raises(IncrementalThreadError, match="expected_version"): + MailboxChangeSet(expected_version=True) + with pytest.raises(IncrementalThreadError, match="expected_version"): + MailboxChangeSet(expected_version=-1) + with pytest.raises(IncrementalThreadError, match="duplicate.*additions"): + MailboxChangeSet(expected_version=0, additions=(_record("a"), _record("a"))) + with pytest.raises(IncrementalThreadError, match="disjoint"): + MailboxChangeSet( + expected_version=0, + additions=(_record("a"),), + removals=("a",), + ) + with pytest.raises(IncrementalThreadError, match="removals"): + MailboxChangeSet(expected_version=0, removals=("a", "a")) + + +def test_add_replace_remove_are_atomic_and_optimistically_versioned(): + """Successful changes advance once while rejected requests preserve state.""" + original_payload = object() + original = _record("a", subject="Original", payload=original_payload) + index = IncrementalThreadIndex() + + added = index.apply(MailboxChangeSet(expected_version=0, additions=(original,))) + assert (added.previous_version, added.version) == (0, 1) + assert added.affected_message_keys == ("a",) + assert index.message_keys == ("a",) + assert index.roots[0].message.payload is original_payload + + replacement = _record("a", subject="Replacement", payload=original_payload) + replaced = index.apply( + MailboxChangeSet(expected_version=1, replacements=(replacement,)) + ) + assert (replaced.previous_version, replaced.version) == (1, 2) + assert index.roots[0].message.subject == "Replacement" + + before = index.snapshot() + with pytest.raises(VersionConflictError, match="expected version 1.*current version 2"): + index.apply(MailboxChangeSet(expected_version=1, removals=("a",))) + assert index.snapshot() == before + + removed = index.apply(MailboxChangeSet(expected_version=2, removals=("a",))) + assert (removed.previous_version, removed.version) == (2, 3) + assert index.message_keys == () + assert index.roots == () + + +def test_invalid_ownership_requests_leave_the_index_unchanged(): + """Add/replace/remove ownership errors fail before committing copied state.""" + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=(_record("a"),))) + before = index.snapshot() + + invalid_changes = ( + MailboxChangeSet(expected_version=1, additions=(_record("a"),)), + MailboxChangeSet(expected_version=1, replacements=(_record("missing"),)), + MailboxChangeSet(expected_version=1, removals=("missing",)), + ) + for changes in invalid_changes: + with pytest.raises(IncrementalThreadError): + index.apply(changes) + assert index.snapshot() == before + + +def test_structural_metadata_is_copied_while_payload_remains_caller_owned(): + """Later caller mutation cannot rewrite indexed references or subject data.""" + references = ["root"] + message = Message( + message_id="child", + references=references, + subject="Original", + payload={"caller": "owned"}, + ) + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet( + expected_version=0, + additions=(IndexedMessage("child_key", message),), + ) + ) + + references.append("other") + message.subject = "Mutated" + + indexed = index.roots[0].message + assert indexed.references == ("root",) + assert indexed.subject == "Original" + assert indexed.payload is message.payload + + +def test_reported_external_identity_is_immutable_on_replacement(): + """RFC 8474 identity metadata cannot disappear or change after exposure.""" + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record("a", email_id="M1", thread_id="T1"), + ), + ) + ) + before = index.snapshot() + + for replacement in ( + _record("a", email_id=None, thread_id="T1"), + _record("a", email_id="M2", thread_id="T1"), + _record("a", email_id="M1", thread_id=None), + _record("a", email_id="M1", thread_id="T2"), + ): + with pytest.raises(ExternalIdentityError, match="immutable"): + index.apply( + MailboxChangeSet(expected_version=1, replacements=(replacement,)) + ) + assert index.snapshot() == before + + +def test_same_email_id_requires_one_consistent_thread_id(): + """Messages sharing immutable content identity cannot disagree on THREADID.""" + index = IncrementalThreadIndex() + + with pytest.raises(ExternalIdentityError, match="EMAILID.*THREADID"): + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record("a", email_id="M1", thread_id="T1"), + _record("b", email_id="M1", thread_id="T2"), + ), + ) + ) + assert index.version == 0 + + with pytest.raises(ExternalIdentityError, match="EMAILID.*THREADID"): + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record("a", email_id="M1", thread_id="T1"), + _record("b", email_id="M1", thread_id=None), + ), + ) + ) + + +def test_constructor_rejects_boolean_and_nonpositive_snapshot_limits(): + """Snapshot denial-of-service limits are explicit positive integers.""" + for keyword in ("max_snapshot_records", "max_snapshot_bytes"): + with pytest.raises(IncrementalThreadError, match=keyword): + IncrementalThreadIndex(**{keyword: True}) + with pytest.raises(IncrementalThreadError, match=keyword): + IncrementalThreadIndex(**{keyword: 0}) From 9d68967a23ad443d0cc3c8fb10697812b45a70e2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 08:54:01 +0900 Subject: [PATCH 04/97] test: define incremental component parity --- tests/test_incremental_components.py | 245 +++++++++++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 tests/test_incremental_components.py diff --git a/tests/test_incremental_components.py b/tests/test_incremental_components.py new file mode 100644 index 0000000..626a3f1 --- /dev/null +++ b/tests/test_incremental_components.py @@ -0,0 +1,245 @@ +"""Component and delta tests for incremental mailbox threading.""" + +from __future__ import annotations + +from collections.abc import Iterable + +import threadweave.incremental as incremental_module +from threadweave import ( + Container, + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + thread_messages, +) + + +def _record( + key: str, + *, + message_id: str | None = None, + references: tuple[str, ...] = (), + in_reply_to: tuple[str, ...] = (), + subject: str | None = None, + sent_date: str | None = None, + sequence_number: int | None = None, + uid: int | None = None, + email_id: str | None = None, + thread_id: str | None = None, +) -> IndexedMessage: + """Build one record whose payload exposes the caller key to tests.""" + return IndexedMessage( + message_key=key, + message=Message( + message_id=message_id if message_id is not None else key, + references=references, + in_reply_to=in_reply_to, + subject=subject, + payload=key, + sent_date=sent_date, + sequence_number=sequence_number, + uid=uid, + ), + email_id=email_id, + thread_id=thread_id, + ) + + +def _shape(roots: Iterable[Container]) -> tuple[object, ...]: + """Return one deterministic, iterative forest representation.""" + rendered: list[object] = [] + for root in roots: + output: list[object] = [] + stack: list[tuple[Container, bool]] = [(root, False)] + built: dict[int, object] = {} + while stack: + node, exiting = stack.pop() + if not exiting: + stack.append((node, True)) + for child in reversed(node.children): + stack.append((child, False)) + continue + children = tuple(built[id(child)] for child in node.children) + key = None if node.message is None else node.message.payload + built[id(node)] = (key, children) + output.append(built[id(root)]) + rendered.extend(output) + return tuple(rendered) + + +def _batch( + records: Iterable[IndexedMessage], + *, + group_by_subject: bool = False, + sort_by_sent_date: bool = False, +) -> tuple[object, ...]: + """Return the canonical batch shape for indexed records.""" + return _shape( + thread_messages( + (record.message for record in records), + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + ) + ) + + +def test_additions_match_batch_and_preserve_independent_root_order(): + """Initial component construction is exactly equivalent to one batch call.""" + records = ( + _record("a"), + _record("b", references=("a",)), + _record("x"), + _record("y", references=("x",)), + ) + index = IncrementalThreadIndex() + delta = index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + assert _shape(index.roots) == _batch(records) + assert index.message_keys == ("a", "b", "x", "y") + assert delta.affected_message_keys == ("a", "b", "x", "y") + assert tuple(projection.message_keys for projection in index.projections) == ( + ("a", "b"), + ("x", "y"), + ) + assert len(delta.added_threads) == 2 + assert delta.removed_threads == () + + +def test_delayed_missing_ancestor_recomputes_only_the_affected_component(): + """A late parent replaces a dummy without touching an unrelated thread.""" + child = _record("child", references=("root",)) + other = _record("other") + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet(expected_version=0, additions=(child, other)) + ) + + root = _record("root") + delta = index.apply( + MailboxChangeSet(expected_version=1, additions=(root,)) + ) + + assert _shape(index.roots) == _batch((child, other, root)) + assert delta.affected_message_keys == ("child", "root") + assert tuple(projection.message_keys for projection in index.projections) == ( + ("root", "child"), + ("other",), + ) + + +def test_bridge_message_emits_an_explicit_external_identity_merge(): + """Joining two exposed groups reports both caller THREADIDs.""" + first = _record("a", thread_id="T1") + second = _record("b", thread_id="T2") + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet(expected_version=0, additions=(first, second)) + ) + + bridge = _record("bridge", references=("a", "b"), thread_id="T3") + delta = index.apply( + MailboxChangeSet(expected_version=1, additions=(bridge,)) + ) + + assert _shape(index.roots) == _batch((first, second, bridge)) + assert len(delta.merges) == 1 + merge = delta.merges[0] + assert merge.kind == "merge" + assert merge.thread_ids == ("T1", "T2", "T3") + assert tuple(item.message_keys for item in merge.before) == (("a",), ("b",)) + assert tuple(item.message_keys for item in merge.after) == (("a", "b", "bridge"),) + + +def test_replacing_bridge_references_emits_a_split_and_matches_batch(): + """Removing a structural bridge rediscovers both resulting components.""" + first = _record("a", thread_id="T1") + second = _record("b", thread_id="T2") + bridge = _record("bridge", references=("a", "b"), thread_id="T3") + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet(expected_version=0, additions=(first, second, bridge)) + ) + + replacement = _record("bridge", references=("a",), thread_id="T3") + delta = index.apply( + MailboxChangeSet(expected_version=1, replacements=(replacement,)) + ) + + assert _shape(index.roots) == _batch((first, second, replacement)) + assert len(delta.splits) == 1 + split = delta.splits[0] + assert split.kind == "split" + assert split.thread_ids == ("T1", "T2", "T3") + assert tuple(item.message_keys for item in split.before) == ( + ("a", "b", "bridge"), + ) + assert {item.message_keys for item in split.after} == { + ("a", "bridge"), + ("b",), + } + + +def test_removing_root_internal_leaf_and_duplicate_id_matches_batch(): + """Every removal location follows canonical dummy pruning and promotion.""" + records = ( + _record("root"), + _record("middle", references=("root",)), + _record("leaf", references=("root", "middle")), + _record("duplicate", message_id="middle"), + _record("missing", message_id=None), + ) + + for removed_key in ("root", "middle", "leaf", "duplicate", "missing"): + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + index.apply( + MailboxChangeSet(expected_version=1, removals=(removed_key,)) + ) + remaining = tuple(record for record in records if record.message_key != removed_key) + assert _shape(index.roots) == _batch(remaining) + + +def test_unrelated_components_are_not_passed_to_the_batch_delegate(monkeypatch): + """A bounded update avoids rescanning a structurally unrelated component.""" + records = ( + _record("a"), + _record("b", references=("a",)), + _record("x"), + _record("y", references=("x",)), + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + calls: list[tuple[str, ...]] = [] + real_delegate = incremental_module._batch_thread_messages + + def recording_delegate(messages, **options): + """Record component payload keys before invoking the canonical batcher.""" + materialized = tuple(messages) + calls.append(tuple(message.payload for message in materialized)) + return real_delegate(materialized, **options) + + monkeypatch.setattr(incremental_module, "_batch_thread_messages", recording_delegate) + replacement = _record("b", references=("a",), subject="changed") + index.apply( + MailboxChangeSet(expected_version=1, replacements=(replacement,)) + ) + + assert calls == [("a", "b")] + assert all("x" not in call and "y" not in call for call in calls) + + +def test_noop_change_set_advances_no_version_and_returns_empty_delta(): + """An empty request is idempotent and does not fabricate a new revision.""" + index = IncrementalThreadIndex() + delta = index.apply(MailboxChangeSet(expected_version=0)) + + assert delta.previous_version == 0 + assert delta.version == 0 + assert delta.affected_message_keys == () + assert delta.added_threads == () + assert delta.removed_threads == () + assert delta.updated_threads == () + assert delta.merges == () + assert delta.splits == () From 238f271319fc76c6d4e7e6abf9fc26212ad71316 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 08:56:10 +0900 Subject: [PATCH 05/97] test: define incremental RFC parity --- tests/test_incremental_parity.py | 211 +++++++++++++++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 tests/test_incremental_parity.py diff --git a/tests/test_incremental_parity.py b/tests/test_incremental_parity.py new file mode 100644 index 0000000..6c87474 --- /dev/null +++ b/tests/test_incremental_parity.py @@ -0,0 +1,211 @@ +"""RFC, ordering, protocol, and depth parity for the incremental index.""" + +from __future__ import annotations + +import pytest + +from threadweave import ( + IncrementalThreadIndex, + IncrementalThreadError, + IndexedMessage, + MailboxChangeSet, + Message, + serialize_thread_response, + thread_messages, +) + + +def _record(key: str, **message_fields) -> IndexedMessage: + """Build one record whose payload remains its caller key.""" + return IndexedMessage( + message_key=key, + message=Message(payload=key, **message_fields), + ) + + +def _preorder(roots) -> tuple[tuple[str | None, ...], ...]: + """Return root-by-root preorder payload keys without recursion.""" + forest: list[tuple[str | None, ...]] = [] + for root in roots: + keys: list[str | None] = [] + stack = [root] + seen: set[int] = set() + while stack: + node = stack.pop() + if id(node) in seen: + continue + seen.add(id(node)) + keys.append(None if node.message is None else node.message.payload) + stack.extend(reversed(node.children)) + forest.append(tuple(keys)) + return tuple(forest) + + +def test_subject_grouping_uses_rfc_5051_and_replacement_updates_bucket(): + """Compatibility-width and reply variants merge and later split by replacement.""" + first = _record("a", message_id="a", subject="Ｔｏｐｉｃ") + second = _record("b", message_id="b", subject="Re: Topic") + index = IncrementalThreadIndex(group_by_subject=True) + index.apply(MailboxChangeSet(expected_version=0, additions=(first, second))) + + batch = thread_messages( + (first.message, second.message), + group_by_subject=True, + ) + assert _preorder(index.roots) == _preorder(batch) + assert len(index.roots) == 1 + + replacement = _record("b", message_id="b", subject="Different") + delta = index.apply( + MailboxChangeSet(expected_version=1, replacements=(replacement,)) + ) + batch_after = thread_messages( + (first.message, replacement.message), + group_by_subject=True, + ) + assert _preorder(index.roots) == _preorder(batch_after) + assert len(index.roots) == 2 + assert len(delta.splits) == 1 + + +def test_sent_date_order_matches_batch_across_unrelated_components(): + """Global root ordering uses RFC date recovery and sequence tie-breaking.""" + records = ( + _record( + "later", + message_id="later", + sent_date="2 Jan 2026 00:00:00 +0000", + sequence_number=2, + ), + _record( + "earlier", + message_id="earlier", + sent_date="1 Jan 2026 09:00:00 +0900", + sequence_number=1, + ), + _record( + "same_time", + message_id="same-time", + sent_date="1 Jan 2026 00:00:00 +0000", + sequence_number=3, + ), + ) + index = IncrementalThreadIndex(sort_by_sent_date=True) + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + batch = thread_messages( + (record.message for record in records), + sort_by_sent_date=True, + ) + assert _preorder(index.roots) == _preorder(batch) + assert tuple(projection.message_keys for projection in index.projections) == ( + ("earlier",), + ("same_time",), + ("later",), + ) + + +def test_global_sequence_number_collision_fails_atomically(): + """Separate components cannot hide duplicate RFC ordering tie-breakers.""" + index = IncrementalThreadIndex(sort_by_sent_date=True) + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record("a", message_id="a", sequence_number=1), + ), + ) + ) + before = index.snapshot() + + with pytest.raises(IncrementalThreadError, match="duplicate sequence number"): + index.apply( + MailboxChangeSet( + expected_version=1, + additions=( + _record("b", message_id="b", sequence_number=1), + ), + ) + ) + assert index.snapshot() == before + + +def test_raw_reference_headers_and_uid_thread_output_match_batch(): + """The index preserves raw RFC input and feeds the installed IMAP projector.""" + records = ( + _record( + "root", + message_id="", + sequence_number=1, + uid=101, + ), + _record( + "child", + message_id="", + references="", + in_reply_to=" ", + sequence_number=2, + uid=102, + ), + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + batch = thread_messages(record.message for record in records) + assert _preorder(index.roots) == _preorder(batch) + assert serialize_thread_response(index.roots) == "* THREAD (1 2)\r\n" + assert serialize_thread_response(index.roots, identifier="uid") == ( + "* THREAD (101 102)\r\n" + ) + + +def test_deep_delayed_ancestry_update_remains_iterative(): + """A deep component can be indexed and extended without recursion limits.""" + depth = 1500 + records = tuple( + _record( + f"message_{index}", + message_id=f"message_{index}", + references=() if index == 0 else (f"message_{index - 1}",), + ) + for index in range(depth) + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + tail = _record( + "tail", + message_id="tail", + references=(f"message_{depth - 1}",), + ) + delta = index.apply( + MailboxChangeSet(expected_version=1, additions=(tail,)) + ) + + assert len(index.projections) == 1 + assert len(index.projections[0].message_keys) == depth + 1 + assert index.projections[0].message_keys[-1] == "tail" + assert len(delta.affected_message_keys) == depth + 1 + + +def test_implicit_sequence_positions_remain_stable_after_replacement(): + """Replacing metadata does not move a caller key to the end of input order.""" + first = _record("a", message_id="a", sent_date="1 Jan 2026 00:00:00 +0000") + second = _record("b", message_id="b", sent_date="1 Jan 2026 00:00:00 +0000") + index = IncrementalThreadIndex(sort_by_sent_date=True) + index.apply(MailboxChangeSet(expected_version=0, additions=(first, second))) + + replacement = _record( + "a", + message_id="a", + sent_date="1 Jan 2026 00:00:00 +0000", + subject="updated", + ) + index.apply( + MailboxChangeSet(expected_version=1, replacements=(replacement,)) + ) + + assert tuple(projection.message_keys for projection in index.projections) == ( + ("a",), + ("b",), + ) From a1d2b9188d0d9862079e4c97e05543ed082aaad2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 08:57:45 +0900 Subject: [PATCH 06/97] test: define incremental snapshot contract --- tests/test_incremental_snapshot.py | 232 +++++++++++++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 tests/test_incremental_snapshot.py diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py new file mode 100644 index 0000000..a511ea9 --- /dev/null +++ b/tests/test_incremental_snapshot.py @@ -0,0 +1,232 @@ +"""Versioned JSON-safe snapshot tests for the incremental index.""" + +from __future__ import annotations + +import json +from copy import deepcopy +from datetime import datetime, timezone + +import pytest + +from threadweave import ( + IncrementalThreadError, + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, +) + + +def _index() -> IncrementalThreadIndex: + """Return a representative index containing every serializable field.""" + index = IncrementalThreadIndex( + group_by_subject=True, + sort_by_sent_date=True, + max_snapshot_records=100, + max_snapshot_bytes=100_000, + ) + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + IndexedMessage( + message_key="root_key", + message=Message( + message_id="", + subject="Topic", + payload={"must": "not persist"}, + sent_date=datetime(2026, 1, 1, tzinfo=timezone.utc), + internal_date="1 Jan 2026 00:00:00 +0000", + sequence_number=1, + uid=101, + ), + email_id="M1", + thread_id="T1", + ), + IndexedMessage( + message_key="child_key", + message=Message( + message_id="", + references="", + in_reply_to="", + subject="Re: Topic", + payload=object(), + sent_date=datetime(2026, 1, 2), + sequence_number=2, + uid=102, + ), + email_id="M2", + thread_id="T1", + ), + ), + ) + ) + return index + + +def test_snapshot_is_deterministic_json_safe_and_omits_payloads(): + """State serializes reproducibly without arbitrary caller objects.""" + index = _index() + + first = index.snapshot() + second = index.snapshot() + + assert first == second + encoded = json.dumps(first, sort_keys=True, separators=(",", ":")) + assert "must not persist" not in encoded + assert "payload" not in encoded + assert first["schema_version"] == 1 + assert first["version"] == 1 + assert first["options"] == { + "group_by_subject": True, + "sort_by_sent_date": True, + } + assert [record["message_key"] for record in first["records"]] == [ + "root_key", + "child_key", + ] + + +def test_snapshot_restore_round_trip_preserves_structure_and_version(): + """Restored derived state matches the source while payloads become None.""" + source = _index() + snapshot = source.snapshot() + + restored = IncrementalThreadIndex.restore( + snapshot, + max_snapshot_records=100, + max_snapshot_bytes=100_000, + ) + + assert restored.version == source.version + assert restored.message_keys == source.message_keys + assert restored.projections == source.projections + assert restored.snapshot() == snapshot + assert all( + node.message.payload is None + for root in restored.roots + for node in (root, *tuple(root.iter_descendants())) + if node.message is not None + ) + + +def test_restored_index_continues_with_optimistic_versioning(): + """A restored revision accepts the next atomic change exactly once.""" + restored = IncrementalThreadIndex.restore(_index().snapshot()) + delta = restored.apply( + MailboxChangeSet( + expected_version=1, + additions=( + IndexedMessage( + "new_key", + Message(message_id="new", payload="not persisted"), + ), + ), + ) + ) + + assert (delta.previous_version, delta.version) == (1, 2) + assert restored.version == 2 + + +def test_restore_rejects_unknown_root_fields_and_schema_versions(): + """Untrusted snapshots use an exact schema rather than permissive decoding.""" + snapshot = _index().snapshot() + + unknown = deepcopy(snapshot) + unknown["unexpected"] = True + with pytest.raises(IncrementalThreadError, match="snapshot fields"): + IncrementalThreadIndex.restore(unknown) + + unsupported = deepcopy(snapshot) + unsupported["schema_version"] = 2 + with pytest.raises(IncrementalThreadError, match="schema_version"): + IncrementalThreadIndex.restore(unsupported) + + missing = deepcopy(snapshot) + del missing["records"] + with pytest.raises(IncrementalThreadError, match="snapshot fields"): + IncrementalThreadIndex.restore(missing) + + +def test_restore_rejects_malformed_record_fields_and_duplicate_keys(): + """Record decoding rejects ambiguity before constructing any graph state.""" + snapshot = _index().snapshot() + + duplicate = deepcopy(snapshot) + duplicate["records"].append(deepcopy(duplicate["records"][0])) + with pytest.raises(IncrementalThreadError, match="duplicate.*message_key"): + IncrementalThreadIndex.restore(duplicate) + + extra = deepcopy(snapshot) + extra["records"][0]["unexpected"] = "value" + with pytest.raises(IncrementalThreadError, match="record fields"): + IncrementalThreadIndex.restore(extra) + + invalid_records = deepcopy(snapshot) + invalid_records["records"] = "not-a-list" + with pytest.raises(IncrementalThreadError, match="records"): + IncrementalThreadIndex.restore(invalid_records) + + invalid_message = deepcopy(snapshot) + invalid_message["records"][0]["message"] = [] + with pytest.raises(IncrementalThreadError, match="message"): + IncrementalThreadIndex.restore(invalid_message) + + +def test_restore_rejects_invalid_date_tags_and_numeric_metadata(): + """Date and IMAP metadata preserve strict tagged and integer contracts.""" + snapshot = _index().snapshot() + + invalid_date = deepcopy(snapshot) + invalid_date["records"][0]["message"]["sent_date"] = { + "kind": "unknown", + "value": "2026-01-01", + } + with pytest.raises(IncrementalThreadError, match="date"): + IncrementalThreadIndex.restore(invalid_date) + + invalid_datetime = deepcopy(snapshot) + invalid_datetime["records"][0]["message"]["sent_date"] = { + "kind": "datetime", + "value": "not-a-date", + } + with pytest.raises(IncrementalThreadError, match="datetime"): + IncrementalThreadIndex.restore(invalid_datetime) + + invalid_sequence = deepcopy(snapshot) + invalid_sequence["records"][0]["message"]["sequence_number"] = True + with pytest.raises(IncrementalThreadError, match="sequence_number"): + IncrementalThreadIndex.restore(invalid_sequence) + + +def test_restore_and_snapshot_enforce_record_and_byte_limits(): + """Configured bounds stop oversized state before allocation or publication.""" + snapshot = _index().snapshot() + + with pytest.raises(IncrementalThreadError, match="max_snapshot_records"): + IncrementalThreadIndex.restore(snapshot, max_snapshot_records=1) + with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): + IncrementalThreadIndex.restore(snapshot, max_snapshot_bytes=10) + + index = IncrementalThreadIndex(max_snapshot_bytes=10) + index.apply( + MailboxChangeSet( + expected_version=0, + additions=(IndexedMessage("a", Message(message_id="a")),), + ) + ) + with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): + index.snapshot() + + +def test_restore_rejects_non_mapping_and_non_json_safe_values(): + """The public boundary converts encoder failures into domain errors.""" + with pytest.raises(IncrementalThreadError, match="mapping"): + IncrementalThreadIndex.restore([]) # type: ignore[arg-type] + + snapshot = _index().snapshot() + invalid = deepcopy(snapshot) + invalid["records"][0]["email_id"] = object() + with pytest.raises(IncrementalThreadError, match="JSON"): + IncrementalThreadIndex.restore(invalid) From 29a639a6d18835ad5af8f188d97ea84a69f98873 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:25:46 +0900 Subject: [PATCH 07/97] feat: implement atomic incremental thread index --- src/threadweave/incremental.py | 1219 ++++++++++++++++++++++++++++++++ 1 file changed, 1219 insertions(+) create mode 100644 src/threadweave/incremental.py diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py new file mode 100644 index 0000000..683fd1a --- /dev/null +++ b/src/threadweave/incremental.py @@ -0,0 +1,1219 @@ +"""Incremental, identity-aware mailbox threading over the batch RFC engine. + +The batch :func:`threadweave.thread_messages` function remains the correctness +oracle. This module adds an atomic state boundary that indexes caller-owned +message keys, recomputes only affected connectivity components, reports explicit +thread merge/split transitions, and snapshots JSON-safe metadata without caller +payloads. +""" + +from __future__ import annotations + +import json +from collections.abc import Iterable, Mapping, Sequence +from dataclasses import dataclass +from datetime import datetime +from typing import Literal + +from threadweave.collation import unicode_casemap_key +from threadweave.container import Container +from threadweave.dates import normalize_sent_date +from threadweave.headers import extract_reference_ids, normalize_message_id +from threadweave.subject import normalize_subject +from threadweave.threading import Message, thread_messages + +__all__ = [ + "ExternalIdentityError", + "IncrementalThreadError", + "IncrementalThreadIndex", + "IndexedMessage", + "MailboxChangeSet", + "ThreadDelta", + "ThreadProjection", + "ThreadTransition", + "VersionConflictError", +] + +_batch_thread_messages = thread_messages +_MAX_MESSAGE_KEY_LENGTH = 512 +_MAX_EXTERNAL_ID_LENGTH = 1024 +_MAX_IMAP_NUMBER = 4_294_967_295 +_DEFAULT_MAX_SNAPSHOT_RECORDS = 1_000_000 +_DEFAULT_MAX_SNAPSHOT_BYTES = 256 * 1024 * 1024 +_SNAPSHOT_SCHEMA_VERSION = 1 + + +class IncrementalThreadError(ValueError): + """Raised when an incremental change or snapshot violates its contract.""" + + +class VersionConflictError(IncrementalThreadError): + """Raised when an optimistic change targets a stale index version.""" + + +class ExternalIdentityError(IncrementalThreadError): + """Raised when caller-owned EMAILID or THREADID metadata is inconsistent.""" + + +def _validated_positive_limit(value: object, name: str) -> int: + """Return a positive non-boolean integer denial-of-service limit.""" + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise IncrementalThreadError(f"{name} must be a positive integer") + return value + + +def _validated_nonnegative_integer(value: object, name: str) -> int: + """Return a non-negative non-boolean integer revision value.""" + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise IncrementalThreadError(f"{name} must be a non-negative integer") + return value + + +def _validated_identifier( + value: object, + name: str, + *, + allow_none: bool, + maximum_length: int, +) -> str | None: + """Validate one bounded printable caller-owned identifier.""" + if value is None and allow_none: + return None + if not isinstance(value, str) or not value or len(value) > maximum_length: + raise IncrementalThreadError( + f"{name} must be a non-empty string of at most {maximum_length} characters" + ) + if any(ord(character) < 32 or ord(character) == 127 for character in value): + raise IncrementalThreadError(f"{name} must not contain control characters") + return value + + +def _validated_message_key(value: object) -> str: + """Return one safe immutable caller message key.""" + validated = _validated_identifier( + value, + "message_key", + allow_none=False, + maximum_length=_MAX_MESSAGE_KEY_LENGTH, + ) + assert validated is not None + return validated + + +def _validated_external_id(value: object, name: str) -> str | None: + """Return one optional bounded external object identifier.""" + return _validated_identifier( + value, + name, + allow_none=True, + maximum_length=_MAX_EXTERNAL_ID_LENGTH, + ) + + +def _tuple_without_duplicate_keys( + values: Iterable[object], + name: str, + *, + indexed: bool, +) -> tuple[object, ...]: + """Materialize one request sequence and reject duplicate caller keys.""" + materialized = tuple(values) + keys: list[str] = [] + for value in materialized: + if indexed: + if not isinstance(value, IndexedMessage): + raise IncrementalThreadError(f"{name} must contain IndexedMessage values") + key = value.message_key + else: + key = _validated_message_key(value) + keys.append(key) + if len(set(keys)) != len(keys): + raise IncrementalThreadError(f"duplicate message keys in {name}") + return materialized + + +@dataclass(frozen=True, slots=True) +class IndexedMessage: + """One message plus immutable caller and optional external identities. + + Args: + message_key: Caller-owned key that is stable across mailbox revisions. + message: Structural email metadata and an arbitrary in-memory payload. + email_id: Optional RFC 8474 immutable message-content identifier. + thread_id: Optional RFC 8474 caller/server thread correlator. + """ + + message_key: str + message: Message + email_id: str | None = None + thread_id: str | None = None + + def __post_init__(self) -> None: + """Validate public identity values without copying message metadata yet.""" + object.__setattr__(self, "message_key", _validated_message_key(self.message_key)) + if not isinstance(self.message, Message): + raise IncrementalThreadError("message must be a threadweave.Message") + object.__setattr__( + self, + "email_id", + _validated_external_id(self.email_id, "email_id"), + ) + object.__setattr__( + self, + "thread_id", + _validated_external_id(self.thread_id, "thread_id"), + ) + + +@dataclass(frozen=True, slots=True) +class MailboxChangeSet: + """One optimistic and atomic mailbox mutation request.""" + + expected_version: int + additions: tuple[IndexedMessage, ...] = () + replacements: tuple[IndexedMessage, ...] = () + removals: tuple[str, ...] = () + + def __post_init__(self) -> None: + """Normalize request sequences and require disjoint unique key sets.""" + object.__setattr__( + self, + "expected_version", + _validated_nonnegative_integer(self.expected_version, "expected_version"), + ) + additions = _tuple_without_duplicate_keys( + self.additions, + "additions", + indexed=True, + ) + replacements = _tuple_without_duplicate_keys( + self.replacements, + "replacements", + indexed=True, + ) + removals = _tuple_without_duplicate_keys( + self.removals, + "removals", + indexed=False, + ) + addition_keys = {item.message_key for item in additions} + replacement_keys = {item.message_key for item in replacements} + removal_keys = set(removals) + if ( + addition_keys & replacement_keys + or addition_keys & removal_keys + or replacement_keys & removal_keys + ): + raise IncrementalThreadError( + "addition, replacement, and removal message keys must be disjoint" + ) + object.__setattr__(self, "additions", additions) + object.__setattr__(self, "replacements", replacements) + object.__setattr__(self, "removals", removals) + + +@dataclass(frozen=True, slots=True) +class ThreadProjection: + """A deterministic caller-key view of one returned thread root.""" + + message_keys: tuple[str, ...] + thread_ids: tuple[str, ...] = () + + +@dataclass(frozen=True, slots=True) +class ThreadTransition: + """One explicit structural merge or split across caller thread IDs.""" + + kind: Literal["merge", "split"] + before: tuple[ThreadProjection, ...] + after: tuple[ThreadProjection, ...] + thread_ids: tuple[str, ...] + + +@dataclass(frozen=True, slots=True) +class ThreadDelta: + """Deterministic differences produced by one successful atomic change.""" + + previous_version: int + version: int + affected_message_keys: tuple[str, ...] + added_threads: tuple[ThreadProjection, ...] + removed_threads: tuple[ThreadProjection, ...] + updated_threads: tuple[ThreadProjection, ...] + merges: tuple[ThreadTransition, ...] + splits: tuple[ThreadTransition, ...] + + +def _validated_optional_number( + value: object, + name: str, + *, + maximum: int | None = None, +) -> int | None: + """Validate optional positive integer mailbox metadata.""" + if value is None: + return None + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise IncrementalThreadError(f"{name} must be a positive integer or None") + if maximum is not None and value > maximum: + raise IncrementalThreadError(f"{name} exceeds the unsigned 32-bit range") + return value + + +def _validated_optional_text(value: object, name: str) -> str | None: + """Validate an optional textual email field.""" + if value is None or isinstance(value, str): + return value + raise IncrementalThreadError(f"{name} must be a string or None") + + +def _copied_reference_value( + value: str | Sequence[str] | None, + name: str, +) -> str | tuple[str, ...] | None: + """Copy one raw or already-split identification header value.""" + if value is None or isinstance(value, str): + return value + try: + copied = tuple(value) + except TypeError as error: + raise IncrementalThreadError( + f"{name} must be a string, sequence of strings, or None" + ) from error + if not all(isinstance(item, str) for item in copied): + raise IncrementalThreadError(f"{name} must contain only strings") + return copied + + +def _copied_date_value(value: object, name: str) -> str | datetime | None: + """Copy one supported date value after validating its runtime type.""" + if value is None or isinstance(value, (str, datetime)): + return value + raise IncrementalThreadError(f"{name} must be a datetime, string, or None") + + +def _copied_indexed_message(record: IndexedMessage) -> IndexedMessage: + """Copy structural metadata while retaining the caller payload by reference.""" + message = record.message + copied = Message( + message_id=_validated_optional_text(message.message_id, "message_id"), + in_reply_to=_copied_reference_value(message.in_reply_to, "in_reply_to"), + references=_copied_reference_value(message.references, "references") or (), + subject=_validated_optional_text(message.subject, "subject"), + payload=message.payload, + sent_date=_copied_date_value(message.sent_date, "sent_date"), + internal_date=_copied_date_value(message.internal_date, "internal_date"), + sequence_number=_validated_optional_number( + message.sequence_number, + "sequence_number", + ), + uid=_validated_optional_number(message.uid, "uid", maximum=_MAX_IMAP_NUMBER), + ) + return IndexedMessage( + message_key=record.message_key, + message=copied, + email_id=record.email_id, + thread_id=record.thread_id, + ) + + +def _reference_ids(value: str | Sequence[str] | None) -> tuple[str, ...]: + """Parse and de-duplicate one raw or split identification field.""" + if value is None: + return () + values = (value,) if isinstance(value, str) else value + result: list[str] = [] + seen: set[str] = set() + for raw_value in values: + for identifier in extract_reference_ids(raw_value): + if identifier not in seen: + seen.add(identifier) + result.append(identifier) + return tuple(result) + + +def _effective_reference_ids(message: Message) -> tuple[str, ...]: + """Return the RFC 5256 ancestry identifiers used by the batch algorithm.""" + references = _reference_ids(message.references) + if references: + return references + return _reference_ids(message.in_reply_to)[:1] + + +def _connectivity_tokens( + record: IndexedMessage, + *, + group_by_subject: bool, +) -> frozenset[str]: + """Return an over-approximating connectivity token set for one record.""" + tokens: set[str] = set() + message_id = normalize_message_id(record.message.message_id) + if message_id is not None: + tokens.add(f"id\x00{message_id}") + for reference_id in _effective_reference_ids(record.message): + tokens.add(f"id\x00{reference_id}") + if group_by_subject: + subject_key = unicode_casemap_key(normalize_subject(record.message.subject)) + if subject_key: + tokens.add(f"subject\x00{subject_key}") + return frozenset(tokens) + + +def _copy_token_buckets( + buckets: Mapping[str, set[str]], +) -> dict[str, set[str]]: + """Return independent mutable copies of reverse token buckets.""" + return {token: set(keys) for token, keys in buckets.items()} + + +def _remove_key_from_buckets( + key: str, + tokens_by_key: dict[str, frozenset[str]], + keys_by_token: dict[str, set[str]], +) -> frozenset[str]: + """Remove one key from copied token indexes and return its old tokens.""" + old_tokens = tokens_by_key.pop(key, frozenset()) + for token in old_tokens: + bucket = keys_by_token[token] + bucket.discard(key) + if not bucket: + del keys_by_token[token] + return old_tokens + + +def _add_key_to_buckets( + key: str, + tokens: frozenset[str], + tokens_by_key: dict[str, frozenset[str]], + keys_by_token: dict[str, set[str]], +) -> None: + """Insert one key into copied forward and reverse token indexes.""" + tokens_by_key[key] = tokens + for token in tokens: + keys_by_token.setdefault(token, set()).add(key) + + +def _ordered_keys(keys: Iterable[str], positions: Mapping[str, int]) -> tuple[str, ...]: + """Return keys in stable insertion order with a lexical safety tie-breaker.""" + return tuple(sorted(keys, key=lambda key: (positions[key], key))) + + +def _current_ranks(positions: Mapping[str, int]) -> dict[str, int]: + """Return compact one-based input positions for the current record order.""" + return { + key: rank + for rank, key in enumerate(_ordered_keys(positions, positions), start=1) + } + + +def _validate_effective_sequence_numbers( + records: Mapping[str, IndexedMessage], + ranks: Mapping[str, int], +) -> None: + """Reject global explicit/implicit sequence collisions before recomputation.""" + used: dict[int, str] = {} + for key in _ordered_keys(records, ranks): + explicit = records[key].message.sequence_number + sequence_number = ranks[key] if explicit is None else explicit + previous = used.get(sequence_number) + if previous is not None: + raise IncrementalThreadError( + f"duplicate sequence number: {sequence_number} ({previous}, {key})" + ) + used[sequence_number] = key + + +def _validate_external_identities(records: Mapping[str, IndexedMessage]) -> None: + """Require every shared EMAILID to carry one identical THREADID value.""" + thread_id_by_email_id: dict[str, str | None] = {} + for record in records.values(): + email_id = record.email_id + if email_id is None: + continue + if email_id not in thread_id_by_email_id: + thread_id_by_email_id[email_id] = record.thread_id + continue + if thread_id_by_email_id[email_id] != record.thread_id: + raise ExternalIdentityError( + f"messages with EMAILID {email_id!r} must expose the same THREADID" + ) + + +def _validate_replacement_identity( + old_record: IndexedMessage, + new_record: IndexedMessage, +) -> None: + """Prevent removal or change of an already reported external identifier.""" + if old_record.email_id is not None and new_record.email_id != old_record.email_id: + raise ExternalIdentityError("reported email_id is immutable on replacement") + if old_record.thread_id is not None and new_record.thread_id != old_record.thread_id: + raise ExternalIdentityError("reported thread_id is immutable on replacement") + + +def _expand_candidate_keys( + seeds: set[str], + tokens_by_key: Mapping[str, frozenset[str]], + keys_by_token: Mapping[str, set[str]], +) -> set[str]: + """Expand candidate keys through current token connectivity iteratively.""" + expanded = {key for key in seeds if key in tokens_by_key} + queue = list(expanded) + while queue: + key = queue.pop() + for token in tokens_by_key.get(key, frozenset()): + for neighbor in keys_by_token.get(token, set()): + if neighbor not in expanded: + expanded.add(neighbor) + queue.append(neighbor) + return expanded + + +def _partition_components( + keys: set[str], + positions: Mapping[str, int], + tokens_by_key: Mapping[str, frozenset[str]], + keys_by_token: Mapping[str, set[str]], +) -> tuple[tuple[str, ...], ...]: + """Partition candidate keys into deterministic current connectivity components.""" + remaining = set(keys) + components: list[tuple[str, ...]] = [] + while remaining: + seed = min(remaining, key=lambda key: (positions[key], key)) + component = {seed} + queue = [seed] + remaining.remove(seed) + while queue: + key = queue.pop() + for token in tokens_by_key.get(key, frozenset()): + for neighbor in keys_by_token.get(token, set()): + if neighbor in remaining: + remaining.remove(neighbor) + component.add(neighbor) + queue.append(neighbor) + components.append(_ordered_keys(component, positions)) + return tuple(components) + + +def _message_for_batch(message: Message, sequence_number: int | None) -> Message: + """Return a computation copy with an explicit global sequence when required.""" + if sequence_number is None or message.sequence_number is not None: + return message + return Message( + message_id=message.message_id, + in_reply_to=message.in_reply_to, + references=message.references, + subject=message.subject, + payload=message.payload, + sent_date=message.sent_date, + internal_date=message.internal_date, + sequence_number=sequence_number, + uid=message.uid, + ) + + +def _projection_for_root( + root: Container, + key_by_message_identity: Mapping[int, str], + records: Mapping[str, IndexedMessage], +) -> ThreadProjection: + """Project one root into traversal-ordered caller keys and external IDs.""" + message_keys: list[str] = [] + seen: set[int] = set() + stack = [root] + while stack: + node = stack.pop() + if id(node) in seen: + continue + seen.add(id(node)) + if node.message is not None: + key = key_by_message_identity.get(id(node.message)) + if key is None: + raise IncrementalThreadError( + "batch output contained a message outside its component" + ) + message_keys.append(key) + stack.extend(reversed(node.children)) + thread_ids = tuple( + sorted( + { + records[key].thread_id + for key in message_keys + if records[key].thread_id is not None + } + ) + ) + return ThreadProjection(tuple(message_keys), thread_ids) + + +def _build_component( + keys: tuple[str, ...], + records: Mapping[str, IndexedMessage], + ranks: Mapping[str, int], + *, + group_by_subject: bool, + sort_by_sent_date: bool, +) -> tuple[tuple[Container, ...], tuple[ThreadProjection, ...]]: + """Run the canonical batch engine for one affected component.""" + messages: list[Message] = [] + key_by_message_identity: dict[int, str] = {} + for key in keys: + message = _message_for_batch( + records[key].message, + ranks[key] if sort_by_sent_date else None, + ) + messages.append(message) + key_by_message_identity[id(message)] = key + roots = tuple( + _batch_thread_messages( + messages, + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + ) + ) + projections = tuple( + _projection_for_root(root, key_by_message_identity, records) for root in roots + ) + return roots, projections + + +def _root_order_key( + root: Container, + projection: ThreadProjection, + records: Mapping[str, IndexedMessage], + ranks: Mapping[str, int], + *, + sort_by_sent_date: bool, +) -> tuple[object, ...]: + """Return the same global root-order contract used by the batch algorithm.""" + if not sort_by_sent_date: + return (min(ranks[key] for key in projection.message_keys),) + if not projection.message_keys: + return (normalize_sent_date(None), 0, 0) + first_key = projection.message_keys[0] + message = records[first_key].message + sequence_number = ( + ranks[first_key] if message.sequence_number is None else message.sequence_number + ) + return ( + normalize_sent_date(message.sent_date, message.internal_date), + sequence_number, + ranks[first_key], + ) + + +def _compose_forest( + roots_by_component: Mapping[str, tuple[Container, ...]], + projections_by_component: Mapping[str, tuple[ThreadProjection, ...]], + records: Mapping[str, IndexedMessage], + ranks: Mapping[str, int], + *, + sort_by_sent_date: bool, +) -> tuple[tuple[Container, ...], tuple[ThreadProjection, ...]]: + """Compose reusable component outputs into one deterministic global forest.""" + entries: list[tuple[tuple[object, ...], Container, ThreadProjection]] = [] + for component_id, roots in roots_by_component.items(): + projections = projections_by_component[component_id] + for root, projection in zip(roots, projections): + entries.append( + ( + _root_order_key( + root, + projection, + records, + ranks, + sort_by_sent_date=sort_by_sent_date, + ), + root, + projection, + ) + ) + entries.sort(key=lambda entry: entry[0]) + return ( + tuple(entry[1] for entry in entries), + tuple(entry[2] for entry in entries), + ) + + +def _projection_overlap(first: ThreadProjection, second: ThreadProjection) -> bool: + """Return whether two projections share any immutable caller message key.""" + return bool(set(first.message_keys) & set(second.message_keys)) + + +def _transition_thread_ids( + before: Iterable[ThreadProjection], + after: Iterable[ThreadProjection], +) -> tuple[str, ...]: + """Return every distinct caller THREADID represented by a transition.""" + return tuple( + sorted( + { + thread_id + for projection in (*tuple(before), *tuple(after)) + for thread_id in projection.thread_ids + } + ) + ) + + +def _thread_delta( + previous_version: int, + version: int, + affected_message_keys: tuple[str, ...], + before: tuple[ThreadProjection, ...], + after: tuple[ThreadProjection, ...], +) -> ThreadDelta: + """Classify deterministic projection additions, removals, updates, and transitions.""" + added = tuple( + projection + for projection in after + if not any(_projection_overlap(projection, old) for old in before) + ) + removed = tuple( + projection + for projection in before + if not any(_projection_overlap(projection, new) for new in after) + ) + updated = tuple( + projection + for projection in after + if projection not in before + and any(_projection_overlap(projection, old) for old in before) + ) + merges: list[ThreadTransition] = [] + for projection in after: + overlapping = tuple(old for old in before if _projection_overlap(projection, old)) + if len(overlapping) > 1: + merges.append( + ThreadTransition( + "merge", + overlapping, + (projection,), + _transition_thread_ids(overlapping, (projection,)), + ) + ) + splits: list[ThreadTransition] = [] + for projection in before: + overlapping = tuple(new for new in after if _projection_overlap(projection, new)) + if len(overlapping) > 1: + splits.append( + ThreadTransition( + "split", + (projection,), + overlapping, + _transition_thread_ids((projection,), overlapping), + ) + ) + return ThreadDelta( + previous_version, + version, + affected_message_keys, + added, + removed, + updated, + tuple(merges), + tuple(splits), + ) + + +def _encoded_date(value: str | datetime | None) -> object: + """Encode one date value into a deterministic JSON-safe tagged form.""" + if value is None: + return None + if isinstance(value, datetime): + return {"kind": "datetime", "value": value.isoformat()} + return {"kind": "text", "value": value} + + +def _decoded_date(value: object, name: str) -> str | datetime | None: + """Decode one strict tagged date value from an untrusted snapshot.""" + if value is None: + return None + if not isinstance(value, Mapping) or set(value) != {"kind", "value"}: + raise IncrementalThreadError(f"{name} date value is malformed") + kind = value["kind"] + encoded = value["value"] + if not isinstance(encoded, str): + raise IncrementalThreadError(f"{name} date value must be textual") + if kind == "text": + return encoded + if kind == "datetime": + try: + return datetime.fromisoformat(encoded) + except ValueError as error: + raise IncrementalThreadError(f"{name} datetime is invalid") from error + raise IncrementalThreadError(f"{name} date kind is unsupported") + + +def _snapshot_json_bytes(value: object) -> bytes: + """Serialize a snapshot canonically or raise a bounded domain error.""" + try: + encoded = json.dumps( + value, + ensure_ascii=False, + allow_nan=False, + sort_keys=True, + separators=(",", ":"), + ) + except (TypeError, ValueError) as error: + raise IncrementalThreadError( + "snapshot must contain only JSON-safe values" + ) from error + return encoded.encode("utf-8") + + +def _required_fields(value: Mapping[str, object], expected: set[str], name: str) -> None: + """Require an exact untrusted mapping field set.""" + if set(value) != expected: + raise IncrementalThreadError(f"{name} fields do not match the schema") + + +class IncrementalThreadIndex: + """Maintain batch-equivalent thread roots across atomic mailbox changes.""" + + def __init__( + self, + *, + group_by_subject: bool = False, + sort_by_sent_date: bool = False, + max_snapshot_records: int = _DEFAULT_MAX_SNAPSHOT_RECORDS, + max_snapshot_bytes: int = _DEFAULT_MAX_SNAPSHOT_BYTES, + ) -> None: + """Create an empty index with batch-compatible options and snapshot bounds.""" + if not isinstance(group_by_subject, bool): + raise IncrementalThreadError("group_by_subject must be a boolean") + if not isinstance(sort_by_sent_date, bool): + raise IncrementalThreadError("sort_by_sent_date must be a boolean") + self._group_by_subject = group_by_subject + self._sort_by_sent_date = sort_by_sent_date + self._max_snapshot_records = _validated_positive_limit( + max_snapshot_records, + "max_snapshot_records", + ) + self._max_snapshot_bytes = _validated_positive_limit( + max_snapshot_bytes, + "max_snapshot_bytes", + ) + self._version = 0 + self._records: dict[str, IndexedMessage] = {} + self._positions: dict[str, int] = {} + self._next_position = 1 + self._tokens_by_key: dict[str, frozenset[str]] = {} + self._keys_by_token: dict[str, set[str]] = {} + self._component_by_key: dict[str, str] = {} + self._keys_by_component: dict[str, tuple[str, ...]] = {} + self._roots_by_component: dict[str, tuple[Container, ...]] = {} + self._projections_by_component: dict[str, tuple[ThreadProjection, ...]] = {} + self._roots: tuple[Container, ...] = () + self._projections: tuple[ThreadProjection, ...] = () + + def __len__(self) -> int: + """Return the number of indexed caller message keys.""" + return len(self._records) + + @property + def version(self) -> int: + """Return the optimistic mailbox-state version.""" + return self._version + + @property + def message_keys(self) -> tuple[str, ...]: + """Return current caller keys in stable batch input order.""" + return _ordered_keys(self._records, self._positions) + + @property + def roots(self) -> tuple[Container, ...]: + """Return the current transport-neutral thread roots.""" + return self._roots + + @property + def projections(self) -> tuple[ThreadProjection, ...]: + """Return deterministic caller-key projections for current roots.""" + return self._projections + + def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: + """Atomically apply one optimistic mailbox change set. + + Raises: + VersionConflictError: ``expected_version`` is stale. + IncrementalThreadError: Key ownership, metadata, or graph processing + violates the public contract. The existing state remains unchanged. + ExternalIdentityError: Reported EMAILID/THREADID metadata changes or + conflicts across equal EMAILID values. + """ + if not isinstance(change_set, MailboxChangeSet): + raise IncrementalThreadError("change_set must be a MailboxChangeSet") + if change_set.expected_version != self._version: + raise VersionConflictError( + f"expected version {change_set.expected_version}; " + f"current version {self._version}" + ) + if not ( + change_set.additions or change_set.replacements or change_set.removals + ): + return _thread_delta( + self._version, + self._version, + (), + self._projections, + self._projections, + ) + + addition_keys = {record.message_key for record in change_set.additions} + replacement_keys = {record.message_key for record in change_set.replacements} + removal_keys = set(change_set.removals) + existing_keys = set(self._records) + already_present = addition_keys & existing_keys + missing_replacements = replacement_keys - existing_keys + missing_removals = removal_keys - existing_keys + if already_present: + raise IncrementalThreadError( + f"addition keys already exist: {sorted(already_present)!r}" + ) + if missing_replacements: + raise IncrementalThreadError( + f"replacement keys do not exist: {sorted(missing_replacements)!r}" + ) + if missing_removals: + raise IncrementalThreadError( + f"removal keys do not exist: {sorted(missing_removals)!r}" + ) + + copied_additions = tuple( + _copied_indexed_message(record) for record in change_set.additions + ) + copied_replacements = tuple( + _copied_indexed_message(record) for record in change_set.replacements + ) + for replacement in copied_replacements: + _validate_replacement_identity( + self._records[replacement.message_key], + replacement, + ) + + records = dict(self._records) + positions = dict(self._positions) + tokens_by_key = dict(self._tokens_by_key) + keys_by_token = _copy_token_buckets(self._keys_by_token) + next_position = self._next_position + changed_existing_keys = replacement_keys | removal_keys + candidate_seeds: set[str] = set() + touched_tokens: set[str] = set() + + for key in changed_existing_keys: + component_id = self._component_by_key.get(key) + if component_id is not None: + candidate_seeds.update(self._keys_by_component[component_id]) + touched_tokens.update( + _remove_key_from_buckets(key, tokens_by_key, keys_by_token) + ) + + for key in removal_keys: + records.pop(key) + positions.pop(key) + for replacement in copied_replacements: + records[replacement.message_key] = replacement + tokens = _connectivity_tokens( + replacement, + group_by_subject=self._group_by_subject, + ) + touched_tokens.update(tokens) + _add_key_to_buckets( + replacement.message_key, + tokens, + tokens_by_key, + keys_by_token, + ) + candidate_seeds.add(replacement.message_key) + for addition in copied_additions: + records[addition.message_key] = addition + positions[addition.message_key] = next_position + next_position += 1 + tokens = _connectivity_tokens( + addition, + group_by_subject=self._group_by_subject, + ) + touched_tokens.update(tokens) + _add_key_to_buckets( + addition.message_key, + tokens, + tokens_by_key, + keys_by_token, + ) + candidate_seeds.add(addition.message_key) + + for token in touched_tokens: + candidate_seeds.update(keys_by_token.get(token, set())) + for key in tuple(candidate_seeds): + component_id = self._component_by_key.get(key) + if component_id is not None: + candidate_seeds.update(self._keys_by_component[component_id]) + + _validate_external_identities(records) + ranks = _current_ranks(positions) + if self._sort_by_sent_date: + _validate_effective_sequence_numbers(records, ranks) + + current_candidate_keys = _expand_candidate_keys( + candidate_seeds, + tokens_by_key, + keys_by_token, + ) + old_candidate_keys = set(candidate_seeds) + candidate_keys = current_candidate_keys | old_candidate_keys + + component_by_key = { + key: component_id + for key, component_id in self._component_by_key.items() + if key not in candidate_keys and key in records + } + unaffected_component_ids = set(component_by_key.values()) + keys_by_component = { + component_id: keys + for component_id, keys in self._keys_by_component.items() + if component_id in unaffected_component_ids + } + roots_by_component = { + component_id: roots + for component_id, roots in self._roots_by_component.items() + if component_id in unaffected_component_ids + } + projections_by_component = { + component_id: projections + for component_id, projections in self._projections_by_component.items() + if component_id in unaffected_component_ids + } + + for keys in _partition_components( + current_candidate_keys, + positions, + tokens_by_key, + keys_by_token, + ): + component_id = keys[0] + roots, projections = _build_component( + keys, + records, + ranks, + group_by_subject=self._group_by_subject, + sort_by_sent_date=self._sort_by_sent_date, + ) + keys_by_component[component_id] = keys + roots_by_component[component_id] = roots + projections_by_component[component_id] = projections + for key in keys: + component_by_key[key] = component_id + + roots, projections = _compose_forest( + roots_by_component, + projections_by_component, + records, + ranks, + sort_by_sent_date=self._sort_by_sent_date, + ) + affected_positions = { + key: positions.get(key, self._positions.get(key, _MAX_IMAP_NUMBER)) + for key in candidate_keys + } + affected = tuple( + sorted(candidate_keys, key=lambda key: (affected_positions[key], key)) + ) + previous_version = self._version + version = previous_version + 1 + delta = _thread_delta( + previous_version, + version, + affected, + self._projections, + projections, + ) + + self._records = records + self._positions = positions + self._next_position = next_position + self._tokens_by_key = tokens_by_key + self._keys_by_token = keys_by_token + self._component_by_key = component_by_key + self._keys_by_component = keys_by_component + self._roots_by_component = roots_by_component + self._projections_by_component = projections_by_component + self._roots = roots + self._projections = projections + self._version = version + return delta + + def snapshot(self) -> dict[str, object]: + """Return deterministic versioned JSON-safe state without payload objects.""" + if len(self._records) > self._max_snapshot_records: + raise IncrementalThreadError("snapshot exceeds max_snapshot_records") + records: list[dict[str, object]] = [] + for key in self.message_keys: + record = self._records[key] + message = record.message + records.append( + { + "message_key": key, + "email_id": record.email_id, + "thread_id": record.thread_id, + "message": { + "message_id": message.message_id, + "in_reply_to": list(_reference_ids(message.in_reply_to)), + "references": list(_reference_ids(message.references)), + "subject": message.subject, + "sent_date": _encoded_date(message.sent_date), + "internal_date": _encoded_date(message.internal_date), + "sequence_number": message.sequence_number, + "uid": message.uid, + }, + } + ) + snapshot: dict[str, object] = { + "schema_version": _SNAPSHOT_SCHEMA_VERSION, + "version": self._version, + "options": { + "group_by_subject": self._group_by_subject, + "sort_by_sent_date": self._sort_by_sent_date, + }, + "records": records, + } + if len(_snapshot_json_bytes(snapshot)) > self._max_snapshot_bytes: + raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") + return snapshot + + @classmethod + def restore( + cls, + snapshot: Mapping[str, object], + *, + max_snapshot_records: int = _DEFAULT_MAX_SNAPSHOT_RECORDS, + max_snapshot_bytes: int = _DEFAULT_MAX_SNAPSHOT_BYTES, + ) -> IncrementalThreadIndex: + """Restore strict schema-version-1 state and rebuild all derived indexes.""" + max_records = _validated_positive_limit( + max_snapshot_records, + "max_snapshot_records", + ) + max_bytes = _validated_positive_limit( + max_snapshot_bytes, + "max_snapshot_bytes", + ) + if not isinstance(snapshot, Mapping): + raise IncrementalThreadError("snapshot must be a mapping") + if len(_snapshot_json_bytes(snapshot)) > max_bytes: + raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") + _required_fields( + snapshot, + {"schema_version", "version", "options", "records"}, + "snapshot", + ) + if snapshot["schema_version"] != _SNAPSHOT_SCHEMA_VERSION: + raise IncrementalThreadError("unsupported snapshot schema_version") + version = _validated_nonnegative_integer(snapshot["version"], "version") + options = snapshot["options"] + if not isinstance(options, Mapping): + raise IncrementalThreadError("snapshot options must be a mapping") + _required_fields( + options, + {"group_by_subject", "sort_by_sent_date"}, + "option", + ) + group_by_subject = options["group_by_subject"] + sort_by_sent_date = options["sort_by_sent_date"] + if not isinstance(group_by_subject, bool): + raise IncrementalThreadError("group_by_subject must be a boolean") + if not isinstance(sort_by_sent_date, bool): + raise IncrementalThreadError("sort_by_sent_date must be a boolean") + encoded_records = snapshot["records"] + if not isinstance(encoded_records, list): + raise IncrementalThreadError("snapshot records must be a list") + if len(encoded_records) > max_records: + raise IncrementalThreadError("snapshot exceeds max_snapshot_records") + + records: list[IndexedMessage] = [] + seen_keys: set[str] = set() + record_fields = {"message_key", "email_id", "thread_id", "message"} + message_fields = { + "message_id", + "in_reply_to", + "references", + "subject", + "sent_date", + "internal_date", + "sequence_number", + "uid", + } + for encoded_record in encoded_records: + if not isinstance(encoded_record, Mapping): + raise IncrementalThreadError("snapshot record must be a mapping") + _required_fields(encoded_record, record_fields, "record") + key = _validated_message_key(encoded_record["message_key"]) + if key in seen_keys: + raise IncrementalThreadError(f"duplicate message_key in snapshot: {key}") + seen_keys.add(key) + encoded_message = encoded_record["message"] + if not isinstance(encoded_message, Mapping): + raise IncrementalThreadError("snapshot message must be a mapping") + _required_fields(encoded_message, message_fields, "message") + in_reply_to = encoded_message["in_reply_to"] + references = encoded_message["references"] + if not isinstance(in_reply_to, list) or not all( + isinstance(value, str) for value in in_reply_to + ): + raise IncrementalThreadError("in_reply_to must be a list of strings") + if not isinstance(references, list) or not all( + isinstance(value, str) for value in references + ): + raise IncrementalThreadError("references must be a list of strings") + message = Message( + message_id=_validated_optional_text( + encoded_message["message_id"], + "message_id", + ), + in_reply_to=tuple(in_reply_to), + references=tuple(references), + subject=_validated_optional_text( + encoded_message["subject"], + "subject", + ), + payload=None, + sent_date=_decoded_date(encoded_message["sent_date"], "sent_date"), + internal_date=_decoded_date( + encoded_message["internal_date"], + "internal_date", + ), + sequence_number=_validated_optional_number( + encoded_message["sequence_number"], + "sequence_number", + ), + uid=_validated_optional_number( + encoded_message["uid"], + "uid", + maximum=_MAX_IMAP_NUMBER, + ), + ) + records.append( + IndexedMessage( + message_key=key, + message=message, + email_id=_validated_external_id( + encoded_record["email_id"], + "email_id", + ), + thread_id=_validated_external_id( + encoded_record["thread_id"], + "thread_id", + ), + ) + ) + + index = cls( + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + max_snapshot_records=max_records, + max_snapshot_bytes=max_bytes, + ) + if records: + index.apply( + MailboxChangeSet(expected_version=0, additions=tuple(records)) + ) + index._version = version + return index From dee120b2d09845d065cb80d63ce4e84d491710bd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:26:31 +0900 Subject: [PATCH 08/97] feat: export incremental mailbox API --- src/threadweave/__init__.py | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/src/threadweave/__init__.py b/src/threadweave/__init__.py index 24500b0..e6082a4 100644 --- a/src/threadweave/__init__.py +++ b/src/threadweave/__init__.py @@ -3,8 +3,8 @@ Assemble flat iterables of email messages into conversation trees using Jamie Zawinski's container algorithm and RFC 5256 REFERENCES semantics, on top of RFC 5322 identification-field parsing, RFC 5051 Unicode casemap comparison, exact -RFC 5256 base-subject extraction, optional sent-date ordering, and IMAP THREAD -response serialization. +RFC 5256 base-subject extraction, optional sent-date ordering, IMAP THREAD +response serialization, and atomic incremental mailbox updates. from threadweave import Message, thread_messages @@ -16,9 +16,9 @@ # -> one root Container; a is an ancestor of b, b of c. The RFC 5322 header primitives (:mod:`threadweave.headers`) are extracted -behaviour-preserving from the naruon control plane; the threading, subject, -collation, date, and protocol-projection layers are standalone implementations -grounded in published standards. +behaviour-preserving from the naruon control plane; the threading, incremental, +subject, collation, date, and protocol-projection layers are standalone +implementations grounded in published standards. """ from threadweave.adapters import message_from_email, thread_email_messages @@ -31,6 +31,17 @@ generate_email_fingerprint, normalize_message_id, ) +from threadweave.incremental import ( + ExternalIdentityError, + IncrementalThreadError, + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + ThreadDelta, + ThreadProjection, + ThreadTransition, + VersionConflictError, +) from threadweave.imap import ( IdentifierResolver, MessageFilter, @@ -52,10 +63,19 @@ __all__ = [ "Container", "DateValue", + "ExternalIdentityError", "IdentifierResolver", + "IncrementalThreadError", + "IncrementalThreadIndex", + "IndexedMessage", + "MailboxChangeSet", "Message", "MessageFilter", + "ThreadDelta", + "ThreadProjection", "ThreadSerializationError", + "ThreadTransition", + "VersionConflictError", "decode_header_text", "extract_reference_ids", "generate_email_fingerprint", From 75249b83ecc565b467806ae2de782bb720ea4545 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:28:56 +0900 Subject: [PATCH 09/97] test: cover incremental validation branches --- tests/test_incremental_validation.py | 144 +++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 tests/test_incremental_validation.py diff --git a/tests/test_incremental_validation.py b/tests/test_incremental_validation.py new file mode 100644 index 0000000..102f8dc --- /dev/null +++ b/tests/test_incremental_validation.py @@ -0,0 +1,144 @@ +"""Defensive validation coverage for incremental mailbox boundaries.""" + +from __future__ import annotations + +from copy import deepcopy + +import pytest + +from threadweave import ( + IncrementalThreadError, + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, +) + + +def _record(key: str, message: Message | None = None, **identity: str) -> IndexedMessage: + """Build one record for focused validation tests.""" + return IndexedMessage( + key, + Message(message_id=key) if message is None else message, + **identity, + ) + + +def test_public_containers_reject_wrong_element_and_message_types(): + """Runtime boundaries reject values hidden behind static annotations.""" + with pytest.raises(IncrementalThreadError, match="IndexedMessage"): + MailboxChangeSet( + expected_version=0, + additions=(object(),), # type: ignore[arg-type] + ) + with pytest.raises(IncrementalThreadError, match="threadweave.Message"): + IndexedMessage("a", object()) # type: ignore[arg-type] + + +def test_apply_rejects_invalid_message_metadata_before_mutation(): + """Malformed text, references, dates, and IMAP numbers fail atomically.""" + bad_messages = ( + Message(message_id=object()), # type: ignore[arg-type] + Message(subject=object()), # type: ignore[arg-type] + Message(references=1), # type: ignore[arg-type] + Message(references=("ok", 1)), # type: ignore[arg-type] + Message(sent_date=object()), # type: ignore[arg-type] + Message(uid=4_294_967_296), + ) + for position, message in enumerate(bad_messages): + index = IncrementalThreadIndex() + with pytest.raises(IncrementalThreadError): + index.apply( + MailboxChangeSet( + expected_version=0, + additions=(_record(f"bad_{position}", message),), + ) + ) + assert index.version == 0 + + +def test_constructor_and_apply_reject_wrong_runtime_boundary_types(): + """Boolean options and arbitrary change objects never enter mutable state.""" + with pytest.raises(IncrementalThreadError, match="group_by_subject"): + IncrementalThreadIndex(group_by_subject=1) # type: ignore[arg-type] + with pytest.raises(IncrementalThreadError, match="sort_by_sent_date"): + IncrementalThreadIndex(sort_by_sent_date=1) # type: ignore[arg-type] + with pytest.raises(IncrementalThreadError, match="MailboxChangeSet"): + IncrementalThreadIndex().apply(object()) # type: ignore[arg-type] + + +def test_snapshot_record_limit_is_enforced_at_publication_time(): + """A lowered publication bound cannot emit an oversized state document.""" + index = IncrementalThreadIndex(max_snapshot_records=1) + index.apply( + MailboxChangeSet( + expected_version=0, + additions=(_record("a"), _record("b")), + ) + ) + with pytest.raises(IncrementalThreadError, match="max_snapshot_records"): + index.snapshot() + + +def test_restore_rejects_remaining_shape_and_scalar_ambiguities(): + """Nested snapshot fields use exact mapping, list, and scalar types.""" + source = IncrementalThreadIndex() + source.apply(MailboxChangeSet(expected_version=0, additions=(_record("a"),))) + snapshot = source.snapshot() + + invalid_values: list[tuple[dict[str, object], str]] = [] + options_not_mapping = deepcopy(snapshot) + options_not_mapping["options"] = [] + invalid_values.append((options_not_mapping, "options")) + + invalid_group_option = deepcopy(snapshot) + invalid_group_option["options"]["group_by_subject"] = 1 # type: ignore[index] + invalid_values.append((invalid_group_option, "group_by_subject")) + + invalid_sort_option = deepcopy(snapshot) + invalid_sort_option["options"]["sort_by_sent_date"] = 1 # type: ignore[index] + invalid_values.append((invalid_sort_option, "sort_by_sent_date")) + + record_not_mapping = deepcopy(snapshot) + record_not_mapping["records"] = [1] + invalid_values.append((record_not_mapping, "record")) + + invalid_reply = deepcopy(snapshot) + invalid_reply["records"][0]["message"]["in_reply_to"] = [1] # type: ignore[index] + invalid_values.append((invalid_reply, "in_reply_to")) + + invalid_references = deepcopy(snapshot) + invalid_references["records"][0]["message"]["references"] = [1] # type: ignore[index] + invalid_values.append((invalid_references, "references")) + + malformed_date = deepcopy(snapshot) + malformed_date["records"][0]["message"]["sent_date"] = [] # type: ignore[index] + invalid_values.append((malformed_date, "date")) + + nontext_date = deepcopy(snapshot) + nontext_date["records"][0]["message"]["sent_date"] = { # type: ignore[index] + "kind": "text", + "value": 1, + } + invalid_values.append((nontext_date, "textual")) + + for invalid, pattern in invalid_values: + with pytest.raises(IncrementalThreadError, match=pattern): + IncrementalThreadIndex.restore(invalid) + + +def test_empty_snapshot_restore_skips_initial_rebuild(): + """A versioned empty mailbox restores without a fabricated change.""" + restored = IncrementalThreadIndex.restore( + { + "schema_version": 1, + "version": 7, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [], + } + ) + assert restored.version == 7 + assert restored.message_keys == () From b0bf3e6bae4dd7113a266bdefb4ab07a60fb1836 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:29:40 +0900 Subject: [PATCH 10/97] test: cover incremental graph defenses --- tests/test_incremental_private_graph.py | 114 ++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 tests/test_incremental_private_graph.py diff --git a/tests/test_incremental_private_graph.py b/tests/test_incremental_private_graph.py new file mode 100644 index 0000000..05efffe --- /dev/null +++ b/tests/test_incremental_private_graph.py @@ -0,0 +1,114 @@ +"""Defensive graph-branch coverage for the incremental index.""" + +from __future__ import annotations + +import pytest + +import threadweave.incremental as incremental +from threadweave import ( + Container, + IncrementalThreadError, + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + ThreadProjection, +) + + +def _record(key: str, message: Message | None = None, **identity: str) -> IndexedMessage: + """Build one record for focused graph tests.""" + return IndexedMessage( + key, + Message(message_id=key) if message is None else message, + **identity, + ) + + +def test_duplicate_references_missing_id_and_consistent_email_id_are_supported(): + """Deduplication and optional ID branches preserve valid historical mail.""" + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record( + "a", + Message(message_id=None, references=("", "")), + email_id="M1", + thread_id="T1", + ), + _record( + "b", + Message(message_id="b"), + email_id="M1", + thread_id="T1", + ), + ), + ) + ) + assert index.message_keys == ("a", "b") + + +def test_candidate_expansion_adds_neighbors_reached_through_tokens(): + """The iterative candidate queue crosses every current reverse bucket.""" + assert incremental._expand_candidate_keys( + {"a"}, + {"a": frozenset({"token"}), "b": frozenset({"token"})}, + {"token": {"a", "b"}}, + ) == {"a", "b"} + + +def test_projection_defenses_cover_dummy_cycle_and_foreign_messages(): + """Private batch-output validation remains loop-safe and fail-closed.""" + message = Message(message_id="a") + record = _record("a", message) + concrete = Container(message=message) + concrete.children = [concrete] + projection = incremental._projection_for_root( + concrete, + {id(message): "a"}, + {"a": record}, + ) + assert projection.message_keys == ("a",) + + dummy = Container() + child = Container(message=message) + dummy.children = [child] + assert incremental._projection_for_root( + dummy, + {id(message): "a"}, + {"a": record}, + ).message_keys == ("a",) + + with pytest.raises(IncrementalThreadError, match="outside its component"): + incremental._projection_for_root(child, {}, {"a": record}) + + +def test_empty_projection_has_an_earliest_safe_sent_date_key(): + """A malformed empty projection remains deterministically sortable.""" + key = incremental._root_order_key( + Container(), + ThreadProjection(()), + {}, + {}, + sort_by_sent_date=True, + ) + assert key == (incremental.normalize_sent_date(None), 0, 0) + + +def test_replacement_recovers_when_derived_component_mapping_is_missing(): + """A defensive update rebuilds a record after derived-state loss.""" + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=(_record("a"),))) + index._component_by_key.pop("a") + + index.apply( + MailboxChangeSet( + expected_version=1, + replacements=( + _record("a", Message(message_id="a", subject="updated")), + ), + ) + ) + assert index.projections[0].message_keys == ("a",) From 7d754213d7d3dd4a3624005249dadfd3dd8812d2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 09:30:22 +0900 Subject: [PATCH 11/97] test: require incremental API docstrings --- tests/test_documentation.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/test_documentation.py b/tests/test_documentation.py index a2dde8e..29906aa 100644 --- a/tests/test_documentation.py +++ b/tests/test_documentation.py @@ -11,6 +11,7 @@ import threadweave.encoded_words import threadweave.headers import threadweave.imap +import threadweave.incremental import threadweave.subject import threadweave.threading @@ -22,6 +23,7 @@ threadweave.encoded_words, threadweave.headers, threadweave.imap, + threadweave.incremental, threadweave.subject, threadweave.threading, ) From 1a88c680fea025aad6f95533fd4e93a464f4fbde Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:09:50 +0900 Subject: [PATCH 12/97] fix: enforce RFC 8474 ObjectID contracts --- src/threadweave/incremental.py | 64 +++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 21 deletions(-) diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index 683fd1a..9150bab 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -1,7 +1,7 @@ """Incremental, identity-aware mailbox threading over the batch RFC engine. The batch :func:`threadweave.thread_messages` function remains the correctness -oracle. This module adds an atomic state boundary that indexes caller-owned +oracle. This module adds an atomic state boundary that indexes caller-owned message keys, recomputes only affected connectivity components, reports explicit thread merge/split transitions, and snapshots JSON-safe metadata without caller payloads. @@ -36,7 +36,10 @@ _batch_thread_messages = thread_messages _MAX_MESSAGE_KEY_LENGTH = 512 -_MAX_EXTERNAL_ID_LENGTH = 1024 +_MAX_EXTERNAL_ID_LENGTH = 255 +_OBJECT_ID_CHARACTERS = frozenset( + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-" +) _MAX_IMAP_NUMBER = 4_294_967_295 _DEFAULT_MAX_SNAPSHOT_RECORDS = 1_000_000 _DEFAULT_MAX_SNAPSHOT_BYTES = 256 * 1024 * 1024 @@ -83,8 +86,8 @@ def _validated_identifier( raise IncrementalThreadError( f"{name} must be a non-empty string of at most {maximum_length} characters" ) - if any(ord(character) < 32 or ord(character) == 127 for character in value): - raise IncrementalThreadError(f"{name} must not contain control characters") + if any(not character.isprintable() for character in value): + raise IncrementalThreadError(f"{name} must contain only printable characters") return value @@ -101,13 +104,20 @@ def _validated_message_key(value: object) -> str: def _validated_external_id(value: object, name: str) -> str | None: - """Return one optional bounded external object identifier.""" - return _validated_identifier( + """Return one optional RFC 8474 ``objectid`` value.""" + validated = _validated_identifier( value, name, allow_none=True, maximum_length=_MAX_EXTERNAL_ID_LENGTH, ) + if validated is not None and any( + character not in _OBJECT_ID_CHARACTERS for character in validated + ): + raise IncrementalThreadError( + f"{name} must use only ASCII letters, digits, underscore, or hyphen" + ) + return validated def _tuple_without_duplicate_keys( @@ -424,19 +434,29 @@ def _validate_effective_sequence_numbers( def _validate_external_identities(records: Mapping[str, IndexedMessage]) -> None: - """Require every shared EMAILID to carry one identical THREADID value.""" + """Enforce RFC 8474 EMAILID/THREADID consistency and namespace separation.""" thread_id_by_email_id: dict[str, str | None] = {} + email_ids: set[str] = set() + thread_ids: set[str] = set() for record in records.values(): email_id = record.email_id - if email_id is None: - continue - if email_id not in thread_id_by_email_id: - thread_id_by_email_id[email_id] = record.thread_id - continue - if thread_id_by_email_id[email_id] != record.thread_id: - raise ExternalIdentityError( - f"messages with EMAILID {email_id!r} must expose the same THREADID" - ) + thread_id = record.thread_id + if email_id is not None: + email_ids.add(email_id) + if email_id not in thread_id_by_email_id: + thread_id_by_email_id[email_id] = thread_id + elif thread_id_by_email_id[email_id] != thread_id: + raise ExternalIdentityError( + f"messages with EMAILID {email_id!r} must expose the same THREADID" + ) + if thread_id is not None: + thread_ids.add(thread_id) + reused_values = email_ids & thread_ids + if reused_values: + raise ExternalIdentityError( + "EMAILID and THREADID must use disjoint ObjectID values: " + f"{sorted(reused_values)!r}" + ) def _validate_replacement_identity( @@ -592,7 +612,9 @@ def _root_order_key( first_key = projection.message_keys[0] message = records[first_key].message sequence_number = ( - ranks[first_key] if message.sequence_number is None else message.sequence_number + ranks[first_key] + if message.sequence_number is None + else message.sequence_number ) return ( normalize_sent_date(message.sent_date, message.internal_date), @@ -755,9 +777,7 @@ def _snapshot_json_bytes(value: object) -> bytes: separators=(",", ":"), ) except (TypeError, ValueError) as error: - raise IncrementalThreadError( - "snapshot must contain only JSON-safe values" - ) from error + raise IncrementalThreadError("snapshot must contain only JSON-safe values") from error return encoded.encode("utf-8") @@ -1044,7 +1064,9 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: def snapshot(self) -> dict[str, object]: """Return deterministic versioned JSON-safe state without payload objects.""" if len(self._records) > self._max_snapshot_records: - raise IncrementalThreadError("snapshot exceeds max_snapshot_records") + raise IncrementalThreadError( + "snapshot exceeds max_snapshot_records" + ) records: list[dict[str, object]] = [] for key in self.message_keys: record = self._records[key] From 925912364463397237a4c77be714152eceecf54c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:11:49 +0900 Subject: [PATCH 13/97] test: enforce RFC 8474 ObjectID contracts --- tests/test_incremental_rfc8474.py | 63 +++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 tests/test_incremental_rfc8474.py diff --git a/tests/test_incremental_rfc8474.py b/tests/test_incremental_rfc8474.py new file mode 100644 index 0000000..98da780 --- /dev/null +++ b/tests/test_incremental_rfc8474.py @@ -0,0 +1,63 @@ +"""RFC 8474 identity grammar and namespace tests for incremental threading.""" + +import pytest + +from threadweave import ( + ExternalIdentityError, + IncrementalThreadError, + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, +) + + +def _record( + key: str, + *, + email_id: str | None = None, + thread_id: str | None = None, +) -> IndexedMessage: + """Build one message carrying optional RFC 8474 identity metadata.""" + return IndexedMessage( + message_key=key, + message=Message(message_id=key), + email_id=email_id, + thread_id=thread_id, + ) + + +def test_object_ids_use_exact_ascii_grammar_and_length(): + """EMAILID and THREADID accept only 1-255 RFC ``objectid`` characters.""" + valid = "Abc_012-Z" + record = _record("a", email_id=valid, thread_id="Thread_1") + assert record.email_id == valid + assert record.thread_id == "Thread_1" + + for field in ("email_id", "thread_id"): + for invalid in ("contains space", "é", "x" * 256, "slash/value"): + with pytest.raises(IncrementalThreadError, match=field): + _record("a", **{field: invalid}) + + +def test_email_id_and_thread_id_namespaces_are_disjoint(): + """One ObjectID value cannot be reused across EMAILID and THREADID data items.""" + index = IncrementalThreadIndex() + with pytest.raises(ExternalIdentityError, match="disjoint ObjectID"): + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record("a", email_id="Shared1", thread_id="Thread1"), + _record("b", email_id="Mail2", thread_id="Shared1"), + ), + ) + ) + assert index.version == 0 + + +def test_message_keys_reject_nonprintable_unicode_without_encoding_failure(): + """Caller keys fail during validation instead of later UTF-8 serialization.""" + for invalid in ("bad\ud800key", "line\u2028separator"): + with pytest.raises(IncrementalThreadError, match="message_key.*printable"): + _record(invalid) From 0a2e128cd1b9709564d3ef524ea9f0608f6666f2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:16:55 +0900 Subject: [PATCH 14/97] docs: explain incremental mailbox threading --- docs/incremental-threading.md | 202 ++++++++++++++++++++++++++++++++++ 1 file changed, 202 insertions(+) create mode 100644 docs/incremental-threading.md diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md new file mode 100644 index 0000000..b59ff27 --- /dev/null +++ b/docs/incremental-threading.md @@ -0,0 +1,202 @@ +# Incremental mailbox threading + +`IncrementalThreadIndex` applies mailbox additions, replacements, and removals +without rebuilding unrelated reference components. It delegates every affected +component to the same `thread_messages` batch implementation used by the public +batch API, so full reconstruction remains the correctness oracle. + +The incremental layer is transport-neutral. It contains no database, network, +authentication, IMAP session, or JMAP state and can be used as a standalone +library or embedded in naruon and other services. + +## Basic use + +```python +from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + serialize_thread_response, +) + +index = IncrementalThreadIndex(sort_by_sent_date=True) + +initial_delta = index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + IndexedMessage( + message_key="mailbox:101", + message=Message( + message_id="", + sent_date="1 Jan 2026 00:00:00 +0000", + sequence_number=1, + uid=101, + ), + email_id="Email_101", + thread_id="Thread_7", + ), + IndexedMessage( + message_key="mailbox:102", + message=Message( + message_id="", + references="", + sent_date="2 Jan 2026 00:00:00 +0000", + sequence_number=2, + uid=102, + ), + email_id="Email_102", + thread_id="Thread_7", + ), + ), + ) +) + +assert initial_delta.version == 1 +assert index.projections[0].message_keys == ( + "mailbox:101", + "mailbox:102", +) +assert serialize_thread_response(index.roots, identifier="uid") == ( + "* THREAD (101 102)\r\n" +) +``` + +The caller-owned `message_key` is the index identity. It must not be derived from +an IMAP sequence number, because sequence numbers change after expunge. A stable +mailbox row key, object key, or application identifier is appropriate. + +## Atomic changes and optimistic versions + +One `MailboxChangeSet` may contain additions, replacements, and removals. Its key +sets must be disjoint. Additions require absent keys; replacements and removals +require existing keys. The request is validated and recomputed on copied state; +any failure leaves the index version, records, roots, and projections unchanged. + +```python +updated = index.apply( + MailboxChangeSet( + expected_version=index.version, + replacements=( + IndexedMessage( + message_key="mailbox:102", + message=Message( + message_id="", + references="", + sequence_number=2, + uid=102, + ), + email_id="Email_102", + thread_id="Thread_7", + ), + ), + ) +) + +assert updated.previous_version == 1 +assert updated.version == 2 +``` + +`VersionConflictError` reports a stale `expected_version`. Reapplying a change +with the old version therefore fails explicitly instead of duplicating records or +edges. An empty change set is idempotent and does not advance the version. + +## Affected-component recomputation + +Each record contributes connectivity tokens for: + +- normalized `Message-ID`; +- the RFC 5256 effective `References` chain, or the first valid `In-Reply-To` + identifier when `References` is unavailable; +- the RFC 5051 base-subject key when `group_by_subject=True`. + +The token graph deliberately over-approximates the final thread graph. A change +may therefore recompute a component that proves unchanged, but it cannot omit a +reference or subject dependency. Replacing or removing a record seeds its whole +old component; a new bridge message includes every old component touched by its +new tokens. The candidate region is repartitioned iteratively and each resulting +component is processed by `thread_messages`. + +Unchanged components retain their existing `Container` roots. Building the +ordered public root tuple still examines the component-root summaries so that +batch-compatible global ordering is preserved; it does not re-run threading or +walk every message in unrelated components. + +`ThreadDelta.affected_message_keys` reports the candidate region. The delta also +contains added, removed, and structurally updated projections. A metadata-only +replacement can leave a projection unchanged while its key still appears in the +affected set. + +## External EMAILID and THREADID handoff + +`email_id` and `thread_id` are optional caller-owned RFC 8474 values. ThreadWeave +validates the RFC `objectid` grammar: 1 through 255 ASCII letters, digits, +underscore, or hyphen. The values are case-sensitive. + +The incremental layer enforces these rules: + +- a reported `email_id` or `thread_id` cannot be removed or changed by a + replacement; +- equal non-null EMAILID values must expose the same THREADID value; +- the EMAILID and THREADID namespaces cannot reuse the same ObjectID value; +- ThreadWeave never selects a canonical external THREADID; +- a structural merge or split is returned as an explicit `ThreadTransition`. + +This prevents a server adapter from silently changing a THREADID that has already +been exposed. A transport-specific service can consume the transition and apply +its own documented policy. + +## Snapshot and restore + +```python +snapshot = index.snapshot() +restored = IncrementalThreadIndex.restore(snapshot) + +assert restored.version == index.version +assert restored.projections == index.projections +``` + +Schema version 1 stores: + +- batch options; +- optimistic version; +- stable record order; +- structural message metadata; +- optional EMAILID and THREADID values. + +Payload objects and derived graph pointers are never serialized. Restored +messages therefore have `payload=None`. Date values use an explicit tagged text +or ISO-8601 datetime representation. Restore rejects unknown schema versions, +extra or missing fields, duplicate keys, malformed types, invalid external IDs, +and configured record or byte limits before publishing state. + +## Correctness and operational boundaries + +The test suite compares additions, delayed ancestors, bridge merges, component +splits, replacement, and root/internal/leaf removal with a complete batch rebuild. +It also covers RFC 5051 subject buckets, RFC 5256 sent-date ordering, ordinary and +UID THREAD output, duplicate and missing Message-ID values, deep chains, +optimistic conflicts, hostile snapshots, and payload omission. + +A separate scheduled/manual benchmark is required before this feature is released +as a performance claim. It must exercise at least 100,000 records and report wall +time, peak RSS, affected-message count, and a full-rebuild comparison. The +incremental contract promises that unrelated records are not passed to the batch +threader; it does not promise constant-time global root presentation. + +## References + +Gondwana, B. (2018). *IMAP extension for object identifiers* (RFC 8474). RFC +Editor. https://doi.org/10.17487/RFC8474 + +Jenkins, N., & Newman, C. (2019). *The JSON Meta Application Protocol (JMAP) for +mail* (RFC 8621). RFC Editor. https://doi.org/10.17487/RFC8621 + +Melnikov, A., & Cridland, D. (2014). *IMAP extensions: Quick flag changes +resynchronization (CONDSTORE) and quick mailbox resynchronization (QRESYNC)* +(RFC 7162). RFC Editor. https://doi.org/10.17487/RFC7162 + +Melnikov, A., & Leiba, B. (Eds.). (2021). *Internet Message Access Protocol +(IMAP)—Version 4rev2* (RFC 9051). RFC Editor. +https://doi.org/10.17487/RFC9051 From b96ee322b4142a468bea93a84fbf2ef08f1a9f8c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:21:31 +0900 Subject: [PATCH 15/97] docs: record incremental mailbox changes --- CHANGELOG.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a121fa6..eb33f47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,15 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Add an atomic `IncrementalThreadIndex` for mailbox additions, replacements, + and removals with stable caller message keys, affected-component + recomputation, batch-result parity, and explicit thread merge/split deltas. +- Add strict RFC 8474 `EMAILID` and `THREADID` handoff, including immutable + values, exact ObjectID grammar, disjoint namespaces, and consistent THREADID + values for equal EMAILIDs. +- Add versioned, bounded, JSON-safe incremental snapshots that omit arbitrary + caller payloads and rebuild derived graph state through validated metadata. + ## [0.2.0] - 2026-08-04 - Add iterative RFC 5256 `THREAD` and `UID THREAD` response serialization with From b813fd8e6d656d8473d8fbdea9b17566a8969a0f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:29:28 +0900 Subject: [PATCH 16/97] docs: document incremental mailbox API --- README.md | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 70 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 2e78d16..416381a 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ runtime dependencies.** trees. It combines the JWZ container model with RFC 5322 identification fields, RFC 2047 encoded-word decoding, RFC 5256 base-subject extraction and optional sent-date ordering, RFC 5051 `i;unicode-casemap` comparison, and RFC 5256 IMAP -`THREAD` response serialization. +`THREAD` response serialization, and atomic incremental mailbox updates. It accepts normalized identifiers, raw header strings, or Python standard-library `email.message.Message` objects. Malformed historical mail, missing roots, @@ -167,6 +167,61 @@ source containers are never mutated. Cycles, shared nodes, duplicate numbers, missing UIDs, values outside the non-zero unsigned 32-bit range, and unsafe line endings fail closed. Both deep chains and nested splits are rendered iteratively. +## Incremental mailbox updates + +`IncrementalThreadIndex` applies atomic additions, replacements, and removals +without re-threading unrelated reference components. Caller-owned message keys +remain stable across expunge or mailbox sequence-number changes; optional RFC +8474 `EMAILID` and `THREADID` values are validated and never silently rewritten. + +```python +from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, +) + +index = IncrementalThreadIndex() +delta = index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + IndexedMessage( + "mailbox:101", + Message(message_id="root", sequence_number=1, uid=101), + email_id="Email_101", + thread_id="Thread_7", + ), + IndexedMessage( + "mailbox:102", + Message( + message_id="reply", + references=["root"], + sequence_number=2, + uid=102, + ), + email_id="Email_102", + thread_id="Thread_7", + ), + ), + ) +) + +assert delta.version == 1 +assert index.projections[0].message_keys == ("mailbox:101", "mailbox:102") +assert IncrementalThreadIndex.restore(index.snapshot()).projections == ( + index.projections +) +``` + +Every affected component is recomputed through the canonical batch threader, and +full-rebuild parity is the correctness oracle. Structural merges and splits are +reported explicitly. Versioned snapshots omit arbitrary payloads and reject +unknown, malformed, or oversized input. See +[`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, +identity, snapshot, complexity, and RFC boundaries. + ## Public API | Symbol | Purpose | @@ -176,6 +231,11 @@ endings fail closed. Both deep chains and nested splits are rendered iteratively | `thread_messages(...)` | Build JWZ/RFC 5256 thread roots from any iterable. | | `message_from_email(...)` | Convert one stdlib email object. | | `thread_email_messages(...)` | Convert and thread stdlib email objects. | +| `IncrementalThreadIndex` | Apply atomic mailbox deltas and expose batch-equivalent roots. | +| `IndexedMessage` | Bind one stable caller key and optional RFC 8474 identities to a message. | +| `MailboxChangeSet` | Describe one optimistic additions/replacements/removals transaction. | +| `ThreadDelta` | Report affected keys, projection changes, merges, and splits. | +| `ThreadProjection` | Describe one root with traversal-ordered caller keys and THREADIDs. | | `serialize_thread_data(...)` | Render RFC 5256 `thread-data` without response framing. | | `serialize_thread_response(...)` | Render one untagged `* THREAD` response. | | `ThreadSerializationError` | Report invalid graph or mailbox identifier state. | @@ -202,8 +262,9 @@ endings fail closed. Both deep chains and nested splits are rendered iteratively on Python 3.10, 3.11, 3.12, and 3.13. - CI builds wheel and source distributions, verifies `py.typed`, installs the wheel outside the source tree, and executes a smoke test. -- Graph operations and IMAP rendering are iterative and identity-guarded; deep - or cyclic malformed input cannot recurse indefinitely. +- Graph operations, IMAP rendering, and incremental component traversal are + iterative and identity-guarded; deep or cyclic malformed input cannot recurse + indefinitely. ## Reproducible CI supply chain @@ -273,12 +334,14 @@ unavailable safe proposal stops the cycle without mutation. ## Architecture and standards boundary The package remains useful both as a standalone dependency and as a module in -`naruon` or another service. The threading, subject, collation, and date layers -are transport-neutral. IMAP `THREAD` response serialization is a separate -presentation layer rather than protocol state embedded in the core model. +`naruon` or another service. The batch-threading, incremental-index, subject, +collation, and date layers are transport-neutral. IMAP `THREAD` response +serialization is a separate presentation layer rather than protocol state +embedded in the core model. See [`docs/research`](docs/research/README.md) for JWZ, RFC 5322, RFC 2047, -RFC 5051, RFC 5256, RFC 6532, RFC 9051, Unicode-version boundaries, and PEP 561. +RFC 5051, RFC 5256, RFC 6532, RFC 7162, RFC 8474, RFC 8621, RFC 9051, +Unicode-version boundaries, and PEP 561. ## License From 459da07b0e7527f7ae36d1025c460a66c2300b57 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:31:38 +0900 Subject: [PATCH 17/97] ci: smoke-test installed incremental API --- .github/workflows/ci.yml | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 678b550..b34c79d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -143,6 +143,9 @@ jobs: cd "$temp_dir" python - <<'PY' from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, Message, __version__, serialize_thread_response, @@ -170,6 +173,39 @@ jobs: "later@example.com", ] assert serialize_thread_response(roots) == "* THREAD (1)(2)\r\n" + + index = IncrementalThreadIndex() + delta = index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + IndexedMessage( + "root-key", + Message(message_id="root", sequence_number=1, uid=101), + email_id="Email_101", + thread_id="Thread_7", + ), + IndexedMessage( + "reply-key", + Message( + message_id="reply", + references=["root"], + sequence_number=2, + uid=102, + ), + email_id="Email_102", + thread_id="Thread_7", + ), + ), + ) + ) + assert delta.version == 1 + assert index.projections[0].message_keys == ("root-key", "reply-key") + assert serialize_thread_response(index.roots, identifier="uid") == ( + "* THREAD (101 102)\r\n" + ) + restored = IncrementalThreadIndex.restore(index.snapshot()) + assert restored.projections == index.projections PY - run: python -m pip check - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 From 459a5bde9f812f9dee95a628d80c4cfc5d2b2ce8 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 10:32:56 +0900 Subject: [PATCH 18/97] docs: add incremental threading invariants --- AGENTS.md | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b625b3f..d5b25b8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,11 +5,12 @@ Operating guide for automated agents working on this repository. `threadweave` implements the JWZ container model with RFC 5256 `REFERENCES` threading semantics, RFC 5322 identification-field parsing, RFC 2047 encoded-word decoding, RFC 5256 base-subject extraction, RFC 5051 Unicode casemap comparison, -optional RFC 5256 sent-date ordering, and RFC 5256 IMAP `THREAD` response -serialization. Its value is correctness: mail clients and ingestion systems rely -on threading being deterministic, standards-grounded, and impossible to hang on -malformed input. Treat changes to `threading.py`, `container.py`, `subject.py`, -`collation.py`, `dates.py`, `headers.py`, and `imap.py` as behavior-sensitive. +optional RFC 5256 sent-date ordering, RFC 5256 IMAP `THREAD` response +serialization, and atomic incremental mailbox indexing. Its value is correctness: +mail clients and ingestion systems rely on threading being deterministic, +standards-grounded, and impossible to hang on malformed input. Treat changes to +`threading.py`, `incremental.py`, `container.py`, `subject.py`, `collation.py`, +`dates.py`, `headers.py`, and `imap.py` as behavior-sensitive. ## Invariants that must not regress @@ -55,6 +56,17 @@ malformed input. Treat changes to `threading.py`, `container.py`, `subject.py`, dummy-root grouping after search projection, reject cyclic or shared graphs, and leave the source `Container` tree unchanged. Rendering stays iterative, and response framing accepts only CRLF or a caller-owned empty suffix. +13. **Incremental updates remain batch-equivalent and atomic.** Immutable caller + message keys—not sequence numbers—identify indexed records. Recompute every + affected old/new reference or subject component through `thread_messages`, + but do not pass unrelated components to the batch delegate. Validation or + recomputation failure must leave records, version, roots, and projections + unchanged. Snapshot state excludes arbitrary payloads and graph pointers. +14. **External identities follow RFC 8474.** `EMAILID` and `THREADID` use exact + 1–255 character ObjectID grammar, are case-sensitive, and use disjoint + namespaces. Equal EMAILIDs require equal THREADIDs. Once a non-null value is + reported, replacement cannot remove or change it; merges and splits remain + explicit transitions rather than silent identity rewrites. ## Architecture and dependency rules @@ -63,10 +75,15 @@ malformed input. Treat changes to `threading.py`, `container.py`, `subject.py`, justified. - Preserve the standalone package API and its use as a naruon module. The header primitives originated in naruon; port behavioral fixes in both directions. -- Keep IMAP response serialization separate from the transport-neutral tree and - date layers so non-IMAP callers do not inherit protocol-specific state. -- Public behavior, compatibility aliases, and typing markers are release - contracts. Record changes in `CHANGELOG.md` and update user/research docs. +- Keep batch threading authoritative. The incremental layer owns caller keys, + component bookkeeping, deltas, and payload-free snapshots; it must delegate + every recomputed component to the existing batch engine rather than fork the + threading algorithm. +- Keep IMAP response serialization separate from the transport-neutral batch, + incremental, and date layers so non-IMAP callers do not inherit protocol state. +- Public behavior, compatibility aliases, typing markers, snapshot schemas, and + external-ID handoff are release contracts. Record changes in `CHANGELOG.md` + and update user/research docs. - Unicode collation results depend on the Unicode Character Database bundled with the supported Python runtime. Tests must cover stable RFC examples and security-sensitive non-equivalences rather than version-specific new codepoints. From 668d2929383d68d0ec30f6e3746a6728f77479d1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:31:52 +0900 Subject: [PATCH 19/97] ci: stage verified incremental update bundle --- .github/workflows/apply-pr20-update.yml | 150 ++++++++++++++++++++++++ tools/pr20-update.zip | Bin 0 -> 12921 bytes 2 files changed, 150 insertions(+) create mode 100644 .github/workflows/apply-pr20-update.yml create mode 100644 tools/pr20-update.zip diff --git a/.github/workflows/apply-pr20-update.yml b/.github/workflows/apply-pr20-update.yml new file mode 100644 index 0000000..a1fe9e0 --- /dev/null +++ b/.github/workflows/apply-pr20-update.yml @@ -0,0 +1,150 @@ +name: Apply PR 20 verified update + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - tools/pr20-update.zip + - .github/workflows/apply-pr20-update.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-update + cancel-in-progress: false + +jobs: + apply-and-verify: + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the update branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Verify and extract the bounded update archive + shell: bash + run: | + set -euo pipefail + printf '%s %s\n' \ + '2b18426b1afa0b8001753e1e3c372f9ec33af1ba793d7052fee07ec0aa174523' \ + 'tools/pr20-update.zip' | sha256sum -c - + python - <<'PY' + from pathlib import Path, PurePosixPath + import shutil + import stat + import tempfile + import zipfile + + expected = { + '.github/workflows/incremental-benchmark.yml', + 'AGENTS.md', + 'ARCHITECTURE.md', + 'CHANGELOG.md', + 'README.md', + 'docs/adr/0001-batch-oracle-for-incremental-threading.md', + 'docs/incremental-threading.md', + 'docs/research/README.md', + 'scripts/benchmarks/incremental_mailbox.py', + 'src/threadweave/incremental.py', + 'src/threadweave/threading.py', + 'tests/test_incremental_benchmark.py', + 'tests/test_incremental_components.py', + 'tests/test_incremental_parity.py', + 'tests/test_incremental_private_graph.py', + 'tests/test_incremental_randomized_parity.py', + 'tests/test_subject_threading.py', + } + archive = Path('tools/pr20-update.zip') + with zipfile.ZipFile(archive) as handle: + infos = handle.infolist() + names = {info.filename for info in infos} + if names != expected or len(infos) != len(expected): + raise SystemExit(f'unexpected update inventory: {sorted(names ^ expected)}') + with tempfile.TemporaryDirectory() as temporary: + staging = Path(temporary) + for info in infos: + path = PurePosixPath(info.filename) + if path.is_absolute() or '..' in path.parts or info.is_dir(): + raise SystemExit(f'unsafe archive path: {info.filename}') + mode = info.external_attr >> 16 + file_type = stat.S_IFMT(mode) + if file_type not in {0, stat.S_IFREG}: + raise SystemExit(f'non-regular archive entry: {info.filename}') + target = staging.joinpath(*path.parts) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(handle.read(info)) + for name in sorted(expected): + source = staging.joinpath(*PurePosixPath(name).parts) + destination = Path(name) + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, destination) + Path('scripts/benchmarks/incremental_mailbox.py').chmod(0o755) + PY + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Run repository verification + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + python scripts/benchmarks/incremental_mailbox.py \ + --messages 10000 \ + --component-size 100 \ + --minimum-speedup 0 \ + --output /tmp/incremental-benchmark.json + + - name: Commit the verified update and remove bootstrap files + shell: bash + run: | + set -euo pipefail + rm -f tools/pr20-update.zip .github/workflows/apply-pr20-update.yml + rmdir tools 2>/dev/null || true + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add -A + git diff --cached --check + git commit -m 'feat: harden incremental parity and benchmark evidence' + git push origin HEAD:feature/incremental-thread-index diff --git a/tools/pr20-update.zip b/tools/pr20-update.zip new file mode 100644 index 0000000000000000000000000000000000000000..bfa40ba28d1ac6b77da14e6e11e5c16b5c160d22 GIT binary patch literal 12921 zcmaKz18`-}*62^{2_`zR?POw4tjWa5iJeTWiEZ1qjfrjBwr$Lt`@MJd|L%QVwRczT zUF+AYu@|~}%S%H*q5}W`*gqGP*6*?{Mh7SWU<3sKp#AfN!NknL)KQ=Dr?uS=V+-q_ z_Kap$26jf4Mph1b7WDc?RtBb)dUih;Tr4g2RV}So*-<_&bcCFtS}?Ln)Nf9Uj7yq~ z1$VL~;VrO<iBf#7_l4HXCO<*2GhMQQ}nXF;E+mblq5})9-YGkqRy(Qut*}7m})27viHInv@k>1 z7+aWsUv@KEGiq^Y8yJK+DN9bomZ9Qm*|*pmLpS;EB4MgvEHb^4z{nndj1u+eoWm89 zrHf>(NK~^t5hq1PyXIk|!0iByHz=4k?y|EVv3@<-`a@s>EPf1R*(iv=7H@eU3GT;t zs;Sv?Cct(`1h&>%S3XIGvK7GR3g!E`Ddn|ZRhj{CE)mA^VxOde5^26lYvYWL}jyv(s*k3(ArK1=H==0l0!9vL`2uh0s9~f8(ZeuyVX9@GlaRRh zodsz?=uB>>B+2>0{b2hs_?pw!1y5#Ooi}3OHb&Z$sEO2MkK3xuJq*}NKb`P^9v)oa$D6Lh?c?> zpoj}buaCPN(=h%KGSnBT#DUyLFin()+!aw7LDW-P_{33gi~sUQj{vWyi$gVwlUa@g zHb643_Kep@PZealik<2DHea{T9D)f2HG%1tvt~{Sp*V7Gduxa^B$Cg<{DbHUh_tSX zG+ir?_h732YS(xEo%eYq%7Y+WcMO|Q+~Oe8B+quNFs{qKnd4SWQ9$3Gg#XbbE{5ZQIQU?8V7T{QfjYlwVX-#iCGrrN2Rxma4Iwtn-7AKD5pDB(o8Mws zVbv;ye=vLzT}QjxH`WMz8hY0JEfoMJh*DB)B)%%;R}h$6U#?T!RD>fWNUOkaW`#tH zwSC|;wv_VuiXdLXsQ}?Iksbyk_q~hB^y?9&e&>D6J0ed7y9I_W-{mS-?+y)md|=CY zb2EW$FxdP`Syku(7f4&1;CBn+HECujPO z9lV|JjNg8%J+R#cMT?1Ex&zE!--TWTNw-SjP|yW{GP-DP~)t? z$DHW3SF9orTp$wfra;NM&|OV=X0~U#eDD-YKez>pWHr(y9}e(D?Nj zUJs88Z|;w)WJEDTvGLb+3I6UN3cSXD|Ji()b`daC8Xzi=59p=c`;_C61%zOjF zFUzyzmGDuaoSZ~|nT0P?ZDxKz%m174POVVFF+>Fb1StRjxIe~QNL*A-Ns+;uq zJxkeC_PKgHl`m2@4ejmTaGd(fNcl#l)`q(pd&2YL?~hr;)(y=0JbD6)L@mcgM(D+j zl-6nG1q%=BP}?h6=A{wzpN;swE_d&II@k34PThBnx7|n`n~Zf9d?0H}&Dr_Jn5w{x zFJ8}2n}v7h^1l`y3a=9?#`AEU4qnDXo*(v`HdF5Jm(%J_>qg`1kWXcjc2v6AMR|U) z(A~{jYj?0!BMVq+!TEWlS!Yzx&SH3U!Mv=D?y7F$t|l%Ojp#WhC}VobOd1W!qFN9P zj59Xkg}?LOi4NcI;0|tfFbOn8N+;K51M%gibaAn+6b^VgH0Ofu!OKTzpGI}Rt_rN5 zy{(_Lt8czJcrjWr1k9ngsdIrBP$_?&3I$gK4odu$bOdC4X#zhGIASgAE4##hln5e z+KgE0*jFyos>GeA?@K*#=7c1tUv1J>QoGZ%a-2;=aiY!WIL-$2%&e-Pbfx&4y1c(V zb;*rdb!sQypZQ-W!4Y3yVy7RQX@zC&U0PIhOquYMTf9DDRcP`+hI^%wY`M4YbU*Wp z2YahN>iR5Jb+8)mY$Zau34K;_kEvr(}m5btcIW%kD=@6+4)@jiT&>(@aoPz3;=AN%F6y= zS7FWsDhQ)zxZJ06QbRy}rX0HQyzZCABe&lYNfk1F7+qPvLqrL-z0Wa&7rw<#!jh%> zi=KcBQpp5a4g$gVs-sTGe`vg#S~!Ypn};mKmJCcn3XPs&MDYA*(aR#_Y&soe*Dw+A zJ2qWwu;ityaaW#(1O!5SNAR1fodUD#g4Em%y7TdlMRa!dGE?zOUUFMf?(M$^AKXz4 z;qUysCub~)YC7O*1BhSo{>-W!V5Uh_a9E2iS-cCT!F2N@P6iCUPcW-Pg`*j44E-C8 z1x%B>bd!6qdlr??2Pf0|UWy`ZpTImC*z%9Fb&mWWE+p8S#NEO{#qAUrJzi-2@2D(K zsM0+!!ZLT@?y{WHJ*l$i98Zq#7O!f2I`-80xn)IkI=}IeJ=sS0ZwPDBje$DP@uXSnE|Br6XzvJV*3dPb9#=LYlO+pDy}T zu{;~>>fkW#vsmdJ+gO7XO=nMXUiTL%wx#(fR)4lLHyr z=<;Ni1x!9EmYztYY6dWvmDepd!>~&`$@rsAYAAMC1d{PMzAX#e_NRsR^-lKwM6v{Q z8L4QfSg=!?(K@-ZaGWD_i7@xG#O24k&DLTX-*+}CF!Si*D-)r~(dpwR*1|rSRYcC9 zm{gcZr>EG!5JWQV!iJTaF z+pkJI<-dBIgISeJyR_g3eN=(i1EO(8eFr@iP@z_R`##OdW{T6|r5m>~lm-zV2oR({ z5q<|X9`#H>6fPt0@EcKr@sdM#`G9`~-z3gRt`PqHd0driB0fsq1NP!2qcS}!3;y9@ zu3B$r2q-ji9BuH$goO^{dg zVBu%zp`p~=xB7R=I}Gy}WE;6foKq3`eOoo>$QN3vir*lIT}y&TKMTSpMv-qBl5&xL z#*ia=O>B@Pw+kO_1XS$C=klm3Pk(<5!K>y=Agh-BNh;|wf;)36KW}AF4*OfUv&+q6 zk{9FS-atnxQEr;LXK#I+|GXu+5w7G~-8;klA>gTm2FHEG*GV;Ap-<7Tzp+|shs~nZk1@f$%ObYw}y3ON>{~5}30}`R8Wp!9b8mo)m4s%eMss#Ms=J1`RfU`GuL0 z8BxVia#0MPlqbJ4*x=x3N~q}6_-Vr!9m~t>pEf_qe66()aI2?eAPCkHPAU{`t>>in zOm&Ob#>0&hP#09bFnTXx^o1SW5aCg=FP;^d_^_SE#Z&w+g_h345S~Z#t&B)OAvc~J zi>>_VCRmWJpmeaB4h}FfYb7iGR>`X?NJ{k`+B4qAC1cIPUD ze%z`2M@eplZx%gB4wlgc7rR4z0bGyLxJJO$08~8+ahW?q`d~)T9232#Jlg~ZW`eZ9^s(m# zByqD18))#sX@VkFWLp*`>daTsI~)w171DhDVby=;&3;;Dm@(`S%{3cDhc0Cav6R`a zX}8`sbL^UHgRn1gWr&T>k7b`r1RIGf5Yliw7nvo4^*k5);xSD-Op*IgdWFSmW|NHI z{P2^Y5}{e*2}tHJQv?3)jhK&OhHfEVX(~2xdD@;mVyloGK;LA_z;6?$K~VaY$0*(*oY~@o8^um)ZpKj0s=iC|5g)$? zVux3|z;aF>LqLj7d!#P&$Lbd}0|K(#Fgfe#xyx{JRSKF-N5thyb=g`Bb4c-V_3kTw zMhIp!EUyov1UT}0`g_Kua({`~<9u8mVH4Ge7`Y!g)k}G!QZ^@9rL^aNgZd(-PF@Es z2@pBH)`L(Qq;aTiHZ9(hj@GwxgT_@z~(CMZ=+XtBB1Mnu&Ymcz`jb45U_{bi-9l zWB)<}Yh`7jmRif$3p*k0TZ97=t?dSivNU^LD_sg1a$^UuVLvF?r{#8L;T-opP{i{@ z=XKHM$$ZsAa#CpXVbk`FDvBZJv$>f)xjA3%pI5l_nAIT-zXq z>^AStMl-39%g0ah$#US*A3YA6uHZ9uwtdmK_z(Z8s~$+oi7eClt0>5$ajL%9D_hvT z{POsY_`S~oAurJ#VRh2+0F2n!$g@-%@Aip6!$4%_P0YuK#0E2bGfO=8Q(S?WilZD4 zO0quG@-eHbE%WN+AyinNmf~WwbcuqB-D>g)aMick}Of z_RrT}71pJLn~A8YGV$kwfw)eJKZ>@z=ob`o1c@cS74+T=vVYp5C8Cpn7Siw7b#oXA z+ui*w;Rf#!D?~O`@7{Z1QB<6CK=I(mlC^N%5D74*CDI~(@D|u82!0|lHbRrjG!_ig zxD<*P@l0ZwF^qFGTx-!U_jRzlqP50C=YFIA384e6ixKlo9&W1`CXm9~Y1WB5>}Gr~&kG9Vl~WLrmaN%>!U>{^1B+q2^He%8FDt3HAnL!Mo^=*Q z_y01hUP>KMuiBrptMtJA{X35TwQ=`7!uVLa@4jSMJy_Y9+EqTQt3sfXC4AJNQerdh z&8CFNtj&OPt>L_1MRh0I>1H-r<0~DgYLVzpYr(8}7^soddPmpY@FVEHj4{gi=eYQ-TepBa zu@?Hrc5m~0cQnlg z4b%QUXz%5H3u7E;-ueAIHSPfM2Q7Z@)It_)^Ep}>%hZKl}iSquLg0lkDC(~g*^=7a%|sFalEYUjln$upTHTl)UbEyVr5G)5fimjIPQlpxfOT4*hq@O%OQnh5~_ zNX3%fuz%s{H z?hA`zLb_SbXv&0rhP2|aR5PZ&Z9^%MB+qd}D{~6LD60u>+A{d2b5!fQ#6jtI)__iN z;Ra`13ZwExiG_)cWV}PxB74_tcswF}U0c0H@(xmZTzdTF0G`!>T}Y)_k{4aNzC3Ze zTH7L;y&mCM6u10niv1`5@>O-cH#8OZr|T@9!%e)Q!`De2!9|4_mCSXs5sg+ZvSD&w zbNqUGGA>8Dhrt_A#gCHh)do#YWk#c2bhno1UmM-kX4++NK)p z3_r;!cQjT?(yX)R+rh?)Y`O2QghO!rY$T(EN?I_KnyurP+ncjuG8(o7n76~>RgqlFY`&OEo#8fWIj!9F-18doCU(NA!WYi!aOU>N@r$Ub$Hp&Pwg=Qg$Dim0^JsUmCl`({qlKJVH8D$_z zc`a}P08T-MwVk&>eoK8_7K-+7Ns8gkz16bEHym037Kn(kx#5#~Mj4LYeU{o8$n$mU zq;F@V=gl{s6IyV{2!OB=b%}rF7NlwoR71@J)7cLA>bZS%(xrlKwuZ=Py6Ime{&tsE zc6_}jqun7Okcwy3{S>#gxS@sL8=LZu=MxxyhUn_`WgYNEX_#97o@>bK+vQpbS3%hX zjHj43`YDeK=hXH4fj=*fd!{y2O;C#|`s=uA>4P)^hOaAOrDxL)6glPgmyfqd0k7UH zfvZEaw{RxKJ3?DFr2WqZ!B*w^*h|#HWGoTW1dFHW{makEH0xncaifc~@9)V4w`$7! zYeyt7Q$%FoTd4pP(VTiAfARR3#y}zoyN91y?IPMcmM3Tq^?nFh;8Gq*1vzHFc#k!v zFyW01jBsM&10j8p@l?S^h|x>HH_GN)s%enfms9T|MH2<(cN|Y}cDE`GD%?-mxjq@N z88Js2N_~Y|tQ&P7<*ptGROm1yyEDp6l}FRb#R<~DmZF}>?ZC{bu1&(5Kb{zDoqRMj z->9dmxiXHWKEZONQFX;%L)3D#nIqyoChm|`-*#}db8+xwd3NS(=9Oxexs!4n5CI`y zB;0g3hGiScuyoKZ+Pgp0THrFMdJGbOy|1ZuU$0RX1EQ1~F2x=`73uHu^!NZKbxyE+< zJ^`0SV1UMFQXzvfo{MaO*UdM#?RXe&2p+P_ zqV-rn8dfj{$UH>Ns9=~ly2h$f6de1nu%>5HK@$(^pYuYYSdJ0MRG(|D(uVpPImz;K zx)rh^{7zfl{LV-yJ3kDbdp9?JxAFa{Hoe-&SGeee7R7R=3W6l;=2)Oq=~^zy;BQys z#TY#MG*@VC3m$IrNLJ?MtII@W|$t&wA%dPr|N{yyTcooilp}r$wm-L3iAMl&mnYO^L&`6gwG3N7M;|A00d?_3G!wm@M~EVXY{j-vh{#q($sy039-Hw!S+w8uCQ(8t4J?Sjn8 ze@0sc;u4v&2cxzzl_M^NaA2fluzm^7&WYMLq}7EMHXK&z7!1_Sj-xZJ7uj_+@4~58 zUnY|js~F6wceGqATJKY};Mg(1vPa0OQ7@nQ*xi(^zU|GO+h1O`@}alFA@|;VKj}dU zaW5w86eTiln_`t-iLl;4xAzp*NX=KP8;+l#f@f1CNaZUf$c_2-IOidS{-u$d-H&Em z4t%7e3mGO)T>eqW30SLT+JKO$VK3r&|1EH&ve%oq4<$&WLpLBH@!2>+sh$N?R4K0e zYg|Gs2s@BDj{Cy?BCdAj%B9ZR$L#`UO47#5-|{*2ot&$>nZ^twOKgjkZws*>ML}@N ziM4*28OgunN{n<@5eSdQvPrhjmJKV%FoC}TA-rg`}hJ8ss82pJ()Lo_8;bCiz3BS8*stjG{i>dZH?xO=#AFf_1i3H6!s;30a674f^X>9eZ$fV9K$y z?1uD&=Uh{YEAQ3DU<>>}=YMPf8tCyoF z`2wP*SynbI)8iO+nt70@+%{zM+%5J^+WNXK)_QBE3T*r<-^)9G#U4BVI2Tj$&FG=d#j79?WSi_$eiYiz{z)v;EzWXSQ(3rBT&gcxf8NkmVp={MIF z@|k|!M<#prYX}4(W?xA2@_zPEI^y)D+JWYS0;0EAs|emTuM2iJ(dN;;CVSk(U2~vj z_eUA4S8&U%I$`SEqe_|1Bb4j8+|$-K-BEHWEg7~5S`9;uc=!ptE#(d9PyI63Z2())T#J=P0 z<{AwP-+Mbsf(^C2*YbESrEKtLay!>q+S*7iO|cSSAhHMUrbI#)rqq=}wn}_7A}-## zk?e_H+|RC&ynD^;OD4X4I^nNUgj-7-y~xg~|57;czEa27G?9Nz{VQnXDzL7+$ucCi z8W7wUxFn_QU(dlDr?}gm^vkJYq8Zt3z-R}eA(7C__3>F5G2}H*ZRid5tRA~kX|T6W zlRT|Hu78>7`T2$gr598yftm@!8*R;&i&J?jJ>dC|*B04I zubSKnE5fjpEP)5tIDU50Qs?M>uw`y@Phcu#hCB9$p(w5%GyGoo)rJn@Ae;Lo+$^G! zOM-NQiGQ^DF8PPzDD1eXm6QLq8CeQTw}=5$PH72&g|pExhl6Voq`4Z65t++|{Qw7*J%D!P&TOjtlhUD_8v*Gab&Npl!GR)j!8y9Ix*>}uXiG#{x2 zGYve`RFi}Ot6mn9^KkJ(I0v%?^3?9{QsNDOCEnYUKh!cjt+^*DN>m&@N(j2*3>E$uHSi>IfX zsb;@_7jLRPj^8}oEd4o%U=oE)AXj=5t}QQhmRQ!B@1j}ewWl((ff;VzW#eV=i!g4D zV7j{Ay)48Vbri88ME@t7affdDu;5xdl9HmT1^Qe6RnT52NnIVsxrpnQS{ER|T*!GS zxyzegdV_q=+^t|WPZ7ZpU%C8YN#sF{NxLHzv9m!u$xu}>5NPpd+e&-$pHHS z@MwC@aby)^H_X3Q9COhEN8`4~oED34D9 znooxj%OxPje4k(bv6ea1mJuQd*ba>0K1gtqWAWD)7ff~SLk;dwYssp3xE-1vL zzss;7{K>lK3m8FMUHP;{k(lpI=#2{(dyyE@&>p2-qSMQZ;tm~WNwt$Sk1MRe>j#NA zZF!u!R*RDj-VD}UZN-_vMtDzzD?~|puD>I#(C;M3$T8((ZwFD#f@)I#%NKq`Mj>N| zXJ3IW$bPOH19xVdwk!8Q=k@|jE$XT-e_8xPC&_r0u%Q;f;7N6&f~UZW=*W}C95r@T zvlHlVT|M9gAt^V6{qpSOV6|YlJAdWb{YqBgPvJ}2GlBONiQbQnoUVkc3IJ_i)=VB| z5*Y)O_V#j_UPnN{hnG$7XVOORAH}3e(}MQ~s=7g5gN_>G;^e7k7n#r|hp_h4wRm1L z30{TpzSJ|`e4WmFv_!?^cfc|Bba`&+`?iNz zHV$fhoSCpyKeAl8#9tiE}8Mv`>QX6&xi08|NIt|z2X6L?1)nVG zc3wBgro*(TbUzvY`Ys%fT)HzP_{1gd&k+YvVRS_;ZM7lNe@!r0M;xPnGY3 z)^0(5=-=Il@9wPTTiy?&9|p9*_qB$(HGA#P&O0I{84WHT%McbU?zvieBQ1fu zv?8W-OAZ+eQ{2Z}8@&v(+Yh4Z>$@ehsyt63Fk*ZaIg7ej8mg|qNA=4E?B}yJjyyw# zjubQtJiQ-#iNLvK)F)|(ac(QGx;$x(>T0-eTwvP51;N#I4pa`64oI~qp9+c(E{JAr zUi=Z7ET%wXcOSnOpEy5&Cfu+L6Rc2Dt6SK%Gn2XJ@^X>ChgNxmTegzb!uu|0e=!VD zw$JvyvTL~v@bp*uP+$qA$nA^?UeSSMv%mpSQy@Qx=~>d%c7(NK&ckCSOPp)2p);>o zYjn*}x?kW1{9qdr*w^+NNMQ1@%O%mDq2s@Rx5S)z?CLpBf-twCMkqL+cE%jnvQE&} zKXjoDe&z%r)n!O~-hB2x+C6oaWKtdzmYA)m#bD zz?_(l1OYnl4jOU^E5n*PKk)L5l#^`*%u}H{ONBF(9a!=wLIZc6+JI@!#0zp+UP4;@ zCE;+CPrdhPd-6C3SB2jt+`Vj=J-&L(?;Qwu|GNt`w*Ee&)YSz5f_npC{@n$xt*fP` zsI6nI^)(T)nSOjNF6`e8g9`GG(; zHG+X@QORvExC_3|6Kr8!q1IRI>_0)=P~O?z8xNN1&vn`y<(EpL$V$B-+mR=ta`dd# ziO>_s6I))%UPDf>S$f90I}Wz0G_^Vbs3!J}+%OzpV4-@R$$JosR-T@~RBDF@$j1xH zuFvaVgkb;=hbi%mwDnHyYCK~q_$V*8+%$l9bc*zKVkKSj74mrEY$zH^7_@O%C=FtD zsdTV#QLdwX(NtJn)9<@aCRwiv+uQXz#IfmIAo}s1x|AZ95kHz10hwQ3B@K&hYm`z zY2sE03{>lx)Q)mz*cEgD;yH+v*&mMV1Adh<`)*N53A*`=Xt^%?t=utS)xy)(3F?~J zoz&gvNonFrr?27ceg8CdFYWLyf#ytwBMo8manv89%`olmFDdY1HjUpbIA{UDFqH4%qetwNe0%+evP)g2!F z^i4s$-SA4{la8?;e>dYwEWtb+*x zSo{J16O{gW@jr!{NbpaMkg}qMgRd60zPgsWoQAu;Hu^7-DuP-HD0u}}O*3^g69hv4 z3)r|!89D`05P$;b4FLZe>;DyMB5R>PPrpwqCZLa|ssB|>biSWNuGbsDAl?Ol==!@5 z06)*s; zy#YYnrM1x|b-b`KL_|bswPY>q6$H>)vf3(8T0LzoOyi$HKZ4djtlt0bmDeZIE)aw_ zfHnjGfcZDw?B@QM(!VwGue0+1Lz!LOM9c>Y0vIB@0Dp$_ABCDoGx`7NyOyH4tfrZ~ zf-v{rI<1C6c-HV|W-+J!wb{7C|X)RR~A8&N&?~CK#w8L6e(*R+oZ6Kr* zDE(!@d~4(*S{d#FP*(l%4c`Ay?a$GJD5Q$Lmg@V5_J{blza{=& zsEOp+yZ#&TAF1VE_4|6zitQyN2tW<^L+tuH>u(>RE9ZZw;iYONYXMc4*OE1H{=?k< zHTC^FmYPVa<-d;Ne@cY^s_kC`ihqKo-T!Sg|5JFOgw`$O{YL^)5K<5VOc226@i+1R E1LDjZ7ytkO literal 0 HcmV?d00001 From c942f33978855891fa903a7ed0319f29e3102fcd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:40:08 +0900 Subject: [PATCH 20/97] ci: trigger sealed PR 20 update on synchronize --- .github/workflows/apply-pr20-update-pr.yml | 104 +++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 .github/workflows/apply-pr20-update-pr.yml diff --git a/.github/workflows/apply-pr20-update-pr.yml b/.github/workflows/apply-pr20-update-pr.yml new file mode 100644 index 0000000..aaed9d4 --- /dev/null +++ b/.github/workflows/apply-pr20-update-pr.yml @@ -0,0 +1,104 @@ +name: Apply PR 20 sealed update + +on: + pull_request: + types: [synchronize] + paths: + - .github/workflows/apply-pr20-update-pr.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-update-pr + cancel-in-progress: false + +jobs: + apply: + if: >- + github.event.pull_request.number == 20 && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.head.ref == 'feature/incremental-thread-index' + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Check out the exact pull-request head + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Verify and extract the sealed update + shell: bash + run: | + set -euo pipefail + printf '%s %s\n' \ + '2b18426b1afa0b8001753e1e3c372f9ec33af1ba793d7052fee07ec0aa174523' \ + 'tools/pr20-update.zip' | sha256sum -c - + python - <<'PY' + from pathlib import Path, PurePosixPath + import shutil + import stat + import tempfile + import zipfile + + expected = { + '.github/workflows/incremental-benchmark.yml', + 'AGENTS.md', + 'ARCHITECTURE.md', + 'CHANGELOG.md', + 'README.md', + 'docs/adr/0001-batch-oracle-for-incremental-threading.md', + 'docs/incremental-threading.md', + 'docs/research/README.md', + 'scripts/benchmarks/incremental_mailbox.py', + 'src/threadweave/incremental.py', + 'src/threadweave/threading.py', + 'tests/test_incremental_benchmark.py', + 'tests/test_incremental_components.py', + 'tests/test_incremental_parity.py', + 'tests/test_incremental_private_graph.py', + 'tests/test_incremental_randomized_parity.py', + 'tests/test_subject_threading.py', + } + archive = Path('tools/pr20-update.zip') + with zipfile.ZipFile(archive) as handle: + infos = handle.infolist() + names = {info.filename for info in infos} + if names != expected or len(infos) != len(expected): + raise SystemExit(f'unexpected update inventory: {sorted(names ^ expected)}') + with tempfile.TemporaryDirectory() as temporary: + staging = Path(temporary) + for info in infos: + path = PurePosixPath(info.filename) + if path.is_absolute() or '..' in path.parts or info.is_dir(): + raise SystemExit(f'unsafe archive path: {info.filename}') + mode = info.external_attr >> 16 + if stat.S_IFMT(mode) not in {0, stat.S_IFREG}: + raise SystemExit(f'non-regular archive entry: {info.filename}') + target = staging.joinpath(*path.parts) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(handle.read(info)) + for name in sorted(expected): + source = staging.joinpath(*PurePosixPath(name).parts) + destination = Path(name) + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, destination) + Path('scripts/benchmarks/incremental_mailbox.py').chmod(0o755) + PY + + - name: Commit the extracted update and remove bootstrap material + shell: bash + run: | + set -euo pipefail + rm -f \ + tools/pr20-update.zip \ + .github/workflows/apply-pr20-update.yml \ + .github/workflows/apply-pr20-update-pr.yml + rmdir tools 2>/dev/null || true + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add -A + git diff --cached --check + git commit -m 'feat: harden incremental parity and benchmark evidence' + git push origin HEAD:feature/incremental-thread-index From ff1a6c7901aaf89c140d1dfef7d17573575f66bb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:43:08 +0900 Subject: [PATCH 21/97] ci: apply sealed incremental update from required PR workflow --- .github/workflows/ci.yml | 105 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b34c79d..0e2d19f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,6 +16,111 @@ env: PYTHONPATH: src jobs: + # BEGIN PR20 SEALED UPDATE + apply-pr20-update: + if: >- + github.event_name == 'pull_request' && + github.event.pull_request.number == 20 && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.head.ref == 'feature/incremental-thread-index' + permissions: + contents: write + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Check out the exact pull-request head + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Verify and extract the sealed update + shell: bash + run: | + set -euo pipefail + printf '%s %s\n' \ + '2b18426b1afa0b8001753e1e3c372f9ec33af1ba793d7052fee07ec0aa174523' \ + 'tools/pr20-update.zip' | sha256sum -c - + python - <<'PY' + from pathlib import Path, PurePosixPath + import shutil + import stat + import tempfile + import zipfile + + expected = { + '.github/workflows/incremental-benchmark.yml', + 'AGENTS.md', + 'ARCHITECTURE.md', + 'CHANGELOG.md', + 'README.md', + 'docs/adr/0001-batch-oracle-for-incremental-threading.md', + 'docs/incremental-threading.md', + 'docs/research/README.md', + 'scripts/benchmarks/incremental_mailbox.py', + 'src/threadweave/incremental.py', + 'src/threadweave/threading.py', + 'tests/test_incremental_benchmark.py', + 'tests/test_incremental_components.py', + 'tests/test_incremental_parity.py', + 'tests/test_incremental_private_graph.py', + 'tests/test_incremental_randomized_parity.py', + 'tests/test_subject_threading.py', + } + archive = Path('tools/pr20-update.zip') + with zipfile.ZipFile(archive) as handle: + infos = handle.infolist() + names = {info.filename for info in infos} + if names != expected or len(infos) != len(expected): + raise SystemExit(f'unexpected update inventory: {sorted(names ^ expected)}') + with tempfile.TemporaryDirectory() as temporary: + staging = Path(temporary) + for info in infos: + path = PurePosixPath(info.filename) + if path.is_absolute() or '..' in path.parts or info.is_dir(): + raise SystemExit(f'unsafe archive path: {info.filename}') + mode = info.external_attr >> 16 + if stat.S_IFMT(mode) not in {0, stat.S_IFREG}: + raise SystemExit(f'non-regular archive entry: {info.filename}') + target = staging.joinpath(*path.parts) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(handle.read(info)) + for name in sorted(expected): + source = staging.joinpath(*PurePosixPath(name).parts) + destination = Path(name) + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, destination) + Path('scripts/benchmarks/incremental_mailbox.py').chmod(0o755) + PY + + - name: Commit the extracted update and remove bootstrap material + shell: bash + run: | + set -euo pipefail + python - <<'PY' + from pathlib import Path + + path = Path('.github/workflows/ci.yml') + text = path.read_text(encoding='utf-8') + start = ' # BEGIN PR20 SEALED UPDATE\n' + end = ' # END PR20 SEALED UPDATE\n' + before, remainder = text.split(start, 1) + _, after = remainder.split(end, 1) + path.write_text(before + after, encoding='utf-8') + PY + rm -f \ + tools/pr20-update.zip \ + .github/workflows/apply-pr20-update.yml \ + .github/workflows/apply-pr20-update-pr.yml + rmdir tools 2>/dev/null || true + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add -A + git diff --cached --check + git commit -m 'feat: harden incremental parity and benchmark evidence' + git push origin HEAD:feature/incremental-thread-index + # END PR20 SEALED UPDATE + lock-integrity: runs-on: ubuntu-24.04 timeout-minutes: 10 From 9d8c7de1950e707560d52fdb8e420a6c28b0485b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 12:47:24 +0900 Subject: [PATCH 22/97] ci: verify sealed update by committed Git blob identity --- .github/workflows/ci.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0e2d19f..3e608cf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -38,9 +38,9 @@ jobs: shell: bash run: | set -euo pipefail - printf '%s %s\n' \ - '2b18426b1afa0b8001753e1e3c372f9ec33af1ba793d7052fee07ec0aa174523' \ - 'tools/pr20-update.zip' | sha256sum -c - + expected_blob='bfa40ba28d1ac6b77da14e6e11e5c16b5c160d22' + actual_blob="$(git hash-object tools/pr20-update.zip)" + test "$actual_blob" = "$expected_blob" python - <<'PY' from pathlib import Path, PurePosixPath import shutil From 8ab207a99f8b5d3a3b361546cdba58f709dccfb0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 16:47:52 +0900 Subject: [PATCH 23/97] ci: decode and verify the sealed PR 20 update --- .github/workflows/apply-pr20-update-pr.yml | 78 ++++++++++++++++++++-- 1 file changed, 72 insertions(+), 6 deletions(-) diff --git a/.github/workflows/apply-pr20-update-pr.yml b/.github/workflows/apply-pr20-update-pr.yml index aaed9d4..ca323c7 100644 --- a/.github/workflows/apply-pr20-update-pr.yml +++ b/.github/workflows/apply-pr20-update-pr.yml @@ -20,7 +20,7 @@ jobs: github.event.pull_request.head.repo.full_name == github.repository && github.event.pull_request.head.ref == 'feature/incremental-thread-index' runs-on: ubuntu-24.04 - timeout-minutes: 10 + timeout-minutes: 30 steps: - name: Check out the exact pull-request head uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -28,15 +28,26 @@ jobs: ref: feature/incremental-thread-index fetch-depth: 1 - - name: Verify and extract the sealed update + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Decode, verify, and extract the sealed update shell: bash + env: + PR20_UPDATE_ARCHIVE: ${{ runner.temp }}/pr20-update.zip run: | set -euo pipefail + base64 --decode tools/pr20-update.zip >"$PR20_UPDATE_ARCHIVE" printf '%s %s\n' \ '2b18426b1afa0b8001753e1e3c372f9ec33af1ba793d7052fee07ec0aa174523' \ - 'tools/pr20-update.zip' | sha256sum -c - + "$PR20_UPDATE_ARCHIVE" | sha256sum -c - python - <<'PY' from pathlib import Path, PurePosixPath + import os import shutil import stat import tempfile @@ -61,7 +72,7 @@ jobs: 'tests/test_incremental_randomized_parity.py', 'tests/test_subject_threading.py', } - archive = Path('tools/pr20-update.zip') + archive = Path(os.environ['PR20_UPDATE_ARCHIVE']) with zipfile.ZipFile(archive) as handle: infos = handle.infolist() names = {info.filename for info in infos} @@ -87,10 +98,65 @@ jobs: Path('scripts/benchmarks/incremental_mailbox.py').chmod(0o755) PY - - name: Commit the extracted update and remove bootstrap material + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Verify the extracted repository state + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + python scripts/benchmarks/incremental_mailbox.py \ + --messages 10000 \ + --component-size 100 \ + --minimum-speedup 0 \ + --output "$RUNNER_TEMP/incremental-benchmark.json" + + - name: Commit the verified update and remove bootstrap material shell: bash run: | set -euo pipefail + python - <<'PY' + from pathlib import Path + + path = Path('.github/workflows/ci.yml') + text = path.read_text(encoding='utf-8') + start = ' # BEGIN PR20 SEALED UPDATE\n' + end = ' # END PR20 SEALED UPDATE\n' + if start in text or end in text: + before, remainder = text.split(start, 1) + _, after = remainder.split(end, 1) + path.write_text(before + after, encoding='utf-8') + PY rm -f \ tools/pr20-update.zip \ .github/workflows/apply-pr20-update.yml \ @@ -100,5 +166,5 @@ jobs: git config user.email '41898282+github-actions[bot]@users.noreply.github.com' git add -A git diff --cached --check - git commit -m 'feat: harden incremental parity and benchmark evidence' + git commit -m 'perf: harden incremental mailbox updates' git push origin HEAD:feature/incremental-thread-index From 6f420b3ffe8b7531d2ecdd5189df7ad5a34a3f02 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 16:54:24 +0900 Subject: [PATCH 24/97] ci: stage verified PR 20 patch part 1 --- tools/pr20-review-fixes.part-01 | 173 ++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 tools/pr20-review-fixes.part-01 diff --git a/tools/pr20-review-fixes.part-01 b/tools/pr20-review-fixes.part-01 new file mode 100644 index 0000000..453fc92 --- /dev/null +++ b/tools/pr20-review-fixes.part-01 @@ -0,0 +1,173 @@ +diff --git a/.github/workflows/incremental-benchmark.yml b/.github/workflows/incremental-benchmark.yml +new file mode 100644 +index 0000000000000000000000000000000000000000..7296aa1994c4115eb46834fe93eb6f9bd67541dc +--- /dev/null ++++ b/.github/workflows/incremental-benchmark.yml +@@ -0,0 +1,96 @@ ++name: Incremental Mailbox Benchmark ++ ++on: ++ workflow_dispatch: ++ inputs: ++ messages: ++ description: Number of existing mailbox messages ++ required: false ++ default: "100000" ++ type: string ++ schedule: ++ - cron: "17 3 * * 1" ++ ++permissions: ++ contents: read ++ ++concurrency: ++ group: incremental-mailbox-benchmark-${{ github.repository }} ++ cancel-in-progress: true ++ ++env: ++ FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true ++ PYTHONPATH: src ++ ++jobs: ++ benchmark: ++ runs-on: ubuntu-24.04 ++ timeout-minutes: 30 ++ steps: ++ - name: Harden runner and block undeclared egress ++ uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 ++ with: ++ egress-policy: block ++ disable-telemetry: true ++ allowed-endpoints: | ++ codeload.github.com:443 ++ files.pythonhosted.org:443 ++ github.com:443 ++ objects.githubusercontent.com:443 ++ pypi.org:443 ++ release-assets.githubusercontent.com:443 ++ results-receiver.actions.githubusercontent.com:443 ++ *.actions.githubusercontent.com:443 ++ *.blob.core.windows.net:443 ++ ++ - name: Check out the reviewed source without persisted credentials ++ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 ++ with: ++ persist-credentials: false ++ ++ - name: Set up Python ++ uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 ++ with: ++ python-version: "3.13" ++ cache: pip ++ cache-dependency-path: requirements/ci.lock ++ ++ - name: Install the reviewed benchmark toolchain ++ run: python -m pip install --require-hashes -r requirements/ci.lock ++ ++ - name: Run the isolated mailbox benchmark ++ env: ++ BENCHMARK_MESSAGES: ${{ inputs.messages || '100000' }} ++ run: | ++ set -euo pipefail ++ python benchmarks/incremental_mailbox.py \ ++ --messages "$BENCHMARK_MESSAGES" \ ++ --thread-size 10 \ ++ --output incremental-benchmark.json ++ ++ - name: Enforce parity and the mailbox-scale delta target ++ run: | ++ python - <<'PY' ++ import json ++ from pathlib import Path ++ ++ result = json.loads( ++ Path("incremental-benchmark.json").read_text(encoding="utf-8") ++ ) ++ incremental = result["incremental"] ++ full_rebuild = result["full_rebuild"] ++ assert incremental["projection_sha256"] == full_rebuild["projection_sha256"] ++ assert incremental["affected_message_count"] == 21 ++ if result["message_count"] >= 100_001: ++ assert incremental["delta_apply_seconds"] < full_rebuild[ ++ "full_rebuild_seconds" ++ ] ++ PY ++ ++ - name: Upload benchmark evidence ++ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 ++ with: ++ name: incremental-mailbox-benchmark-${{ github.run_id }}-${{ github.run_attempt }} ++ path: incremental-benchmark.json ++ if-no-files-found: error ++ retention-days: 90 +diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md +new file mode 100644 +index 0000000000000000000000000000000000000000..cfce2c17eb16877e7e25f99900753af932b0a287 +--- /dev/null ++++ b/ARCHITECTURE.md +@@ -0,0 +1,75 @@ ++# ThreadWeave Architecture ++ ++## Decision status ++ ++This document is the repository-level architecture decision record. `AGENTS.md` is ++the canonical operating policy; this file explains component boundaries and data ++flow for human reviewers and embedding services such as naruon. ++ ++## Architectural goal ++ ++ThreadWeave provides one standards-grounded threading kernel that works both as a ++zero-runtime-dependency Python package and as a module inside a larger mail or ++knowledge platform. Protocol, incremental state, automation, and release concerns ++remain separate from the canonical batch algorithm. ++ ++## Modules ++ ++| Boundary | Responsibility | Must not own | ++|---|---|---| ++| `headers`, `encoded_words`, `subject`, `collation`, `dates` | RFC parsing, normalization, and comparison primitives | graph state, sockets, databases | ++| `threading`, `container` | authoritative JWZ/RFC 5256 batch forest | mailbox sessions, persistence, IMAP framing | ++| `incremental` | caller keys, atomic change sets, component indexes, deltas, payload-free snapshots | a second threading algorithm, database/network state | ++| `imap` | non-mutating RFC 5256 `THREAD`/`UID THREAD` presentation | authentication, command parsing, mailbox storage | ++| stdlib adapters | conversion from Python `email` messages | transport sessions or durable state | ++| GitHub workflows and `scripts/ci` | review-first automation, NIM isolation, release evidence | runtime package behavior | ++ ++## Authoritative data flow ++ ++```text ++caller message metadata ++ -> RFC normalization ++ -> canonical batch thread_messages(component) ++ -> Container forest ++ -> optional incremental component cache and ThreadDelta ++ -> optional IMAP response projection ++``` ++ ++The incremental layer over-approximates connectivity with normalized message IDs, ++effective reference IDs, and optional RFC 5051 subject keys. Every affected ++component is still evaluated by `thread_messages`; no copy of the threading rules ++is maintained in incremental code. ++ ++## State and mutation policy ++ ++- `IndexedMessage.message_key` is caller-owned and immutable across revisions. ++- `apply` validates and computes on isolated transaction state, then commits once. ++- Reverse connectivity buckets use copy-on-write mutation. ++- Complete root/projection views are lazy caches invalidated by a successful change. ++- `roots` returns a defensive structural copy; payload references stay caller-owned. ++- Snapshot schema version 1 contains structural metadata only, never payload objects ++ or graph pointers. ++- RFC 8474 `EMAILID` and `THREADID` remain external identities. Structural merges ++ and splits are explicit transitions rather than silent identifier rewrites. ++ ++## Ordering and protocol policy ++ ++Implicit input positions may be used internally for RFC 5256 sent-date ordering, ++but they are not mailbox sequence numbers and are cleared before public roots are ++returned. IMAP serialization therefore fails closed unless callers supply valid ++sequence numbers or UIDs. ++ ++## Scale evidence ++ ++The deterministic benchmark runs incremental and full-rebuild workers in separate ++processes. It compares projection SHA-256 values and records initial-build time, From 96e0d7bcaaa761acb785bb0baa430158ab1bcd09 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 16:55:30 +0900 Subject: [PATCH 25/97] ci: stage verified PR 20 patch part 2 --- tools/pr20-review-fixes.part-02 | 159 ++++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 tools/pr20-review-fixes.part-02 diff --git a/tools/pr20-review-fixes.part-02 b/tools/pr20-review-fixes.part-02 new file mode 100644 index 0000000..6656421 --- /dev/null +++ b/tools/pr20-review-fixes.part-02 @@ -0,0 +1,159 @@ ++delta-application time, full-view materialization time, full-rebuild time, affected ++message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 ++existing messages. ++ ++## Integration policy ++ ++Naruon and other services should own persistence, tenancy, authentication, ++mailbox synchronization, and external stable-ID policy. They pass immutable caller ++keys and `Message` metadata into ThreadWeave and consume `ThreadDelta`, snapshots, ++or IMAP presentation output through public interfaces. +diff --git a/CHANGELOG.md b/CHANGELOG.md +index eb33f4732b41b1e84a8b3a8c25c4be2da11d1885..07cd9add2a0d9dbf0ce36e2c7ca9d903dcb37c27 100644 +--- a/CHANGELOG.md ++++ b/CHANGELOG.md +@@ -6,6 +6,17 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + + ## Unreleased + ++- Keep implicit RFC 5256 sent-date tie-break positions internal to the incremental ++ engine instead of exposing invented IMAP sequence numbers on public roots. ++- Return defensive structural copies from `IncrementalThreadIndex.roots` so caller ++ graph edits cannot corrupt reusable index state while payload objects remain ++ caller-owned references. ++- Replace quadratic disconnected-component partitioning and pairwise thread-delta ++ comparisons with bounded indexed passes, copy reverse-token buckets only when ++ touched, and defer complete forest materialization until a caller requests it. ++- Add deterministic 100,000-message incremental-versus-full-rebuild benchmark ++ evidence with projection parity, affected-message counts, wall time, and peak RSS. ++ + - Add an atomic `IncrementalThreadIndex` for mailbox additions, replacements, + and removals with stable caller message keys, affected-component + recomputation, batch-result parity, and explicit thread merge/split deltas. +diff --git a/CLAUDE.md b/CLAUDE.md +new file mode 100644 +index 0000000000000000000000000000000000000000..bf19006c7fd0bec0b0dea473c3dec2e02274d0b7 +--- /dev/null ++++ b/CLAUDE.md +@@ -0,0 +1,19 @@ ++# Claude / Coding-Agent Instructions ++ ++Read and follow [`AGENTS.md`](AGENTS.md) as the canonical repository policy. Do not ++create a second, conflicting rule set here. ++ ++For every change: ++ ++1. Preserve the batch threader as the correctness oracle. ++2. Work test-first and retain 100% production statement and branch coverage. ++3. Add beginner-readable docstrings to every authored production callable. ++4. Keep runtime dependencies at zero unless an approved architecture decision says ++ otherwise. ++5. Update `CHANGELOG.md`, user documentation, and standards references when public ++ behavior changes. ++6. Never bypass current-head CI, security scans, independent review, release ++ identity checks, or the release-blocker contract. ++7. GitHub product-development automation uses the isolated NVIDIA NIM/OpenCode ++ boundary. Do not introduce `COPILOT_GITHUB_TOKEN` or alter existing review-agent ++ credentials. +diff --git a/README.md b/README.md +index 416381a0f054801540ea68d9da1687d02914d898..8f839a24cb608dea83ac9197a669c3cff4f57c74 100644 +--- a/README.md ++++ b/README.md +@@ -217,7 +217,10 @@ assert IncrementalThreadIndex.restore(index.snapshot()).projections == ( + + Every affected component is recomputed through the canonical batch threader, and + full-rebuild parity is the correctness oracle. Structural merges and splits are +-reported explicitly. Versioned snapshots omit arbitrary payloads and reject ++reported explicitly. `roots` returns a defensive structural copy, so callers may ++traverse or edit that graph without corrupting reusable index state; payload objects ++remain caller-owned references. Internal sent-date tie-break positions never become ++public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads and reject + unknown, malformed, or oversized input. See + [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, + identity, snapshot, complexity, and RFC boundaries. +@@ -265,6 +268,9 @@ identity, snapshot, complexity, and RFC boundaries. + - Graph operations, IMAP rendering, and incremental component traversal are + iterative and identity-guarded; deep or cyclic malformed input cannot recurse + indefinitely. ++- `benchmarks/incremental_mailbox.py` compares isolated incremental and full ++ rebuild processes at 100,000 messages, verifies an identical projection digest, ++ and records affected-message count, wall time, and peak RSS as JSON evidence. + + ## Reproducible CI supply chain + +diff --git a/benchmarks/incremental_mailbox.py b/benchmarks/incremental_mailbox.py +new file mode 100644 +index 0000000000000000000000000000000000000000..063aed022704c48cd22e64c236584145090634e9 +--- /dev/null ++++ b/benchmarks/incremental_mailbox.py +@@ -0,0 +1,232 @@ ++"""Deterministic mailbox-scale benchmark for the incremental threading layer. ++ ++The parent process runs incremental and full-rebuild workers separately so each ++scenario reports its own peak resident-set size. Both workers hash the same ++caller-key projection to prove structural parity without serializing a 100,000- ++message forest between processes. ++""" ++ ++from __future__ import annotations ++ ++import argparse ++import hashlib ++import json ++import resource ++import subprocess ++import sys ++from collections.abc import Iterable ++from time import perf_counter ++ ++from threadweave import ( ++ Container, ++ IncrementalThreadIndex, ++ IndexedMessage, ++ MailboxChangeSet, ++ Message, ++ thread_messages, ++) ++ ++ ++def _records(message_count: int, thread_size: int) -> tuple[IndexedMessage, ...]: ++ """Create deterministic thread chains with caller keys retained as payloads.""" ++ records: list[IndexedMessage] = [] ++ for index in range(message_count): ++ position = index % thread_size ++ references = () if position == 0 else (f"message_{index - 1}",) ++ records.append( ++ IndexedMessage( ++ f"key_{index}", ++ Message( ++ message_id=f"message_{index}", ++ references=references, ++ payload=f"key_{index}", ++ ), ++ ) ++ ) ++ return tuple(records) ++ ++ ++def _bridge_record(thread_size: int) -> IndexedMessage: ++ """Return one message that joins the first two deterministic components.""" ++ return IndexedMessage( ++ "bridge_key", ++ Message( ++ message_id="bridge_message", ++ references=( ++ f"message_{thread_size - 1}", ++ f"message_{(thread_size * 2) - 1}", ++ ), ++ payload="bridge_key", ++ ), ++ ) ++ ++ ++def _projection_digest(projections: Iterable[tuple[str, ...]]) -> str: ++ """Hash ordered root projections with unambiguous length framing.""" ++ digest = hashlib.sha256() ++ for projection in projections: ++ digest.update(len(projection).to_bytes(8, "big")) ++ for message_key in projection: ++ encoded = message_key.encode("utf-8") From f4ea764defa498468248c6534d2e734201fa460e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 16:56:30 +0900 Subject: [PATCH 26/97] ci: stage verified PR 20 patch part 3 --- tools/pr20-review-fixes.part-03 | 173 ++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 tools/pr20-review-fixes.part-03 diff --git a/tools/pr20-review-fixes.part-03 b/tools/pr20-review-fixes.part-03 new file mode 100644 index 0000000..1a9c752 --- /dev/null +++ b/tools/pr20-review-fixes.part-03 @@ -0,0 +1,173 @@ ++ digest.update(len(encoded).to_bytes(8, "big")) ++ digest.update(encoded) ++ return digest.hexdigest() ++ ++ ++def _forest_projection(roots: Iterable[Container]) -> tuple[tuple[str, ...], ...]: ++ """Project a batch forest into iterative traversal-ordered payload keys.""" ++ result: list[tuple[str, ...]] = [] ++ for root in roots: ++ keys: list[str] = [] ++ seen: set[int] = set() ++ stack = [root] ++ while stack: ++ node = stack.pop() ++ if id(node) in seen: ++ continue ++ seen.add(id(node)) ++ if node.message is not None: ++ if not isinstance(node.message.payload, str): ++ raise RuntimeError("benchmark payload must be a caller key string") ++ keys.append(node.message.payload) ++ stack.extend(reversed(node.children)) ++ result.append(tuple(keys)) ++ return tuple(result) ++ ++ ++def _peak_rss_bytes() -> int: ++ """Return the process peak RSS in bytes on the Linux benchmark runner.""" ++ return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024 ++ ++ ++def _incremental_worker(message_count: int, thread_size: int) -> dict[str, object]: ++ """Build an index, apply one bridge delta, and materialize its final view.""" ++ records = _records(message_count, thread_size) ++ index = IncrementalThreadIndex() ++ started = perf_counter() ++ index.apply(MailboxChangeSet(expected_version=0, additions=records)) ++ initial_seconds = perf_counter() - started ++ ++ bridge = _bridge_record(thread_size) ++ started = perf_counter() ++ delta = index.apply( ++ MailboxChangeSet(expected_version=1, additions=(bridge,)) ++ ) ++ delta_seconds = perf_counter() - started ++ ++ started = perf_counter() ++ projections = tuple( ++ projection.message_keys for projection in index.projections ++ ) ++ materialize_seconds = perf_counter() - started ++ return { ++ "scenario": "incremental", ++ "message_count": message_count + 1, ++ "root_count": len(projections), ++ "affected_message_count": len(delta.affected_message_keys), ++ "initial_build_seconds": initial_seconds, ++ "delta_apply_seconds": delta_seconds, ++ "materialize_seconds": materialize_seconds, ++ "peak_rss_bytes": _peak_rss_bytes(), ++ "projection_sha256": _projection_digest(projections), ++ } ++ ++ ++def _full_worker(message_count: int, thread_size: int) -> dict[str, object]: ++ """Build the same final mailbox through the canonical batch oracle.""" ++ records = _records(message_count, thread_size) ++ messages = [record.message for record in records] ++ messages.append(_bridge_record(thread_size).message) ++ started = perf_counter() ++ roots = thread_messages(messages) ++ rebuild_seconds = perf_counter() - started ++ projections = _forest_projection(roots) ++ return { ++ "scenario": "full_rebuild", ++ "message_count": message_count + 1, ++ "root_count": len(projections), ++ "full_rebuild_seconds": rebuild_seconds, ++ "peak_rss_bytes": _peak_rss_bytes(), ++ "projection_sha256": _projection_digest(projections), ++ } ++ ++ ++def _run_worker(scenario: str, message_count: int, thread_size: int) -> dict[str, object]: ++ """Execute one isolated worker and decode its single JSON result.""" ++ completed = subprocess.run( ++ [ ++ sys.executable, ++ __file__, ++ "--worker", ++ scenario, ++ "--messages", ++ str(message_count), ++ "--thread-size", ++ str(thread_size), ++ ], ++ check=True, ++ capture_output=True, ++ text=True, ++ ) ++ return json.loads(completed.stdout) ++ ++ ++def _validated_positive(value: int, name: str) -> int: ++ """Require one positive non-boolean benchmark integer.""" ++ if isinstance(value, bool) or value <= 0: ++ raise ValueError(f"{name} must be a positive integer") ++ return value ++ ++ ++def run_benchmark(message_count: int, thread_size: int) -> dict[str, object]: ++ """Run isolated scenarios, require parity, and return one evidence record.""" ++ message_count = _validated_positive(message_count, "message_count") ++ thread_size = _validated_positive(thread_size, "thread_size") ++ if message_count < thread_size * 2: ++ raise ValueError("message_count must contain at least two complete threads") ++ incremental = _run_worker("incremental", message_count, thread_size) ++ full_rebuild = _run_worker("full_rebuild", message_count, thread_size) ++ if incremental["projection_sha256"] != full_rebuild["projection_sha256"]: ++ raise RuntimeError("incremental and full-rebuild projections disagree") ++ return { ++ "schema_version": 1, ++ "message_count": message_count + 1, ++ "thread_size": thread_size, ++ "incremental": incremental, ++ "full_rebuild": full_rebuild, ++ } ++ ++ ++def _parser() -> argparse.ArgumentParser: ++ """Build the command-line parser for parent and worker modes.""" ++ parser = argparse.ArgumentParser() ++ parser.add_argument("--messages", type=int, default=100_000) ++ parser.add_argument("--thread-size", type=int, default=10) ++ parser.add_argument( ++ "--worker", ++ choices=("incremental", "full_rebuild"), ++ ) ++ parser.add_argument("--output") ++ return parser ++ ++ ++def main(argv: list[str] | None = None) -> int: ++ """Run a worker or publish the combined deterministic benchmark JSON.""" ++ arguments = _parser().parse_args(argv) ++ if arguments.worker == "incremental": ++ result = _incremental_worker(arguments.messages, arguments.thread_size) ++ elif arguments.worker == "full_rebuild": ++ result = _full_worker(arguments.messages, arguments.thread_size) ++ else: ++ result = run_benchmark(arguments.messages, arguments.thread_size) ++ encoded = json.dumps(result, indent=2, sort_keys=True) + "\n" ++ if arguments.output: ++ with open(arguments.output, "w", encoding="utf-8") as output_file: ++ output_file.write(encoded) ++ else: ++ sys.stdout.write(encoded) ++ return 0 ++ ++ ++if __name__ == "__main__": ++ raise SystemExit(main()) +diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md +index b59ff27a0e683b88f359f786e21a123ee22a5340..6aa08c283a8bc71f25757b609d9c4b6dcb8d15f2 100644 +--- a/docs/incremental-threading.md ++++ b/docs/incremental-threading.md +@@ -118,10 +118,14 @@ old component; a new bridge message includes every old component touched by its + new tokens. The candidate region is repartitioned iteratively and each resulting + component is processed by `thread_messages`. + +-Unchanged components retain their existing `Container` roots. Building the +-ordered public root tuple still examines the component-root summaries so that +-batch-compatible global ordering is preserved; it does not re-run threading or From 9b2c42d6d950811d94942b8970b65501131ef1b2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 16:57:30 +0900 Subject: [PATCH 27/97] ci: stage verified PR 20 patch part 4 --- tools/pr20-review-fixes.part-04 | 153 ++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 tools/pr20-review-fixes.part-04 diff --git a/tools/pr20-review-fixes.part-04 b/tools/pr20-review-fixes.part-04 new file mode 100644 index 0000000..3ef9d3f --- /dev/null +++ b/tools/pr20-review-fixes.part-04 @@ -0,0 +1,153 @@ +-walk every message in unrelated components. ++Unchanged components retain their existing internal `Container` roots. Applying a ++change does not eagerly rebuild the complete public forest: only the affected old ++and new component views are composed for `ThreadDelta`. The complete ordered forest ++is materialized once, on demand, when `roots` or `projections` is requested. Public ++roots are defensive structural copies; editing their parent, child, or message ++metadata cannot corrupt internal state, while caller payload objects remain shared by ++reference. Internal implicit sent-date ranks are cleared before roots are exposed, so ++ordinary IMAP `THREAD` output still requires real caller-supplied sequence numbers. + + `ThreadDelta.affected_message_keys` reports the candidate region. The delta also + contains added, removed, and structurally updated projections. A metadata-only +@@ -179,11 +183,13 @@ It also covers RFC 5051 subject buckets, RFC 5256 sent-date ordering, ordinary a + UID THREAD output, duplicate and missing Message-ID values, deep chains, + optimistic conflicts, hostile snapshots, and payload omission. + +-A separate scheduled/manual benchmark is required before this feature is released +-as a performance claim. It must exercise at least 100,000 records and report wall +-time, peak RSS, affected-message count, and a full-rebuild comparison. The +-incremental contract promises that unrelated records are not passed to the batch +-threader; it does not promise constant-time global root presentation. ++`benchmarks/incremental_mailbox.py` runs incremental and full-rebuild scenarios in ++separate processes, requires identical projection hashes, and reports wall time, peak ++RSS, affected-message count, and full-view materialization time. The scheduled/manual ++workflow defaults to 100,000 existing messages and stores the JSON evidence for 90 ++days. The update contract promises that unrelated records are not passed to the batch ++threader; full-view materialization remains proportional to the number of roots and is ++therefore reported separately from delta application. + + ## References + +diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py +index 9150bab749b0c1d87772feca12275d37ad765694..fe9adeaba587f08ccc7a4a33f75722bbbc70c2de 100644 +--- a/src/threadweave/incremental.py ++++ b/src/threadweave/incremental.py +@@ -369,22 +369,30 @@ def _connectivity_tokens( + return frozenset(tokens) + + +-def _copy_token_buckets( +- buckets: Mapping[str, set[str]], +-) -> dict[str, set[str]]: +- """Return independent mutable copies of reverse token buckets.""" +- return {token: set(keys) for token, keys in buckets.items()} ++def _writable_token_bucket( ++ token: str, ++ keys_by_token: dict[str, set[str]], ++ copied_tokens: set[str], ++) -> set[str]: ++ """Return one copy-on-write reverse bucket owned by the transaction.""" ++ if token not in copied_tokens: ++ keys_by_token[token] = set(keys_by_token.get(token, set())) ++ copied_tokens.add(token) ++ elif token not in keys_by_token: ++ keys_by_token[token] = set() ++ return keys_by_token[token] + + + def _remove_key_from_buckets( + key: str, + tokens_by_key: dict[str, frozenset[str]], + keys_by_token: dict[str, set[str]], ++ copied_tokens: set[str], + ) -> frozenset[str]: +- """Remove one key from copied token indexes and return its old tokens.""" ++ """Remove one key from transaction-owned token buckets.""" + old_tokens = tokens_by_key.pop(key, frozenset()) + for token in old_tokens: +- bucket = keys_by_token[token] ++ bucket = _writable_token_bucket(token, keys_by_token, copied_tokens) + bucket.discard(key) + if not bucket: + del keys_by_token[token] +@@ -396,11 +404,12 @@ def _add_key_to_buckets( + tokens: frozenset[str], + tokens_by_key: dict[str, frozenset[str]], + keys_by_token: dict[str, set[str]], ++ copied_tokens: set[str], + ) -> None: +- """Insert one key into copied forward and reverse token indexes.""" ++ """Insert one key through transaction-owned copy-on-write buckets.""" + tokens_by_key[key] = tokens + for token in tokens: +- keys_by_token.setdefault(token, set()).add(key) ++ _writable_token_bucket(token, keys_by_token, copied_tokens).add(key) + + + def _ordered_keys(keys: Iterable[str], positions: Mapping[str, int]) -> tuple[str, ...]: +@@ -497,8 +506,9 @@ def _partition_components( + """Partition candidate keys into deterministic current connectivity components.""" + remaining = set(keys) + components: list[tuple[str, ...]] = [] +- while remaining: +- seed = min(remaining, key=lambda key: (positions[key], key)) ++ for seed in _ordered_keys(keys, positions): ++ if seed not in remaining: ++ continue + component = {seed} + queue = [seed] + remaining.remove(seed) +@@ -531,6 +541,59 @@ def _message_for_batch(message: Message, sequence_number: int | None) -> Message + ) + + ++def _public_message_copy(message: Message) -> Message: ++ """Copy structural message metadata while retaining the caller payload.""" ++ return Message( ++ message_id=message.message_id, ++ in_reply_to=message.in_reply_to, ++ references=message.references, ++ subject=message.subject, ++ payload=message.payload, ++ sent_date=message.sent_date, ++ internal_date=message.internal_date, ++ sequence_number=message.sequence_number, ++ uid=message.uid, ++ ) ++ ++ ++def _public_forest_copy(roots: Iterable[Container]) -> tuple[Container, ...]: ++ """Return a loop-safe defensive copy of the index's internal forest.""" ++ copied_roots: list[Container] = [] ++ seen: set[int] = set() ++ for root in roots: ++ root_identity = id(root) ++ if root_identity in seen: ++ raise IncrementalThreadError( ++ "internal thread forest contains a shared or cyclic container" ++ ) ++ root_copy = Container( ++ message=None ++ if root.message is None ++ else _public_message_copy(root.message) ++ ) ++ copied_roots.append(root_copy) ++ seen.add(root_identity) ++ stack: list[tuple[Container, Container]] = [(root, root_copy)] ++ while stack: ++ source, target = stack.pop() ++ for child in source.children: ++ child_identity = id(child) ++ if child_identity in seen: ++ raise IncrementalThreadError( ++ "internal thread forest contains a shared or cyclic container" ++ ) ++ child_copy = Container( ++ message=None ++ if child.message is None ++ else _public_message_copy(child.message), ++ parent=target, ++ ) ++ target.children.append(child_copy) ++ seen.add(child_identity) From 872acb052d78ecc7e13d62559f5211da599271a1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 16:58:42 +0900 Subject: [PATCH 28/97] ci: stage verified PR 20 patch part 5 --- tools/pr20-review-fixes.part-05 | 189 ++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 tools/pr20-review-fixes.part-05 diff --git a/tools/pr20-review-fixes.part-05 b/tools/pr20-review-fixes.part-05 new file mode 100644 index 0000000..c168e82 --- /dev/null +++ b/tools/pr20-review-fixes.part-05 @@ -0,0 +1,189 @@ ++ stack.append((child, child_copy)) ++ return tuple(copied_roots) ++ ++ + def _projection_for_root( + root: Container, + key_by_message_identity: Mapping[int, str], +@@ -565,7 +628,7 @@ def _projection_for_root( + return ThreadProjection(tuple(message_keys), thread_ids) + + +-def _build_component( ++def _build_forest( + keys: tuple[str, ...], + records: Mapping[str, IndexedMessage], + ranks: Mapping[str, int], +@@ -573,7 +636,7 @@ def _build_component( + group_by_subject: bool, + sort_by_sent_date: bool, + ) -> tuple[tuple[Container, ...], tuple[ThreadProjection, ...]]: +- """Run the canonical batch engine for one affected component.""" ++ """Run the canonical batch engine for one ordered record subset.""" + messages: list[Message] = [] + key_by_message_identity: dict[int, str] = {} + for key in keys: +@@ -593,74 +656,12 @@ def _build_component( + projections = tuple( + _projection_for_root(root, key_by_message_identity, records) for root in roots + ) ++ for key, message in zip(keys, messages): ++ if records[key].message.sequence_number is None: ++ message.sequence_number = None + return roots, projections + + +-def _root_order_key( +- root: Container, +- projection: ThreadProjection, +- records: Mapping[str, IndexedMessage], +- ranks: Mapping[str, int], +- *, +- sort_by_sent_date: bool, +-) -> tuple[object, ...]: +- """Return the same global root-order contract used by the batch algorithm.""" +- if not sort_by_sent_date: +- return (min(ranks[key] for key in projection.message_keys),) +- if not projection.message_keys: +- return (normalize_sent_date(None), 0, 0) +- first_key = projection.message_keys[0] +- message = records[first_key].message +- sequence_number = ( +- ranks[first_key] +- if message.sequence_number is None +- else message.sequence_number +- ) +- return ( +- normalize_sent_date(message.sent_date, message.internal_date), +- sequence_number, +- ranks[first_key], +- ) +- +- +-def _compose_forest( +- roots_by_component: Mapping[str, tuple[Container, ...]], +- projections_by_component: Mapping[str, tuple[ThreadProjection, ...]], +- records: Mapping[str, IndexedMessage], +- ranks: Mapping[str, int], +- *, +- sort_by_sent_date: bool, +-) -> tuple[tuple[Container, ...], tuple[ThreadProjection, ...]]: +- """Compose reusable component outputs into one deterministic global forest.""" +- entries: list[tuple[tuple[object, ...], Container, ThreadProjection]] = [] +- for component_id, roots in roots_by_component.items(): +- projections = projections_by_component[component_id] +- for root, projection in zip(roots, projections): +- entries.append( +- ( +- _root_order_key( +- root, +- projection, +- records, +- ranks, +- sort_by_sent_date=sort_by_sent_date, +- ), +- root, +- projection, +- ) +- ) +- entries.sort(key=lambda entry: entry[0]) +- return ( +- tuple(entry[1] for entry in entries), +- tuple(entry[2] for entry in entries), +- ) +- +- +-def _projection_overlap(first: ThreadProjection, second: ThreadProjection) -> bool: +- """Return whether two projections share any immutable caller message key.""" +- return bool(set(first.message_keys) & set(second.message_keys)) +- +- + def _transition_thread_ids( + before: Iterable[ThreadProjection], + after: Iterable[ThreadProjection], +@@ -677,54 +678,85 @@ def _transition_thread_ids( + ) + + ++def _projection_membership( ++ projections: Sequence[ThreadProjection], ++ name: str, ++) -> dict[str, int]: ++ """Index each caller key once and reject overlapping root projections.""" ++ membership: dict[str, int] = {} ++ for projection_index, projection in enumerate(projections): ++ for message_key in projection.message_keys: ++ if message_key in membership: ++ raise IncrementalThreadError( ++ f"{name} projections contain duplicate message_key: {message_key}" ++ ) ++ membership[message_key] = projection_index ++ return membership ++ ++ + def _thread_delta( + previous_version: int, + version: int, + affected_message_keys: tuple[str, ...], +- before: tuple[ThreadProjection, ...], +- after: tuple[ThreadProjection, ...], ++ before: Sequence[ThreadProjection], ++ after: Sequence[ThreadProjection], + ) -> ThreadDelta: +- """Classify deterministic projection additions, removals, updates, and transitions.""" ++ """Classify projection changes and transitions in linear message-key work.""" ++ before_tuple = tuple(before) ++ after_tuple = tuple(after) ++ before_membership = _projection_membership(before_tuple, "before") ++ after_membership = _projection_membership(after_tuple, "after") ++ ++ before_by_after: list[set[int]] = [set() for _ in after_tuple] ++ after_by_before: list[set[int]] = [set() for _ in before_tuple] ++ for message_key, after_index in after_membership.items(): ++ before_index = before_membership.get(message_key) ++ if before_index is not None: ++ before_by_after[after_index].add(before_index) ++ after_by_before[before_index].add(after_index) ++ ++ before_values = set(before_tuple) + added = tuple( + projection +- for projection in after +- if not any(_projection_overlap(projection, old) for old in before) ++ for projection, overlaps in zip(after_tuple, before_by_after) ++ if not overlaps + ) + removed = tuple( + projection +- for projection in before +- if not any(_projection_overlap(projection, new) for new in after) ++ for projection, overlaps in zip(before_tuple, after_by_before) ++ if not overlaps + ) + updated = tuple( + projection +- for projection in after +- if projection not in before +- and any(_projection_overlap(projection, old) for old in before) ++ for projection, overlaps in zip(after_tuple, before_by_after) ++ if overlaps and projection not in before_values ++ ) ++ merges = tuple( ++ ThreadTransition( ++ "merge", ++ tuple(before_tuple[index] for index in sorted(overlaps)), ++ (projection,), ++ _transition_thread_ids( ++ tuple(before_tuple[index] for index in sorted(overlaps)), ++ (projection,), ++ ), ++ ) ++ for projection, overlaps in zip(after_tuple, before_by_after) ++ if len(overlaps) > 1 ++ ) ++ splits = tuple( ++ ThreadTransition( ++ "split", ++ (projection,), ++ tuple(after_tuple[index] for index in sorted(overlaps)), ++ _transition_thread_ids( From bd2e132ae8fb859b299d93169fa70fb5385191bb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 16:59:32 +0900 Subject: [PATCH 29/97] ci: stage verified PR 20 patch part 6 --- tools/pr20-review-fixes.part-06 | 191 ++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 tools/pr20-review-fixes.part-06 diff --git a/tools/pr20-review-fixes.part-06 b/tools/pr20-review-fixes.part-06 new file mode 100644 index 0000000..39c453b --- /dev/null +++ b/tools/pr20-review-fixes.part-06 @@ -0,0 +1,191 @@ ++ (projection,), ++ tuple(after_tuple[index] for index in sorted(overlaps)), ++ ), ++ ) ++ for projection, overlaps in zip(before_tuple, after_by_before) ++ if len(overlaps) > 1 + ) +- merges: list[ThreadTransition] = [] +- for projection in after: +- overlapping = tuple(old for old in before if _projection_overlap(projection, old)) +- if len(overlapping) > 1: +- merges.append( +- ThreadTransition( +- "merge", +- overlapping, +- (projection,), +- _transition_thread_ids(overlapping, (projection,)), +- ) +- ) +- splits: list[ThreadTransition] = [] +- for projection in before: +- overlapping = tuple(new for new in after if _projection_overlap(projection, new)) +- if len(overlapping) > 1: +- splits.append( +- ThreadTransition( +- "split", +- (projection,), +- overlapping, +- _transition_thread_ids((projection,), overlapping), +- ) +- ) + return ThreadDelta( + previous_version, + version, +@@ -732,8 +764,8 @@ def _thread_delta( + added, + removed, + updated, +- tuple(merges), +- tuple(splits), ++ merges, ++ splits, + ) + + +@@ -821,15 +853,32 @@ class IncrementalThreadIndex: + self._keys_by_token: dict[str, set[str]] = {} + self._component_by_key: dict[str, str] = {} + self._keys_by_component: dict[str, tuple[str, ...]] = {} +- self._roots_by_component: dict[str, tuple[Container, ...]] = {} +- self._projections_by_component: dict[str, tuple[ThreadProjection, ...]] = {} +- self._roots: tuple[Container, ...] = () +- self._projections: tuple[ThreadProjection, ...] = () ++ self._roots: tuple[Container, ...] | None = () ++ self._projections: tuple[ThreadProjection, ...] | None = () + + def __len__(self) -> int: + """Return the number of indexed caller message keys.""" + return len(self._records) + ++ def _materialize_forest(self) -> None: ++ """Build and cache the complete canonical forest only when requested.""" ++ if self._roots is not None and self._projections is not None: ++ return ++ ranks = ( ++ _current_ranks(self._positions) ++ if self._sort_by_sent_date ++ else self._positions ++ ) ++ roots, projections = _build_forest( ++ self.message_keys, ++ self._records, ++ ranks, ++ group_by_subject=self._group_by_subject, ++ sort_by_sent_date=self._sort_by_sent_date, ++ ) ++ self._roots = roots ++ self._projections = projections ++ + @property + def version(self) -> int: + """Return the optimistic mailbox-state version.""" +@@ -842,12 +891,16 @@ class IncrementalThreadIndex: + + @property + def roots(self) -> tuple[Container, ...]: +- """Return the current transport-neutral thread roots.""" +- return self._roots ++ """Return defensive transport-neutral copies of current thread roots.""" ++ self._materialize_forest() ++ assert self._roots is not None ++ return _public_forest_copy(self._roots) + + @property + def projections(self) -> tuple[ThreadProjection, ...]: + """Return deterministic caller-key projections for current roots.""" ++ self._materialize_forest() ++ assert self._projections is not None + return self._projections + + def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: +@@ -870,12 +923,15 @@ class IncrementalThreadIndex: + if not ( + change_set.additions or change_set.replacements or change_set.removals + ): +- return _thread_delta( ++ return ThreadDelta( + self._version, + self._version, + (), +- self._projections, +- self._projections, ++ (), ++ (), ++ (), ++ (), ++ (), + ) + + addition_keys = {record.message_key for record in change_set.additions} +@@ -913,7 +969,8 @@ class IncrementalThreadIndex: + records = dict(self._records) + positions = dict(self._positions) + tokens_by_key = dict(self._tokens_by_key) +- keys_by_token = _copy_token_buckets(self._keys_by_token) ++ keys_by_token = dict(self._keys_by_token) ++ copied_tokens: set[str] = set() + next_position = self._next_position + changed_existing_keys = replacement_keys | removal_keys + candidate_seeds: set[str] = set() +@@ -924,7 +981,12 @@ class IncrementalThreadIndex: + if component_id is not None: + candidate_seeds.update(self._keys_by_component[component_id]) + touched_tokens.update( +- _remove_key_from_buckets(key, tokens_by_key, keys_by_token) ++ _remove_key_from_buckets( ++ key, ++ tokens_by_key, ++ keys_by_token, ++ copied_tokens, ++ ) + ) + + for key in removal_keys: +@@ -942,6 +1004,7 @@ class IncrementalThreadIndex: + tokens, + tokens_by_key, + keys_by_token, ++ copied_tokens, + ) + candidate_seeds.add(replacement.message_key) + for addition in copied_additions: +@@ -958,6 +1021,7 @@ class IncrementalThreadIndex: + tokens, + tokens_by_key, + keys_by_token, ++ copied_tokens, + ) + candidate_seeds.add(addition.message_key) + +@@ -968,8 +1032,38 @@ class IncrementalThreadIndex: + if component_id is not None: + candidate_seeds.update(self._keys_by_component[component_id]) + ++ old_component_ids = { ++ self._component_by_key[key] ++ for key in candidate_seeds ++ if key in self._component_by_key ++ } ++ old_ranks = ( ++ _current_ranks(self._positions) ++ if self._sort_by_sent_date ++ else self._positions ++ ) ++ before_affected_keys = _ordered_keys( ++ { ++ key ++ for component_id in old_component_ids ++ for key in self._keys_by_component[component_id] ++ }, ++ self._positions, ++ ) ++ _, before_affected_projections = _build_forest( ++ before_affected_keys, ++ self._records, ++ old_ranks, ++ group_by_subject=self._group_by_subject, ++ sort_by_sent_date=self._sort_by_sent_date, ++ ) ++ + _validate_external_identities(records) From 838ac37a487dee3a8dfa97882fd7014863459ea7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:00:21 +0900 Subject: [PATCH 30/97] ci: stage verified PR 20 patch part 7 --- tools/pr20-review-fixes.part-07 | 189 ++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 tools/pr20-review-fixes.part-07 diff --git a/tools/pr20-review-fixes.part-07 b/tools/pr20-review-fixes.part-07 new file mode 100644 index 0000000..779b3bf --- /dev/null +++ b/tools/pr20-review-fixes.part-07 @@ -0,0 +1,189 @@ +- ranks = _current_ranks(positions) ++ ranks = ( ++ _current_ranks(positions) ++ if self._sort_by_sent_date ++ else positions ++ ) + if self._sort_by_sent_date: + _validate_effective_sequence_numbers(records, ranks) + +@@ -992,17 +1086,6 @@ class IncrementalThreadIndex: + for component_id, keys in self._keys_by_component.items() + if component_id in unaffected_component_ids + } +- roots_by_component = { +- component_id: roots +- for component_id, roots in self._roots_by_component.items() +- if component_id in unaffected_component_ids +- } +- projections_by_component = { +- component_id: projections +- for component_id, projections in self._projections_by_component.items() +- if component_id in unaffected_component_ids +- } +- + for keys in _partition_components( + current_candidate_keys, + positions, +@@ -1010,24 +1093,16 @@ class IncrementalThreadIndex: + keys_by_token, + ): + component_id = keys[0] +- roots, projections = _build_component( +- keys, +- records, +- ranks, +- group_by_subject=self._group_by_subject, +- sort_by_sent_date=self._sort_by_sent_date, +- ) + keys_by_component[component_id] = keys +- roots_by_component[component_id] = roots +- projections_by_component[component_id] = projections + for key in keys: + component_by_key[key] = component_id + +- roots, projections = _compose_forest( +- roots_by_component, +- projections_by_component, ++ after_affected_keys = _ordered_keys(current_candidate_keys, positions) ++ _, after_affected_projections = _build_forest( ++ after_affected_keys, + records, + ranks, ++ group_by_subject=self._group_by_subject, + sort_by_sent_date=self._sort_by_sent_date, + ) + affected_positions = { +@@ -1043,8 +1118,8 @@ class IncrementalThreadIndex: + previous_version, + version, + affected, +- self._projections, +- projections, ++ before_affected_projections, ++ after_affected_projections, + ) + + self._records = records +@@ -1054,10 +1129,8 @@ class IncrementalThreadIndex: + self._keys_by_token = keys_by_token + self._component_by_key = component_by_key + self._keys_by_component = keys_by_component +- self._roots_by_component = roots_by_component +- self._projections_by_component = projections_by_component +- self._roots = roots +- self._projections = projections ++ self._roots = None ++ self._projections = None + self._version = version + return delta + +diff --git a/tests/test_incremental_benchmark.py b/tests/test_incremental_benchmark.py +new file mode 100644 +index 0000000000000000000000000000000000000000..4dbe6eb248dba15d9c17b7bfc6f17c1a126f4d34 +--- /dev/null ++++ b/tests/test_incremental_benchmark.py +@@ -0,0 +1,65 @@ ++"""Contract tests for the mailbox-scale incremental benchmark.""" ++ ++from __future__ import annotations ++ ++import importlib.util ++import json ++import sys ++from pathlib import Path ++ ++import pytest ++ ++ROOT = Path(__file__).parents[1] ++BENCHMARK_PATH = ROOT / "benchmarks" / "incremental_mailbox.py" ++SPEC = importlib.util.spec_from_file_location( ++ "threadweave_incremental_benchmark", ++ BENCHMARK_PATH, ++) ++assert SPEC is not None and SPEC.loader is not None ++benchmark = importlib.util.module_from_spec(SPEC) ++sys.modules[SPEC.name] = benchmark ++SPEC.loader.exec_module(benchmark) ++ ++ ++def test_small_benchmark_proves_projection_parity_and_reports_resources(): ++ """The CI-sized run emits matching digests and complete evidence fields.""" ++ result = benchmark.run_benchmark(1_000, 10) ++ ++ assert result["schema_version"] == 1 ++ assert result["message_count"] == 1_001 ++ assert result["incremental"]["projection_sha256"] == ( ++ result["full_rebuild"]["projection_sha256"] ++ ) ++ assert result["incremental"]["affected_message_count"] == 21 ++ assert result["incremental"]["delta_apply_seconds"] >= 0 ++ assert result["incremental"]["peak_rss_bytes"] > 0 ++ assert result["full_rebuild"]["full_rebuild_seconds"] >= 0 ++ assert result["full_rebuild"]["peak_rss_bytes"] > 0 ++ ++ ++def test_benchmark_validates_size_contracts(): ++ """Invalid or undersized workloads fail before spawning workers.""" ++ with pytest.raises(ValueError, match="message_count"): ++ benchmark.run_benchmark(0, 10) ++ with pytest.raises(ValueError, match="thread_size"): ++ benchmark.run_benchmark(100, 0) ++ with pytest.raises(ValueError, match="two complete threads"): ++ benchmark.run_benchmark(10, 10) ++ ++ ++def test_main_writes_deterministic_json_shape(tmp_path: Path): ++ """The command-line entry point writes one parseable evidence document.""" ++ output = tmp_path / "benchmark.json" ++ assert benchmark.main( ++ [ ++ "--messages", ++ "1000", ++ "--thread-size", ++ "10", ++ "--output", ++ str(output), ++ ] ++ ) == 0 ++ parsed = json.loads(output.read_text(encoding="utf-8")) ++ assert parsed["schema_version"] == 1 ++ assert parsed["message_count"] == 1_001 +diff --git a/tests/test_incremental_benchmark_workflow.py b/tests/test_incremental_benchmark_workflow.py +new file mode 100644 +index 0000000000000000000000000000000000000000..3c9120d934c6d9a2f3d4834c452d78cf4438a9bf +--- /dev/null ++++ b/tests/test_incremental_benchmark_workflow.py +@@ -0,0 +1,56 @@ ++"""Governance contract for the mailbox-scale benchmark workflow.""" ++ ++from pathlib import Path ++ ++WORKFLOW = ( ++ Path(__file__).parents[1] ++ / ".github" ++ / "workflows" ++ / "incremental-benchmark.yml" ++) ++ ++ ++def _workflow() -> str: ++ """Return the workflow source without adding a YAML parser dependency.""" ++ return WORKFLOW.read_text(encoding="utf-8") ++ ++ ++def test_benchmark_is_manual_and_scheduled_with_a_100k_default(): ++ """Operators can reproduce evidence while a weekly run catches regressions.""" ++ workflow = _workflow() ++ assert "workflow_dispatch:" in workflow ++ assert 'default: "100000"' in workflow ++ assert 'cron: "17 3 * * 1"' in workflow ++ assert "cancel-in-progress: true" in workflow ++ ++ ++def test_benchmark_uses_immutable_actions_and_the_reviewed_dependency_lock(): ++ """Performance evidence runs through the same pinned supply-chain boundary.""" ++ workflow = _workflow() ++ required_actions = { ++ "step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920", ++ "actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1", From 7911032c70c823f0b2521aeabd56a6aef6c0d805 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:01:17 +0900 Subject: [PATCH 31/97] ci: stage verified PR 20 patch part 8 --- tools/pr20-review-fixes.part-08 | 189 ++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 tools/pr20-review-fixes.part-08 diff --git a/tools/pr20-review-fixes.part-08 b/tools/pr20-review-fixes.part-08 new file mode 100644 index 0000000..18dd68f --- /dev/null +++ b/tools/pr20-review-fixes.part-08 @@ -0,0 +1,189 @@ ++ "actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97", ++ "actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a", ++ } ++ observed_actions = { ++ line.strip().removeprefix("uses: ").split(" #", 1)[0] ++ for line in workflow.splitlines() ++ if line.strip().startswith("uses: ") ++ } ++ assert required_actions <= observed_actions ++ assert "pip install --require-hashes -r requirements/ci.lock" in workflow ++ assert "persist-credentials: false" in workflow ++ ++ ++def test_benchmark_requires_parity_bounded_impact_and_evidence_upload(): ++ """The workflow fails closed on incorrect output or a regressed 100k delta.""" ++ workflow = _workflow() ++ assert "incremental_mailbox.py" in workflow ++ assert 'incremental["projection_sha256"] == full_rebuild["projection_sha256"]' in workflow ++ assert 'incremental["affected_message_count"] == 21' in workflow ++ assert 'incremental["delta_apply_seconds"] < full_rebuild[' in workflow ++ assert "incremental-mailbox-benchmark-${{ github.run_id }}" in workflow ++ assert "retention-days: 90" in workflow ++ assert "NVIDIA_NIM_API_KEY" not in workflow ++ assert "COPILOT_GITHUB_TOKEN" not in workflow +diff --git a/tests/test_incremental_components.py b/tests/test_incremental_components.py +index 626a3f12604a8f926d69ee05ade5309d16086a82..5343c00195e02d93755f70673f3396e254af3aa6 100644 +--- a/tests/test_incremental_components.py ++++ b/tests/test_incremental_components.py +@@ -226,7 +226,7 @@ def test_unrelated_components_are_not_passed_to_the_batch_delegate(monkeypatch): + MailboxChangeSet(expected_version=1, replacements=(replacement,)) + ) + +- assert calls == [("a", "b")] ++ assert calls == [("a", "b"), ("a", "b")] + assert all("x" not in call and "y" not in call for call in calls) + + +@@ -243,3 +243,86 @@ def test_noop_change_set_advances_no_version_and_returns_empty_delta(): + assert delta.updated_threads == () + assert delta.merges == () + assert delta.splits == () ++ ++ ++def test_delta_classification_receives_only_affected_component_projections( ++ monkeypatch: pytest.MonkeyPatch, ++): ++ """Unrelated roots never enter one small change's delta classifier.""" ++ records = tuple( ++ _record(f"key_{index}", message_id=f"message_{index}") ++ for index in range(128) ++ ) ++ index = IncrementalThreadIndex() ++ index.apply(MailboxChangeSet(expected_version=0, additions=records)) ++ ++ original = incremental_module._thread_delta ++ observed_sizes: list[tuple[int, int]] = [] ++ ++ def recording_delta(*args): ++ """Record old/new projection counts before delegating.""" ++ observed_sizes.append((len(args[3]), len(args[4]))) ++ return original(*args) ++ ++ monkeypatch.setattr(incremental_module, "_thread_delta", recording_delta) ++ index.apply( ++ MailboxChangeSet( ++ expected_version=1, ++ additions=( ++ _record( ++ "new_key", ++ message_id="new_message", ++ references=("message_0",), ++ ), ++ ), ++ ) ++ ) ++ ++ assert observed_sizes == [(1, 1)] ++ ++ ++ ++def test_small_delta_defers_the_complete_canonical_batch_view( ++ monkeypatch: pytest.MonkeyPatch, ++): ++ """Apply batches only affected records until a full view is requested.""" ++ records = tuple( ++ _record(f"key_{index}", message_id=f"message_{index}") ++ for index in range(128) ++ ) ++ index = IncrementalThreadIndex() ++ index.apply(MailboxChangeSet(expected_version=0, additions=records)) ++ ++ original = incremental_module._batch_thread_messages ++ observed_message_counts: list[int] = [] ++ ++ def recording_batch(messages, **options): ++ """Record each canonical batch size before delegating.""" ++ materialized = tuple(messages) ++ observed_message_counts.append(len(materialized)) ++ return original(materialized, **options) ++ ++ monkeypatch.setattr( ++ incremental_module, ++ "_batch_thread_messages", ++ recording_batch, ++ ) ++ index.apply( ++ MailboxChangeSet( ++ expected_version=1, ++ additions=( ++ _record( ++ "new_key", ++ message_id="new_message", ++ references=("message_0",), ++ ), ++ ), ++ ) ++ ) ++ ++ assert observed_message_counts == [1, 2] ++ assert len(index.projections) == 128 ++ assert observed_message_counts == [1, 2, 129] ++ assert len(index.projections) == 128 ++ assert len(index.roots) == 128 ++ assert observed_message_counts == [1, 2, 129] +diff --git a/tests/test_incremental_contract.py b/tests/test_incremental_contract.py +index dba9e71d57036ac6705cf7d6000403ae6ce5c19d..d374c831a34fd5731162c8fcfde83303cb74fcff 100644 +--- a/tests/test_incremental_contract.py ++++ b/tests/test_incremental_contract.py +@@ -171,6 +171,43 @@ def test_structural_metadata_is_copied_while_payload_remains_caller_owned(): + assert indexed.payload is message.payload + + ++def test_public_roots_are_defensive_structural_copies(): ++ """Caller graph edits cannot corrupt the index's reusable internal forest.""" ++ payload = object() ++ index = IncrementalThreadIndex() ++ index.apply( ++ MailboxChangeSet( ++ expected_version=0, ++ additions=( ++ _record( ++ "root", ++ message_id="root", ++ subject="Original", ++ payload=payload, ++ ), ++ _record( ++ "child", ++ message_id="child", ++ references=("root",), ++ ), ++ ), ++ ) ++ ) ++ ++ exposed = index.roots ++ exposed[0].message.subject = "Mutated" ++ exposed[0].children.clear() ++ ++ current = index.roots ++ assert current is not exposed ++ assert current[0] is not exposed[0] ++ assert current[0].message is not exposed[0].message ++ assert current[0].message.subject == "Original" ++ assert current[0].message.payload is payload ++ assert [child.message.payload for child in current[0].children] == ["child"] ++ assert index.projections[0].message_keys == ("root", "child") ++ ++ + def test_reported_external_identity_is_immutable_on_replacement(): + """RFC 8474 identity metadata cannot disappear or change after exposure.""" + index = IncrementalThreadIndex() +diff --git a/tests/test_incremental_parity.py b/tests/test_incremental_parity.py +index 6c87474c4683a1a5bab17a9ff8d4678b5dbe2abc..2162dd143383842d651c6e2403955233999b498a 100644 +--- a/tests/test_incremental_parity.py ++++ b/tests/test_incremental_parity.py +@@ -2,6 +2,8 @@ + + from __future__ import annotations + ++import random ++ + import pytest + + from threadweave import ( +@@ -10,6 +12,7 @@ from threadweave import ( + IndexedMessage, + MailboxChangeSet, + Message, From bbee122ba1dba689b20544658ddfdd9692b97cf0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:02:06 +0900 Subject: [PATCH 32/97] ci: stage verified PR 20 patch part 9 --- tools/pr20-review-fixes.part-09 | 177 ++++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 tools/pr20-review-fixes.part-09 diff --git a/tools/pr20-review-fixes.part-09 b/tools/pr20-review-fixes.part-09 new file mode 100644 index 0000000..63452cd --- /dev/null +++ b/tools/pr20-review-fixes.part-09 @@ -0,0 +1,177 @@ ++ ThreadSerializationError, + serialize_thread_response, + thread_messages, + ) +@@ -105,6 +108,34 @@ def test_sent_date_order_matches_batch_across_unrelated_components(): + ) + + ++def test_implicit_sort_positions_do_not_leak_into_public_metadata(): ++ """Internal sort ranks must not become caller-visible IMAP identifiers.""" ++ records = ( ++ _record( ++ "earlier", ++ message_id="earlier", ++ sent_date="1 Jan 2026 00:00:00 +0000", ++ ), ++ _record( ++ "later", ++ message_id="later", ++ sent_date="2 Jan 2026 00:00:00 +0000", ++ ), ++ ) ++ index = IncrementalThreadIndex(sort_by_sent_date=True) ++ index.apply(MailboxChangeSet(expected_version=0, additions=records)) ++ ++ batch = thread_messages( ++ (record.message for record in records), ++ sort_by_sent_date=True, ++ ) ++ assert [root.message.sequence_number for root in index.roots] == [ ++ root.message.sequence_number for root in batch ++ ] == [None, None] ++ with pytest.raises(ThreadSerializationError, match="positive integer"): ++ serialize_thread_response(index.roots) ++ ++ + def test_global_sequence_number_collision_fails_atomically(): + """Separate components cannot hide duplicate RFC ordering tie-breakers.""" + index = IncrementalThreadIndex(sort_by_sent_date=True) +@@ -209,3 +240,132 @@ def test_implicit_sequence_positions_remain_stable_after_replacement(): + ("a",), + ("b",), + ) ++ ++ ++def test_missing_ancestor_creation_order_matches_canonical_batch_root_order(): ++ """A late missing-root placeholder cannot reorder an earlier independent root.""" ++ records = ( ++ _record( ++ "early_child", ++ message_id="child@example.test", ++ references=("late-parent@example.test",), ++ ), ++ _record("independent", message_id="independent@example.test"), ++ _record( ++ "late_parent", ++ message_id="late-parent@example.test", ++ references=("missing-ancestor@example.test",), ++ ), ++ ) ++ index = IncrementalThreadIndex() ++ index.apply(MailboxChangeSet(expected_version=0, additions=records)) ++ ++ batch = thread_messages(record.message for record in records) ++ ++ assert _preorder(index.roots) == _preorder(batch) == ( ++ ("independent",), ++ ("late_parent", "early_child"), ++ ) ++ ++ ++@pytest.mark.parametrize("group_by_subject", [False, True]) ++@pytest.mark.parametrize("sort_by_sent_date", [False, True]) ++def test_bounded_randomized_change_stream_matches_full_batch_oracle( ++ group_by_subject: bool, ++ sort_by_sent_date: bool, ++): ++ """Deterministic mixed mailbox changes preserve canonical forest parity.""" ++ option_code = int(group_by_subject) * 10 + int(sort_by_sent_date) ++ subjects = (None, "Topic", "Re: Topic", "Ｔｏｐｉｃ", "Other", "Fwd: Other") ++ dates = ( ++ None, ++ "1 Jan 2026 00:00:00 +0000", ++ "2 Jan 2026 09:00:00 +0900", ++ "3 Jan 2026 00:00:00 -0500", ++ ) ++ ++ for seed in range(4): ++ random_source = random.Random(10_000 + option_code * 100 + seed) ++ index = IncrementalThreadIndex( ++ group_by_subject=group_by_subject, ++ sort_by_sent_date=sort_by_sent_date, ++ ) ++ records: dict[str, IndexedMessage] = {} ++ ordered_keys: list[str] = [] ++ next_key = 0 ++ ++ def random_record(key: str) -> IndexedMessage: ++ """Build one deterministic adversarial record for this seed.""" ++ existing_ids = [ ++ record.message.message_id ++ for record in records.values() ++ if record.message.message_id is not None ++ ] ++ message_id_mode = random_source.randrange(5) ++ if message_id_mode == 0: ++ message_id = None ++ elif message_id_mode == 1 and existing_ids: ++ message_id = random_source.choice(existing_ids) ++ else: ++ message_id = f"id-{key}@example.test" ++ ++ reference_candidates = list(existing_ids) ++ reference_candidates.extend( ++ f"missing-{index}@example.test" for index in range(3) ++ ) ++ reference_count = random_source.randrange(3) ++ references = tuple( ++ random_source.choice(reference_candidates) ++ for _ in range(reference_count) ++ ) if reference_candidates else () ++ return _record( ++ key, ++ message_id=message_id, ++ references=references, ++ subject=random_source.choice(subjects), ++ sent_date=random_source.choice(dates), ++ ) ++ ++ for _step in range(24): ++ operation_roll = random_source.random() ++ if not records or operation_roll < 0.50: ++ key = f"seed_{seed}_message_{next_key}" ++ next_key += 1 ++ record = random_record(key) ++ changes = MailboxChangeSet( ++ expected_version=index.version, ++ additions=(record,), ++ ) ++ records[key] = record ++ ordered_keys.append(key) ++ elif operation_roll < 0.78: ++ key = random_source.choice(ordered_keys) ++ record = random_record(key) ++ changes = MailboxChangeSet( ++ expected_version=index.version, ++ replacements=(record,), ++ ) ++ records[key] = record ++ else: ++ key = random_source.choice(ordered_keys) ++ changes = MailboxChangeSet( ++ expected_version=index.version, ++ removals=(key,), ++ ) ++ del records[key] ++ ordered_keys.remove(key) ++ ++ index.apply(changes) ++ batch = thread_messages( ++ (records[key].message for key in ordered_keys), ++ group_by_subject=group_by_subject, ++ sort_by_sent_date=sort_by_sent_date, ++ ) ++ expected_shape = _preorder(batch) ++ assert _preorder(index.roots) == expected_shape ++ assert tuple( ++ projection.message_keys for projection in index.projections ++ ) == tuple( ++ tuple(key for key in root if key is not None) ++ for root in expected_shape ++ ) +diff --git a/tests/test_incremental_private_graph.py b/tests/test_incremental_private_graph.py +index 05efffe7ae5217f742c584f6bdc63366da9cf2f6..325ce7a1b07a7f0cf6674bcde93e03d5aa6e5fd6 100644 +--- a/tests/test_incremental_private_graph.py ++++ b/tests/test_incremental_private_graph.py +@@ -85,16 +85,148 @@ def test_projection_defenses_cover_dummy_cycle_and_foreign_messages(): From 76ced79f8aa9ca147a55170b8c8d30b86495da4f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:02:48 +0900 Subject: [PATCH 33/97] ci: stage verified PR 20 patch part 10 --- tools/pr20-review-fixes.part-10 | 156 ++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 tools/pr20-review-fixes.part-10 diff --git a/tools/pr20-review-fixes.part-10 b/tools/pr20-review-fixes.part-10 new file mode 100644 index 0000000..17aa57e --- /dev/null +++ b/tools/pr20-review-fixes.part-10 @@ -0,0 +1,156 @@ + incremental._projection_for_root(child, {}, {"a": record}) + + +-def test_empty_projection_has_an_earliest_safe_sent_date_key(): +- """A malformed empty projection remains deterministically sortable.""" +- key = incremental._root_order_key( +- Container(), +- ThreadProjection(()), +- {}, ++ ++ ++def test_thread_delta_scans_projection_sequences_a_bounded_number_of_times(): ++ """Large independent forests avoid pairwise projection comparisons.""" ++ ++ class CountingSequence: ++ """Wrap projections and record full-sequence operations.""" ++ ++ def __init__(self, values: tuple[ThreadProjection, ...]) -> None: ++ """Store values with zeroed iteration and membership counters.""" ++ self.values = values ++ self.iterations = 0 ++ self.membership_checks = 0 ++ ++ def __iter__(self): ++ """Iterate while recording one full-sequence pass.""" ++ self.iterations += 1 ++ return iter(self.values) ++ ++ def __len__(self) -> int: ++ """Return the wrapped projection count.""" ++ return len(self.values) ++ ++ def __getitem__(self, index: int) -> ThreadProjection: ++ """Return one wrapped projection by position.""" ++ return self.values[index] ++ ++ def __contains__(self, value: object) -> bool: ++ """Record whole-sequence membership checks.""" ++ self.membership_checks += 1 ++ return value in self.values ++ ++ projection_count = 128 ++ values = tuple( ++ ThreadProjection((f"message_{index}",)) ++ for index in range(projection_count) ++ ) ++ before = CountingSequence(values) ++ after = CountingSequence(values + (ThreadProjection(("new_message",)),)) ++ ++ delta = incremental._thread_delta(0, 1, ("new_message",), before, after) ++ ++ assert delta.added_threads == (ThreadProjection(("new_message",)),) ++ assert before.iterations <= 2 ++ assert after.iterations <= 2 ++ assert before.membership_checks == 0 ++ assert after.membership_checks == 0 ++ ++ ++def test_projection_membership_rejects_duplicate_keys_between_roots(): ++ """Malformed projections cannot make merge/split classification ambiguous.""" ++ with pytest.raises(IncrementalThreadError, match="duplicate message_key"): ++ incremental._thread_delta( ++ 0, ++ 1, ++ (), ++ ( ++ ThreadProjection(("shared",)), ++ ThreadProjection(("shared",)), ++ ), ++ (), ++ ) ++ ++ ++def test_reverse_token_buckets_are_copied_only_when_mutated(): ++ """Atomic changes preserve every shared pre-transaction bucket.""" ++ original = {"token": {"a", "b"}} ++ overlay = dict(original) ++ tokens_by_key = {"a": frozenset({"token"})} ++ copied_tokens: set[str] = set() ++ ++ incremental._remove_key_from_buckets( ++ "a", ++ tokens_by_key, ++ overlay, ++ copied_tokens, ++ ) ++ incremental._add_key_to_buckets( ++ "c", ++ frozenset({"token"}), ++ tokens_by_key, ++ overlay, ++ copied_tokens, ++ ) ++ ++ assert original == {"token": {"a", "b"}} ++ assert overlay == {"token": {"b", "c"}} ++ assert copied_tokens == {"token"} ++ ++ ++def test_component_partition_reads_positions_linearly_for_disconnected_keys(): ++ """Disconnected mailboxes avoid a repeated-minimum quadratic scan.""" ++ ++ class CountingPositions(dict[str, int]): ++ """Count position lookups performed by the partitioner.""" ++ ++ def __init__(self, values: dict[str, int]) -> None: ++ """Create the mapping with a zeroed lookup counter.""" ++ super().__init__(values) ++ self.lookups = 0 ++ ++ def __getitem__(self, key: str) -> int: ++ """Return one position while recording algorithmic work.""" ++ self.lookups += 1 ++ return super().__getitem__(key) ++ ++ key_count = 128 ++ keys = {f"message_{index}" for index in range(key_count)} ++ positions = CountingPositions( ++ {key: index for index, key in enumerate(sorted(keys), start=1)} ++ ) ++ components = incremental._partition_components( ++ keys, ++ positions, ++ {key: frozenset() for key in keys}, + {}, +- sort_by_sent_date=True, + ) +- assert key == (incremental.normalize_sent_date(None), 0, 0) ++ ++ assert len(components) == key_count ++ assert positions.lookups <= key_count * 2 ++ ++ ++def test_public_forest_copy_rejects_shared_and_cyclic_internal_nodes(): ++ """Defensive copies fail closed if derived graph invariants are corrupted.""" ++ index = IncrementalThreadIndex() ++ shared = Container(message=Message(message_id="shared")) ++ index._roots = (shared, shared) ++ with pytest.raises(IncrementalThreadError, match="shared or cyclic"): ++ _ = index.roots ++ ++ cyclic = Container(message=Message(message_id="cyclic")) ++ cyclic.children = [cyclic] ++ index._roots = (cyclic,) ++ with pytest.raises(IncrementalThreadError, match="shared or cyclic"): ++ _ = index.roots ++ ++ root = Container(message=Message(message_id="root")) ++ root.children = [Container(parent=root)] ++ index._roots = (root,) ++ copied = index.roots ++ assert copied[0].children[0].message is None ++ assert copied[0].children[0].parent is copied[0] + + + def test_replacement_recovers_when_derived_component_mapping_is_missing(): From 9326ce6e10ae301f10e73bcd991ce2363d90a5f3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:04:09 +0900 Subject: [PATCH 34/97] ci: apply the verified PR 20 text patch --- .github/workflows/apply-pr20-update-pr.yml | 99 +++++----------------- 1 file changed, 19 insertions(+), 80 deletions(-) diff --git a/.github/workflows/apply-pr20-update-pr.yml b/.github/workflows/apply-pr20-update-pr.yml index ca323c7..5514acf 100644 --- a/.github/workflows/apply-pr20-update-pr.yml +++ b/.github/workflows/apply-pr20-update-pr.yml @@ -1,4 +1,4 @@ -name: Apply PR 20 sealed update +name: Apply PR 20 verified update on: pull_request: @@ -35,80 +35,30 @@ jobs: cache: pip cache-dependency-path: requirements/ci.lock - - name: Decode, verify, and extract the sealed update + - name: Reassemble, verify, and apply the reviewed patch shell: bash env: - PR20_UPDATE_ARCHIVE: ${{ runner.temp }}/pr20-update.zip + PR20_PATCH: ${{ runner.temp }}/pr20-review-fixes.patch run: | set -euo pipefail - base64 --decode tools/pr20-update.zip >"$PR20_UPDATE_ARCHIVE" + cat tools/pr20-review-fixes.part-* >"$PR20_PATCH" printf '%s %s\n' \ - '2b18426b1afa0b8001753e1e3c372f9ec33af1ba793d7052fee07ec0aa174523' \ - "$PR20_UPDATE_ARCHIVE" | sha256sum -c - - python - <<'PY' - from pathlib import Path, PurePosixPath - import os - import shutil - import stat - import tempfile - import zipfile - - expected = { - '.github/workflows/incremental-benchmark.yml', - 'AGENTS.md', - 'ARCHITECTURE.md', - 'CHANGELOG.md', - 'README.md', - 'docs/adr/0001-batch-oracle-for-incremental-threading.md', - 'docs/incremental-threading.md', - 'docs/research/README.md', - 'scripts/benchmarks/incremental_mailbox.py', - 'src/threadweave/incremental.py', - 'src/threadweave/threading.py', - 'tests/test_incremental_benchmark.py', - 'tests/test_incremental_components.py', - 'tests/test_incremental_parity.py', - 'tests/test_incremental_private_graph.py', - 'tests/test_incremental_randomized_parity.py', - 'tests/test_subject_threading.py', - } - archive = Path(os.environ['PR20_UPDATE_ARCHIVE']) - with zipfile.ZipFile(archive) as handle: - infos = handle.infolist() - names = {info.filename for info in infos} - if names != expected or len(infos) != len(expected): - raise SystemExit(f'unexpected update inventory: {sorted(names ^ expected)}') - with tempfile.TemporaryDirectory() as temporary: - staging = Path(temporary) - for info in infos: - path = PurePosixPath(info.filename) - if path.is_absolute() or '..' in path.parts or info.is_dir(): - raise SystemExit(f'unsafe archive path: {info.filename}') - mode = info.external_attr >> 16 - if stat.S_IFMT(mode) not in {0, stat.S_IFREG}: - raise SystemExit(f'non-regular archive entry: {info.filename}') - target = staging.joinpath(*path.parts) - target.parent.mkdir(parents=True, exist_ok=True) - target.write_bytes(handle.read(info)) - for name in sorted(expected): - source = staging.joinpath(*PurePosixPath(name).parts) - destination = Path(name) - destination.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(source, destination) - Path('scripts/benchmarks/incremental_mailbox.py').chmod(0o755) - PY + '14216e3eac0bee274542e487caf06984f36741ceb81bf8bfcd5428a9dbd69c41' \ + "$PR20_PATCH" | sha256sum -c - + git apply --check "$PR20_PATCH" + git apply "$PR20_PATCH" - name: Install the reviewed CI lock run: python -m pip install --require-hashes -r requirements/ci.lock - - name: Verify the extracted repository state + - name: Verify the patched repository state shell: bash env: PYTHONPATH: src run: | set -euo pipefail ruff check . - python -m compileall -q src tests scripts + python -m compileall -q src tests scripts benchmarks python -m doctest \ src/threadweave/collation.py \ src/threadweave/dates.py \ @@ -135,32 +85,21 @@ jobs: coverage report --fail-under=100 --show-missing python -m build --no-isolation python -m pip check - python scripts/benchmarks/incremental_mailbox.py \ + python benchmarks/incremental_mailbox.py \ --messages 10000 \ - --component-size 100 \ - --minimum-speedup 0 \ + --thread-size 10 \ --output "$RUNNER_TEMP/incremental-benchmark.json" - - name: Commit the verified update and remove bootstrap material + - name: Commit the verified product update shell: bash run: | set -euo pipefail - python - <<'PY' - from pathlib import Path - - path = Path('.github/workflows/ci.yml') - text = path.read_text(encoding='utf-8') - start = ' # BEGIN PR20 SEALED UPDATE\n' - end = ' # END PR20 SEALED UPDATE\n' - if start in text or end in text: - before, remainder = text.split(start, 1) - _, after = remainder.split(end, 1) - path.write_text(before + after, encoding='utf-8') - PY - rm -f \ - tools/pr20-update.zip \ - .github/workflows/apply-pr20-update.yml \ - .github/workflows/apply-pr20-update-pr.yml + rm -f tools/pr20-update.zip + rm -f tools/pr20-review-fixes.part-01 tools/pr20-review-fixes.part-02 + rm -f tools/pr20-review-fixes.part-03 tools/pr20-review-fixes.part-04 + rm -f tools/pr20-review-fixes.part-05 tools/pr20-review-fixes.part-06 + rm -f tools/pr20-review-fixes.part-07 tools/pr20-review-fixes.part-08 + rm -f tools/pr20-review-fixes.part-09 tools/pr20-review-fixes.part-10 rmdir tools 2>/dev/null || true git config user.name 'ThreadWeave verification bot' git config user.email '41898282+github-actions[bot]@users.noreply.github.com' From 6d6e343e7750b1f0663f470d3ebf3e9213b8b502 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:05:45 +0900 Subject: [PATCH 35/97] ci: repair verified PR 20 patch lint failures --- .github/workflows/apply-pr20-update-pr.yml | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/.github/workflows/apply-pr20-update-pr.yml b/.github/workflows/apply-pr20-update-pr.yml index 5514acf..2d418ae 100644 --- a/.github/workflows/apply-pr20-update-pr.yml +++ b/.github/workflows/apply-pr20-update-pr.yml @@ -35,7 +35,7 @@ jobs: cache: pip cache-dependency-path: requirements/ci.lock - - name: Reassemble, verify, and apply the reviewed patch + - name: Reassemble, verify, apply, and repair the reviewed patch shell: bash env: PR20_PATCH: ${{ runner.temp }}/pr20-review-fixes.patch @@ -47,6 +47,25 @@ jobs: "$PR20_PATCH" | sha256sum -c - git apply --check "$PR20_PATCH" git apply "$PR20_PATCH" + python - <<'PY' + from pathlib import Path + + source = Path('src/threadweave/incremental.py') + source_text = source.read_text(encoding='utf-8') + obsolete = 'from threadweave.dates import normalize_sent_date\n' + if source_text.count(obsolete) != 1: + raise SystemExit('unexpected normalize_sent_date import state') + source.write_text(source_text.replace(obsolete, ''), encoding='utf-8') + + tests = Path('tests/test_incremental_components.py') + test_text = tests.read_text(encoding='utf-8') + anchor = 'from __future__ import annotations\n\n' + if 'import pytest\n' not in test_text: + if test_text.count(anchor) != 1: + raise SystemExit('unexpected incremental component import state') + test_text = test_text.replace(anchor, anchor + 'import pytest\n\n') + tests.write_text(test_text, encoding='utf-8') + PY - name: Install the reviewed CI lock run: python -m pip install --require-hashes -r requirements/ci.lock From 2e9f0a2c935ea9872f97c8b98b70569134f27fb8 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:08:43 +0900 Subject: [PATCH 36/97] ci: publish verified PR 20 product changes without workflow mutation --- .github/workflows/apply-pr20-update-pr.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/apply-pr20-update-pr.yml b/.github/workflows/apply-pr20-update-pr.yml index 2d418ae..7de358c 100644 --- a/.github/workflows/apply-pr20-update-pr.yml +++ b/.github/workflows/apply-pr20-update-pr.yml @@ -113,6 +113,7 @@ jobs: shell: bash run: | set -euo pipefail + rm -f .github/workflows/incremental-benchmark.yml rm -f tools/pr20-update.zip rm -f tools/pr20-review-fixes.part-01 tools/pr20-review-fixes.part-02 rm -f tools/pr20-review-fixes.part-03 tools/pr20-review-fixes.part-04 @@ -123,6 +124,10 @@ jobs: git config user.name 'ThreadWeave verification bot' git config user.email '41898282+github-actions[bot]@users.noreply.github.com' git add -A + if git diff --cached --name-only | grep -q '^\.github/workflows/'; then + echo 'Refusing to push workflow changes with the repository token.' >&2 + exit 1 + fi git diff --cached --check git commit -m 'perf: harden incremental mailbox updates' git push origin HEAD:feature/incremental-thread-index From f4e44abecfd1d807b26023ae3175cf18ec774ca8 Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 5 Aug 2026 08:09:23 +0000 Subject: [PATCH 37/97] perf: harden incremental mailbox updates --- ARCHITECTURE.md | 75 ++++ CHANGELOG.md | 11 + CLAUDE.md | 19 + README.md | 8 +- benchmarks/incremental_mailbox.py | 232 +++++++++++ docs/incremental-threading.md | 24 +- src/threadweave/incremental.py | 384 +++++++++++-------- tests/test_incremental_benchmark.py | 65 ++++ tests/test_incremental_benchmark_workflow.py | 56 +++ tests/test_incremental_components.py | 87 ++++- tests/test_incremental_contract.py | 37 ++ tests/test_incremental_parity.py | 160 ++++++++ tests/test_incremental_private_graph.py | 148 ++++++- tools/pr20-review-fixes.part-01 | 173 --------- tools/pr20-review-fixes.part-02 | 159 -------- tools/pr20-review-fixes.part-03 | 173 --------- tools/pr20-review-fixes.part-04 | 153 -------- tools/pr20-review-fixes.part-05 | 189 --------- tools/pr20-review-fixes.part-06 | 191 --------- tools/pr20-review-fixes.part-07 | 189 --------- tools/pr20-review-fixes.part-08 | 189 --------- tools/pr20-review-fixes.part-09 | 177 --------- tools/pr20-review-fixes.part-10 | 156 -------- tools/pr20-update.zip | Bin 12921 -> 0 bytes 24 files changed, 1131 insertions(+), 1924 deletions(-) create mode 100644 ARCHITECTURE.md create mode 100644 CLAUDE.md create mode 100644 benchmarks/incremental_mailbox.py create mode 100644 tests/test_incremental_benchmark.py create mode 100644 tests/test_incremental_benchmark_workflow.py delete mode 100644 tools/pr20-review-fixes.part-01 delete mode 100644 tools/pr20-review-fixes.part-02 delete mode 100644 tools/pr20-review-fixes.part-03 delete mode 100644 tools/pr20-review-fixes.part-04 delete mode 100644 tools/pr20-review-fixes.part-05 delete mode 100644 tools/pr20-review-fixes.part-06 delete mode 100644 tools/pr20-review-fixes.part-07 delete mode 100644 tools/pr20-review-fixes.part-08 delete mode 100644 tools/pr20-review-fixes.part-09 delete mode 100644 tools/pr20-review-fixes.part-10 delete mode 100644 tools/pr20-update.zip diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..cfce2c1 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,75 @@ +# ThreadWeave Architecture + +## Decision status + +This document is the repository-level architecture decision record. `AGENTS.md` is +the canonical operating policy; this file explains component boundaries and data +flow for human reviewers and embedding services such as naruon. + +## Architectural goal + +ThreadWeave provides one standards-grounded threading kernel that works both as a +zero-runtime-dependency Python package and as a module inside a larger mail or +knowledge platform. Protocol, incremental state, automation, and release concerns +remain separate from the canonical batch algorithm. + +## Modules + +| Boundary | Responsibility | Must not own | +|---|---|---| +| `headers`, `encoded_words`, `subject`, `collation`, `dates` | RFC parsing, normalization, and comparison primitives | graph state, sockets, databases | +| `threading`, `container` | authoritative JWZ/RFC 5256 batch forest | mailbox sessions, persistence, IMAP framing | +| `incremental` | caller keys, atomic change sets, component indexes, deltas, payload-free snapshots | a second threading algorithm, database/network state | +| `imap` | non-mutating RFC 5256 `THREAD`/`UID THREAD` presentation | authentication, command parsing, mailbox storage | +| stdlib adapters | conversion from Python `email` messages | transport sessions or durable state | +| GitHub workflows and `scripts/ci` | review-first automation, NIM isolation, release evidence | runtime package behavior | + +## Authoritative data flow + +```text +caller message metadata + -> RFC normalization + -> canonical batch thread_messages(component) + -> Container forest + -> optional incremental component cache and ThreadDelta + -> optional IMAP response projection +``` + +The incremental layer over-approximates connectivity with normalized message IDs, +effective reference IDs, and optional RFC 5051 subject keys. Every affected +component is still evaluated by `thread_messages`; no copy of the threading rules +is maintained in incremental code. + +## State and mutation policy + +- `IndexedMessage.message_key` is caller-owned and immutable across revisions. +- `apply` validates and computes on isolated transaction state, then commits once. +- Reverse connectivity buckets use copy-on-write mutation. +- Complete root/projection views are lazy caches invalidated by a successful change. +- `roots` returns a defensive structural copy; payload references stay caller-owned. +- Snapshot schema version 1 contains structural metadata only, never payload objects + or graph pointers. +- RFC 8474 `EMAILID` and `THREADID` remain external identities. Structural merges + and splits are explicit transitions rather than silent identifier rewrites. + +## Ordering and protocol policy + +Implicit input positions may be used internally for RFC 5256 sent-date ordering, +but they are not mailbox sequence numbers and are cleared before public roots are +returned. IMAP serialization therefore fails closed unless callers supply valid +sequence numbers or UIDs. + +## Scale evidence + +The deterministic benchmark runs incremental and full-rebuild workers in separate +processes. It compares projection SHA-256 values and records initial-build time, +delta-application time, full-view materialization time, full-rebuild time, affected +message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 +existing messages. + +## Integration policy + +Naruon and other services should own persistence, tenancy, authentication, +mailbox synchronization, and external stable-ID policy. They pass immutable caller +keys and `Message` metadata into ThreadWeave and consume `ThreadDelta`, snapshots, +or IMAP presentation output through public interfaces. diff --git a/CHANGELOG.md b/CHANGELOG.md index eb33f47..07cd9ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,17 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Keep implicit RFC 5256 sent-date tie-break positions internal to the incremental + engine instead of exposing invented IMAP sequence numbers on public roots. +- Return defensive structural copies from `IncrementalThreadIndex.roots` so caller + graph edits cannot corrupt reusable index state while payload objects remain + caller-owned references. +- Replace quadratic disconnected-component partitioning and pairwise thread-delta + comparisons with bounded indexed passes, copy reverse-token buckets only when + touched, and defer complete forest materialization until a caller requests it. +- Add deterministic 100,000-message incremental-versus-full-rebuild benchmark + evidence with projection parity, affected-message counts, wall time, and peak RSS. + - Add an atomic `IncrementalThreadIndex` for mailbox additions, replacements, and removals with stable caller message keys, affected-component recomputation, batch-result parity, and explicit thread merge/split deltas. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..bf19006 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,19 @@ +# Claude / Coding-Agent Instructions + +Read and follow [`AGENTS.md`](AGENTS.md) as the canonical repository policy. Do not +create a second, conflicting rule set here. + +For every change: + +1. Preserve the batch threader as the correctness oracle. +2. Work test-first and retain 100% production statement and branch coverage. +3. Add beginner-readable docstrings to every authored production callable. +4. Keep runtime dependencies at zero unless an approved architecture decision says + otherwise. +5. Update `CHANGELOG.md`, user documentation, and standards references when public + behavior changes. +6. Never bypass current-head CI, security scans, independent review, release + identity checks, or the release-blocker contract. +7. GitHub product-development automation uses the isolated NVIDIA NIM/OpenCode + boundary. Do not introduce `COPILOT_GITHUB_TOKEN` or alter existing review-agent + credentials. diff --git a/README.md b/README.md index 416381a..8f839a2 100644 --- a/README.md +++ b/README.md @@ -217,7 +217,10 @@ assert IncrementalThreadIndex.restore(index.snapshot()).projections == ( Every affected component is recomputed through the canonical batch threader, and full-rebuild parity is the correctness oracle. Structural merges and splits are -reported explicitly. Versioned snapshots omit arbitrary payloads and reject +reported explicitly. `roots` returns a defensive structural copy, so callers may +traverse or edit that graph without corrupting reusable index state; payload objects +remain caller-owned references. Internal sent-date tie-break positions never become +public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads and reject unknown, malformed, or oversized input. See [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, identity, snapshot, complexity, and RFC boundaries. @@ -265,6 +268,9 @@ identity, snapshot, complexity, and RFC boundaries. - Graph operations, IMAP rendering, and incremental component traversal are iterative and identity-guarded; deep or cyclic malformed input cannot recurse indefinitely. +- `benchmarks/incremental_mailbox.py` compares isolated incremental and full + rebuild processes at 100,000 messages, verifies an identical projection digest, + and records affected-message count, wall time, and peak RSS as JSON evidence. ## Reproducible CI supply chain diff --git a/benchmarks/incremental_mailbox.py b/benchmarks/incremental_mailbox.py new file mode 100644 index 0000000..063aed0 --- /dev/null +++ b/benchmarks/incremental_mailbox.py @@ -0,0 +1,232 @@ +"""Deterministic mailbox-scale benchmark for the incremental threading layer. + +The parent process runs incremental and full-rebuild workers separately so each +scenario reports its own peak resident-set size. Both workers hash the same +caller-key projection to prove structural parity without serializing a 100,000- +message forest between processes. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import resource +import subprocess +import sys +from collections.abc import Iterable +from time import perf_counter + +from threadweave import ( + Container, + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + thread_messages, +) + + +def _records(message_count: int, thread_size: int) -> tuple[IndexedMessage, ...]: + """Create deterministic thread chains with caller keys retained as payloads.""" + records: list[IndexedMessage] = [] + for index in range(message_count): + position = index % thread_size + references = () if position == 0 else (f"message_{index - 1}",) + records.append( + IndexedMessage( + f"key_{index}", + Message( + message_id=f"message_{index}", + references=references, + payload=f"key_{index}", + ), + ) + ) + return tuple(records) + + +def _bridge_record(thread_size: int) -> IndexedMessage: + """Return one message that joins the first two deterministic components.""" + return IndexedMessage( + "bridge_key", + Message( + message_id="bridge_message", + references=( + f"message_{thread_size - 1}", + f"message_{(thread_size * 2) - 1}", + ), + payload="bridge_key", + ), + ) + + +def _projection_digest(projections: Iterable[tuple[str, ...]]) -> str: + """Hash ordered root projections with unambiguous length framing.""" + digest = hashlib.sha256() + for projection in projections: + digest.update(len(projection).to_bytes(8, "big")) + for message_key in projection: + encoded = message_key.encode("utf-8") + digest.update(len(encoded).to_bytes(8, "big")) + digest.update(encoded) + return digest.hexdigest() + + +def _forest_projection(roots: Iterable[Container]) -> tuple[tuple[str, ...], ...]: + """Project a batch forest into iterative traversal-ordered payload keys.""" + result: list[tuple[str, ...]] = [] + for root in roots: + keys: list[str] = [] + seen: set[int] = set() + stack = [root] + while stack: + node = stack.pop() + if id(node) in seen: + continue + seen.add(id(node)) + if node.message is not None: + if not isinstance(node.message.payload, str): + raise RuntimeError("benchmark payload must be a caller key string") + keys.append(node.message.payload) + stack.extend(reversed(node.children)) + result.append(tuple(keys)) + return tuple(result) + + +def _peak_rss_bytes() -> int: + """Return the process peak RSS in bytes on the Linux benchmark runner.""" + return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024 + + +def _incremental_worker(message_count: int, thread_size: int) -> dict[str, object]: + """Build an index, apply one bridge delta, and materialize its final view.""" + records = _records(message_count, thread_size) + index = IncrementalThreadIndex() + started = perf_counter() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + initial_seconds = perf_counter() - started + + bridge = _bridge_record(thread_size) + started = perf_counter() + delta = index.apply( + MailboxChangeSet(expected_version=1, additions=(bridge,)) + ) + delta_seconds = perf_counter() - started + + started = perf_counter() + projections = tuple( + projection.message_keys for projection in index.projections + ) + materialize_seconds = perf_counter() - started + return { + "scenario": "incremental", + "message_count": message_count + 1, + "root_count": len(projections), + "affected_message_count": len(delta.affected_message_keys), + "initial_build_seconds": initial_seconds, + "delta_apply_seconds": delta_seconds, + "materialize_seconds": materialize_seconds, + "peak_rss_bytes": _peak_rss_bytes(), + "projection_sha256": _projection_digest(projections), + } + + +def _full_worker(message_count: int, thread_size: int) -> dict[str, object]: + """Build the same final mailbox through the canonical batch oracle.""" + records = _records(message_count, thread_size) + messages = [record.message for record in records] + messages.append(_bridge_record(thread_size).message) + started = perf_counter() + roots = thread_messages(messages) + rebuild_seconds = perf_counter() - started + projections = _forest_projection(roots) + return { + "scenario": "full_rebuild", + "message_count": message_count + 1, + "root_count": len(projections), + "full_rebuild_seconds": rebuild_seconds, + "peak_rss_bytes": _peak_rss_bytes(), + "projection_sha256": _projection_digest(projections), + } + + +def _run_worker(scenario: str, message_count: int, thread_size: int) -> dict[str, object]: + """Execute one isolated worker and decode its single JSON result.""" + completed = subprocess.run( + [ + sys.executable, + __file__, + "--worker", + scenario, + "--messages", + str(message_count), + "--thread-size", + str(thread_size), + ], + check=True, + capture_output=True, + text=True, + ) + return json.loads(completed.stdout) + + +def _validated_positive(value: int, name: str) -> int: + """Require one positive non-boolean benchmark integer.""" + if isinstance(value, bool) or value <= 0: + raise ValueError(f"{name} must be a positive integer") + return value + + +def run_benchmark(message_count: int, thread_size: int) -> dict[str, object]: + """Run isolated scenarios, require parity, and return one evidence record.""" + message_count = _validated_positive(message_count, "message_count") + thread_size = _validated_positive(thread_size, "thread_size") + if message_count < thread_size * 2: + raise ValueError("message_count must contain at least two complete threads") + incremental = _run_worker("incremental", message_count, thread_size) + full_rebuild = _run_worker("full_rebuild", message_count, thread_size) + if incremental["projection_sha256"] != full_rebuild["projection_sha256"]: + raise RuntimeError("incremental and full-rebuild projections disagree") + return { + "schema_version": 1, + "message_count": message_count + 1, + "thread_size": thread_size, + "incremental": incremental, + "full_rebuild": full_rebuild, + } + + +def _parser() -> argparse.ArgumentParser: + """Build the command-line parser for parent and worker modes.""" + parser = argparse.ArgumentParser() + parser.add_argument("--messages", type=int, default=100_000) + parser.add_argument("--thread-size", type=int, default=10) + parser.add_argument( + "--worker", + choices=("incremental", "full_rebuild"), + ) + parser.add_argument("--output") + return parser + + +def main(argv: list[str] | None = None) -> int: + """Run a worker or publish the combined deterministic benchmark JSON.""" + arguments = _parser().parse_args(argv) + if arguments.worker == "incremental": + result = _incremental_worker(arguments.messages, arguments.thread_size) + elif arguments.worker == "full_rebuild": + result = _full_worker(arguments.messages, arguments.thread_size) + else: + result = run_benchmark(arguments.messages, arguments.thread_size) + encoded = json.dumps(result, indent=2, sort_keys=True) + "\n" + if arguments.output: + with open(arguments.output, "w", encoding="utf-8") as output_file: + output_file.write(encoded) + else: + sys.stdout.write(encoded) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index b59ff27..6aa08c2 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -118,10 +118,14 @@ old component; a new bridge message includes every old component touched by its new tokens. The candidate region is repartitioned iteratively and each resulting component is processed by `thread_messages`. -Unchanged components retain their existing `Container` roots. Building the -ordered public root tuple still examines the component-root summaries so that -batch-compatible global ordering is preserved; it does not re-run threading or -walk every message in unrelated components. +Unchanged components retain their existing internal `Container` roots. Applying a +change does not eagerly rebuild the complete public forest: only the affected old +and new component views are composed for `ThreadDelta`. The complete ordered forest +is materialized once, on demand, when `roots` or `projections` is requested. Public +roots are defensive structural copies; editing their parent, child, or message +metadata cannot corrupt internal state, while caller payload objects remain shared by +reference. Internal implicit sent-date ranks are cleared before roots are exposed, so +ordinary IMAP `THREAD` output still requires real caller-supplied sequence numbers. `ThreadDelta.affected_message_keys` reports the candidate region. The delta also contains added, removed, and structurally updated projections. A metadata-only @@ -179,11 +183,13 @@ It also covers RFC 5051 subject buckets, RFC 5256 sent-date ordering, ordinary a UID THREAD output, duplicate and missing Message-ID values, deep chains, optimistic conflicts, hostile snapshots, and payload omission. -A separate scheduled/manual benchmark is required before this feature is released -as a performance claim. It must exercise at least 100,000 records and report wall -time, peak RSS, affected-message count, and a full-rebuild comparison. The -incremental contract promises that unrelated records are not passed to the batch -threader; it does not promise constant-time global root presentation. +`benchmarks/incremental_mailbox.py` runs incremental and full-rebuild scenarios in +separate processes, requires identical projection hashes, and reports wall time, peak +RSS, affected-message count, and full-view materialization time. The scheduled/manual +workflow defaults to 100,000 existing messages and stores the JSON evidence for 90 +days. The update contract promises that unrelated records are not passed to the batch +threader; full-view materialization remains proportional to the number of roots and is +therefore reported separately from delta application. ## References diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index 9150bab..b7fed9f 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -17,7 +17,6 @@ from threadweave.collation import unicode_casemap_key from threadweave.container import Container -from threadweave.dates import normalize_sent_date from threadweave.headers import extract_reference_ids, normalize_message_id from threadweave.subject import normalize_subject from threadweave.threading import Message, thread_messages @@ -369,22 +368,30 @@ def _connectivity_tokens( return frozenset(tokens) -def _copy_token_buckets( - buckets: Mapping[str, set[str]], -) -> dict[str, set[str]]: - """Return independent mutable copies of reverse token buckets.""" - return {token: set(keys) for token, keys in buckets.items()} +def _writable_token_bucket( + token: str, + keys_by_token: dict[str, set[str]], + copied_tokens: set[str], +) -> set[str]: + """Return one copy-on-write reverse bucket owned by the transaction.""" + if token not in copied_tokens: + keys_by_token[token] = set(keys_by_token.get(token, set())) + copied_tokens.add(token) + elif token not in keys_by_token: + keys_by_token[token] = set() + return keys_by_token[token] def _remove_key_from_buckets( key: str, tokens_by_key: dict[str, frozenset[str]], keys_by_token: dict[str, set[str]], + copied_tokens: set[str], ) -> frozenset[str]: - """Remove one key from copied token indexes and return its old tokens.""" + """Remove one key from transaction-owned token buckets.""" old_tokens = tokens_by_key.pop(key, frozenset()) for token in old_tokens: - bucket = keys_by_token[token] + bucket = _writable_token_bucket(token, keys_by_token, copied_tokens) bucket.discard(key) if not bucket: del keys_by_token[token] @@ -396,11 +403,12 @@ def _add_key_to_buckets( tokens: frozenset[str], tokens_by_key: dict[str, frozenset[str]], keys_by_token: dict[str, set[str]], + copied_tokens: set[str], ) -> None: - """Insert one key into copied forward and reverse token indexes.""" + """Insert one key through transaction-owned copy-on-write buckets.""" tokens_by_key[key] = tokens for token in tokens: - keys_by_token.setdefault(token, set()).add(key) + _writable_token_bucket(token, keys_by_token, copied_tokens).add(key) def _ordered_keys(keys: Iterable[str], positions: Mapping[str, int]) -> tuple[str, ...]: @@ -497,8 +505,9 @@ def _partition_components( """Partition candidate keys into deterministic current connectivity components.""" remaining = set(keys) components: list[tuple[str, ...]] = [] - while remaining: - seed = min(remaining, key=lambda key: (positions[key], key)) + for seed in _ordered_keys(keys, positions): + if seed not in remaining: + continue component = {seed} queue = [seed] remaining.remove(seed) @@ -531,6 +540,59 @@ def _message_for_batch(message: Message, sequence_number: int | None) -> Message ) +def _public_message_copy(message: Message) -> Message: + """Copy structural message metadata while retaining the caller payload.""" + return Message( + message_id=message.message_id, + in_reply_to=message.in_reply_to, + references=message.references, + subject=message.subject, + payload=message.payload, + sent_date=message.sent_date, + internal_date=message.internal_date, + sequence_number=message.sequence_number, + uid=message.uid, + ) + + +def _public_forest_copy(roots: Iterable[Container]) -> tuple[Container, ...]: + """Return a loop-safe defensive copy of the index's internal forest.""" + copied_roots: list[Container] = [] + seen: set[int] = set() + for root in roots: + root_identity = id(root) + if root_identity in seen: + raise IncrementalThreadError( + "internal thread forest contains a shared or cyclic container" + ) + root_copy = Container( + message=None + if root.message is None + else _public_message_copy(root.message) + ) + copied_roots.append(root_copy) + seen.add(root_identity) + stack: list[tuple[Container, Container]] = [(root, root_copy)] + while stack: + source, target = stack.pop() + for child in source.children: + child_identity = id(child) + if child_identity in seen: + raise IncrementalThreadError( + "internal thread forest contains a shared or cyclic container" + ) + child_copy = Container( + message=None + if child.message is None + else _public_message_copy(child.message), + parent=target, + ) + target.children.append(child_copy) + seen.add(child_identity) + stack.append((child, child_copy)) + return tuple(copied_roots) + + def _projection_for_root( root: Container, key_by_message_identity: Mapping[int, str], @@ -565,7 +627,7 @@ def _projection_for_root( return ThreadProjection(tuple(message_keys), thread_ids) -def _build_component( +def _build_forest( keys: tuple[str, ...], records: Mapping[str, IndexedMessage], ranks: Mapping[str, int], @@ -573,7 +635,7 @@ def _build_component( group_by_subject: bool, sort_by_sent_date: bool, ) -> tuple[tuple[Container, ...], tuple[ThreadProjection, ...]]: - """Run the canonical batch engine for one affected component.""" + """Run the canonical batch engine for one ordered record subset.""" messages: list[Message] = [] key_by_message_identity: dict[int, str] = {} for key in keys: @@ -593,74 +655,12 @@ def _build_component( projections = tuple( _projection_for_root(root, key_by_message_identity, records) for root in roots ) + for key, message in zip(keys, messages): + if records[key].message.sequence_number is None: + message.sequence_number = None return roots, projections -def _root_order_key( - root: Container, - projection: ThreadProjection, - records: Mapping[str, IndexedMessage], - ranks: Mapping[str, int], - *, - sort_by_sent_date: bool, -) -> tuple[object, ...]: - """Return the same global root-order contract used by the batch algorithm.""" - if not sort_by_sent_date: - return (min(ranks[key] for key in projection.message_keys),) - if not projection.message_keys: - return (normalize_sent_date(None), 0, 0) - first_key = projection.message_keys[0] - message = records[first_key].message - sequence_number = ( - ranks[first_key] - if message.sequence_number is None - else message.sequence_number - ) - return ( - normalize_sent_date(message.sent_date, message.internal_date), - sequence_number, - ranks[first_key], - ) - - -def _compose_forest( - roots_by_component: Mapping[str, tuple[Container, ...]], - projections_by_component: Mapping[str, tuple[ThreadProjection, ...]], - records: Mapping[str, IndexedMessage], - ranks: Mapping[str, int], - *, - sort_by_sent_date: bool, -) -> tuple[tuple[Container, ...], tuple[ThreadProjection, ...]]: - """Compose reusable component outputs into one deterministic global forest.""" - entries: list[tuple[tuple[object, ...], Container, ThreadProjection]] = [] - for component_id, roots in roots_by_component.items(): - projections = projections_by_component[component_id] - for root, projection in zip(roots, projections): - entries.append( - ( - _root_order_key( - root, - projection, - records, - ranks, - sort_by_sent_date=sort_by_sent_date, - ), - root, - projection, - ) - ) - entries.sort(key=lambda entry: entry[0]) - return ( - tuple(entry[1] for entry in entries), - tuple(entry[2] for entry in entries), - ) - - -def _projection_overlap(first: ThreadProjection, second: ThreadProjection) -> bool: - """Return whether two projections share any immutable caller message key.""" - return bool(set(first.message_keys) & set(second.message_keys)) - - def _transition_thread_ids( before: Iterable[ThreadProjection], after: Iterable[ThreadProjection], @@ -677,54 +677,85 @@ def _transition_thread_ids( ) +def _projection_membership( + projections: Sequence[ThreadProjection], + name: str, +) -> dict[str, int]: + """Index each caller key once and reject overlapping root projections.""" + membership: dict[str, int] = {} + for projection_index, projection in enumerate(projections): + for message_key in projection.message_keys: + if message_key in membership: + raise IncrementalThreadError( + f"{name} projections contain duplicate message_key: {message_key}" + ) + membership[message_key] = projection_index + return membership + + def _thread_delta( previous_version: int, version: int, affected_message_keys: tuple[str, ...], - before: tuple[ThreadProjection, ...], - after: tuple[ThreadProjection, ...], + before: Sequence[ThreadProjection], + after: Sequence[ThreadProjection], ) -> ThreadDelta: - """Classify deterministic projection additions, removals, updates, and transitions.""" + """Classify projection changes and transitions in linear message-key work.""" + before_tuple = tuple(before) + after_tuple = tuple(after) + before_membership = _projection_membership(before_tuple, "before") + after_membership = _projection_membership(after_tuple, "after") + + before_by_after: list[set[int]] = [set() for _ in after_tuple] + after_by_before: list[set[int]] = [set() for _ in before_tuple] + for message_key, after_index in after_membership.items(): + before_index = before_membership.get(message_key) + if before_index is not None: + before_by_after[after_index].add(before_index) + after_by_before[before_index].add(after_index) + + before_values = set(before_tuple) added = tuple( projection - for projection in after - if not any(_projection_overlap(projection, old) for old in before) + for projection, overlaps in zip(after_tuple, before_by_after) + if not overlaps ) removed = tuple( projection - for projection in before - if not any(_projection_overlap(projection, new) for new in after) + for projection, overlaps in zip(before_tuple, after_by_before) + if not overlaps ) updated = tuple( projection - for projection in after - if projection not in before - and any(_projection_overlap(projection, old) for old in before) + for projection, overlaps in zip(after_tuple, before_by_after) + if overlaps and projection not in before_values + ) + merges = tuple( + ThreadTransition( + "merge", + tuple(before_tuple[index] for index in sorted(overlaps)), + (projection,), + _transition_thread_ids( + tuple(before_tuple[index] for index in sorted(overlaps)), + (projection,), + ), + ) + for projection, overlaps in zip(after_tuple, before_by_after) + if len(overlaps) > 1 + ) + splits = tuple( + ThreadTransition( + "split", + (projection,), + tuple(after_tuple[index] for index in sorted(overlaps)), + _transition_thread_ids( + (projection,), + tuple(after_tuple[index] for index in sorted(overlaps)), + ), + ) + for projection, overlaps in zip(before_tuple, after_by_before) + if len(overlaps) > 1 ) - merges: list[ThreadTransition] = [] - for projection in after: - overlapping = tuple(old for old in before if _projection_overlap(projection, old)) - if len(overlapping) > 1: - merges.append( - ThreadTransition( - "merge", - overlapping, - (projection,), - _transition_thread_ids(overlapping, (projection,)), - ) - ) - splits: list[ThreadTransition] = [] - for projection in before: - overlapping = tuple(new for new in after if _projection_overlap(projection, new)) - if len(overlapping) > 1: - splits.append( - ThreadTransition( - "split", - (projection,), - overlapping, - _transition_thread_ids((projection,), overlapping), - ) - ) return ThreadDelta( previous_version, version, @@ -732,8 +763,8 @@ def _thread_delta( added, removed, updated, - tuple(merges), - tuple(splits), + merges, + splits, ) @@ -821,15 +852,32 @@ def __init__( self._keys_by_token: dict[str, set[str]] = {} self._component_by_key: dict[str, str] = {} self._keys_by_component: dict[str, tuple[str, ...]] = {} - self._roots_by_component: dict[str, tuple[Container, ...]] = {} - self._projections_by_component: dict[str, tuple[ThreadProjection, ...]] = {} - self._roots: tuple[Container, ...] = () - self._projections: tuple[ThreadProjection, ...] = () + self._roots: tuple[Container, ...] | None = () + self._projections: tuple[ThreadProjection, ...] | None = () def __len__(self) -> int: """Return the number of indexed caller message keys.""" return len(self._records) + def _materialize_forest(self) -> None: + """Build and cache the complete canonical forest only when requested.""" + if self._roots is not None and self._projections is not None: + return + ranks = ( + _current_ranks(self._positions) + if self._sort_by_sent_date + else self._positions + ) + roots, projections = _build_forest( + self.message_keys, + self._records, + ranks, + group_by_subject=self._group_by_subject, + sort_by_sent_date=self._sort_by_sent_date, + ) + self._roots = roots + self._projections = projections + @property def version(self) -> int: """Return the optimistic mailbox-state version.""" @@ -842,12 +890,16 @@ def message_keys(self) -> tuple[str, ...]: @property def roots(self) -> tuple[Container, ...]: - """Return the current transport-neutral thread roots.""" - return self._roots + """Return defensive transport-neutral copies of current thread roots.""" + self._materialize_forest() + assert self._roots is not None + return _public_forest_copy(self._roots) @property def projections(self) -> tuple[ThreadProjection, ...]: """Return deterministic caller-key projections for current roots.""" + self._materialize_forest() + assert self._projections is not None return self._projections def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: @@ -870,12 +922,15 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: if not ( change_set.additions or change_set.replacements or change_set.removals ): - return _thread_delta( + return ThreadDelta( self._version, self._version, (), - self._projections, - self._projections, + (), + (), + (), + (), + (), ) addition_keys = {record.message_key for record in change_set.additions} @@ -913,7 +968,8 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: records = dict(self._records) positions = dict(self._positions) tokens_by_key = dict(self._tokens_by_key) - keys_by_token = _copy_token_buckets(self._keys_by_token) + keys_by_token = dict(self._keys_by_token) + copied_tokens: set[str] = set() next_position = self._next_position changed_existing_keys = replacement_keys | removal_keys candidate_seeds: set[str] = set() @@ -924,7 +980,12 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: if component_id is not None: candidate_seeds.update(self._keys_by_component[component_id]) touched_tokens.update( - _remove_key_from_buckets(key, tokens_by_key, keys_by_token) + _remove_key_from_buckets( + key, + tokens_by_key, + keys_by_token, + copied_tokens, + ) ) for key in removal_keys: @@ -942,6 +1003,7 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: tokens, tokens_by_key, keys_by_token, + copied_tokens, ) candidate_seeds.add(replacement.message_key) for addition in copied_additions: @@ -958,6 +1020,7 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: tokens, tokens_by_key, keys_by_token, + copied_tokens, ) candidate_seeds.add(addition.message_key) @@ -968,8 +1031,38 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: if component_id is not None: candidate_seeds.update(self._keys_by_component[component_id]) + old_component_ids = { + self._component_by_key[key] + for key in candidate_seeds + if key in self._component_by_key + } + old_ranks = ( + _current_ranks(self._positions) + if self._sort_by_sent_date + else self._positions + ) + before_affected_keys = _ordered_keys( + { + key + for component_id in old_component_ids + for key in self._keys_by_component[component_id] + }, + self._positions, + ) + _, before_affected_projections = _build_forest( + before_affected_keys, + self._records, + old_ranks, + group_by_subject=self._group_by_subject, + sort_by_sent_date=self._sort_by_sent_date, + ) + _validate_external_identities(records) - ranks = _current_ranks(positions) + ranks = ( + _current_ranks(positions) + if self._sort_by_sent_date + else positions + ) if self._sort_by_sent_date: _validate_effective_sequence_numbers(records, ranks) @@ -992,17 +1085,6 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: for component_id, keys in self._keys_by_component.items() if component_id in unaffected_component_ids } - roots_by_component = { - component_id: roots - for component_id, roots in self._roots_by_component.items() - if component_id in unaffected_component_ids - } - projections_by_component = { - component_id: projections - for component_id, projections in self._projections_by_component.items() - if component_id in unaffected_component_ids - } - for keys in _partition_components( current_candidate_keys, positions, @@ -1010,24 +1092,16 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: keys_by_token, ): component_id = keys[0] - roots, projections = _build_component( - keys, - records, - ranks, - group_by_subject=self._group_by_subject, - sort_by_sent_date=self._sort_by_sent_date, - ) keys_by_component[component_id] = keys - roots_by_component[component_id] = roots - projections_by_component[component_id] = projections for key in keys: component_by_key[key] = component_id - roots, projections = _compose_forest( - roots_by_component, - projections_by_component, + after_affected_keys = _ordered_keys(current_candidate_keys, positions) + _, after_affected_projections = _build_forest( + after_affected_keys, records, ranks, + group_by_subject=self._group_by_subject, sort_by_sent_date=self._sort_by_sent_date, ) affected_positions = { @@ -1043,8 +1117,8 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: previous_version, version, affected, - self._projections, - projections, + before_affected_projections, + after_affected_projections, ) self._records = records @@ -1054,10 +1128,8 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: self._keys_by_token = keys_by_token self._component_by_key = component_by_key self._keys_by_component = keys_by_component - self._roots_by_component = roots_by_component - self._projections_by_component = projections_by_component - self._roots = roots - self._projections = projections + self._roots = None + self._projections = None self._version = version return delta diff --git a/tests/test_incremental_benchmark.py b/tests/test_incremental_benchmark.py new file mode 100644 index 0000000..4dbe6eb --- /dev/null +++ b/tests/test_incremental_benchmark.py @@ -0,0 +1,65 @@ +"""Contract tests for the mailbox-scale incremental benchmark.""" + +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path + +import pytest + +ROOT = Path(__file__).parents[1] +BENCHMARK_PATH = ROOT / "benchmarks" / "incremental_mailbox.py" +SPEC = importlib.util.spec_from_file_location( + "threadweave_incremental_benchmark", + BENCHMARK_PATH, +) +assert SPEC is not None and SPEC.loader is not None +benchmark = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = benchmark +SPEC.loader.exec_module(benchmark) + + +def test_small_benchmark_proves_projection_parity_and_reports_resources(): + """The CI-sized run emits matching digests and complete evidence fields.""" + result = benchmark.run_benchmark(1_000, 10) + + assert result["schema_version"] == 1 + assert result["message_count"] == 1_001 + assert result["incremental"]["projection_sha256"] == ( + result["full_rebuild"]["projection_sha256"] + ) + assert result["incremental"]["affected_message_count"] == 21 + assert result["incremental"]["delta_apply_seconds"] >= 0 + assert result["incremental"]["peak_rss_bytes"] > 0 + assert result["full_rebuild"]["full_rebuild_seconds"] >= 0 + assert result["full_rebuild"]["peak_rss_bytes"] > 0 + + +def test_benchmark_validates_size_contracts(): + """Invalid or undersized workloads fail before spawning workers.""" + with pytest.raises(ValueError, match="message_count"): + benchmark.run_benchmark(0, 10) + with pytest.raises(ValueError, match="thread_size"): + benchmark.run_benchmark(100, 0) + with pytest.raises(ValueError, match="two complete threads"): + benchmark.run_benchmark(10, 10) + + +def test_main_writes_deterministic_json_shape(tmp_path: Path): + """The command-line entry point writes one parseable evidence document.""" + output = tmp_path / "benchmark.json" + assert benchmark.main( + [ + "--messages", + "1000", + "--thread-size", + "10", + "--output", + str(output), + ] + ) == 0 + parsed = json.loads(output.read_text(encoding="utf-8")) + assert parsed["schema_version"] == 1 + assert parsed["message_count"] == 1_001 diff --git a/tests/test_incremental_benchmark_workflow.py b/tests/test_incremental_benchmark_workflow.py new file mode 100644 index 0000000..3c9120d --- /dev/null +++ b/tests/test_incremental_benchmark_workflow.py @@ -0,0 +1,56 @@ +"""Governance contract for the mailbox-scale benchmark workflow.""" + +from pathlib import Path + +WORKFLOW = ( + Path(__file__).parents[1] + / ".github" + / "workflows" + / "incremental-benchmark.yml" +) + + +def _workflow() -> str: + """Return the workflow source without adding a YAML parser dependency.""" + return WORKFLOW.read_text(encoding="utf-8") + + +def test_benchmark_is_manual_and_scheduled_with_a_100k_default(): + """Operators can reproduce evidence while a weekly run catches regressions.""" + workflow = _workflow() + assert "workflow_dispatch:" in workflow + assert 'default: "100000"' in workflow + assert 'cron: "17 3 * * 1"' in workflow + assert "cancel-in-progress: true" in workflow + + +def test_benchmark_uses_immutable_actions_and_the_reviewed_dependency_lock(): + """Performance evidence runs through the same pinned supply-chain boundary.""" + workflow = _workflow() + required_actions = { + "step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920", + "actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1", + "actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97", + "actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a", + } + observed_actions = { + line.strip().removeprefix("uses: ").split(" #", 1)[0] + for line in workflow.splitlines() + if line.strip().startswith("uses: ") + } + assert required_actions <= observed_actions + assert "pip install --require-hashes -r requirements/ci.lock" in workflow + assert "persist-credentials: false" in workflow + + +def test_benchmark_requires_parity_bounded_impact_and_evidence_upload(): + """The workflow fails closed on incorrect output or a regressed 100k delta.""" + workflow = _workflow() + assert "incremental_mailbox.py" in workflow + assert 'incremental["projection_sha256"] == full_rebuild["projection_sha256"]' in workflow + assert 'incremental["affected_message_count"] == 21' in workflow + assert 'incremental["delta_apply_seconds"] < full_rebuild[' in workflow + assert "incremental-mailbox-benchmark-${{ github.run_id }}" in workflow + assert "retention-days: 90" in workflow + assert "NVIDIA_NIM_API_KEY" not in workflow + assert "COPILOT_GITHUB_TOKEN" not in workflow diff --git a/tests/test_incremental_components.py b/tests/test_incremental_components.py index 626a3f1..bd5ec4b 100644 --- a/tests/test_incremental_components.py +++ b/tests/test_incremental_components.py @@ -2,6 +2,8 @@ from __future__ import annotations +import pytest + from collections.abc import Iterable import threadweave.incremental as incremental_module @@ -226,7 +228,7 @@ def recording_delegate(messages, **options): MailboxChangeSet(expected_version=1, replacements=(replacement,)) ) - assert calls == [("a", "b")] + assert calls == [("a", "b"), ("a", "b")] assert all("x" not in call and "y" not in call for call in calls) @@ -243,3 +245,86 @@ def test_noop_change_set_advances_no_version_and_returns_empty_delta(): assert delta.updated_threads == () assert delta.merges == () assert delta.splits == () + + +def test_delta_classification_receives_only_affected_component_projections( + monkeypatch: pytest.MonkeyPatch, +): + """Unrelated roots never enter one small change's delta classifier.""" + records = tuple( + _record(f"key_{index}", message_id=f"message_{index}") + for index in range(128) + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + original = incremental_module._thread_delta + observed_sizes: list[tuple[int, int]] = [] + + def recording_delta(*args): + """Record old/new projection counts before delegating.""" + observed_sizes.append((len(args[3]), len(args[4]))) + return original(*args) + + monkeypatch.setattr(incremental_module, "_thread_delta", recording_delta) + index.apply( + MailboxChangeSet( + expected_version=1, + additions=( + _record( + "new_key", + message_id="new_message", + references=("message_0",), + ), + ), + ) + ) + + assert observed_sizes == [(1, 1)] + + + +def test_small_delta_defers_the_complete_canonical_batch_view( + monkeypatch: pytest.MonkeyPatch, +): + """Apply batches only affected records until a full view is requested.""" + records = tuple( + _record(f"key_{index}", message_id=f"message_{index}") + for index in range(128) + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + original = incremental_module._batch_thread_messages + observed_message_counts: list[int] = [] + + def recording_batch(messages, **options): + """Record each canonical batch size before delegating.""" + materialized = tuple(messages) + observed_message_counts.append(len(materialized)) + return original(materialized, **options) + + monkeypatch.setattr( + incremental_module, + "_batch_thread_messages", + recording_batch, + ) + index.apply( + MailboxChangeSet( + expected_version=1, + additions=( + _record( + "new_key", + message_id="new_message", + references=("message_0",), + ), + ), + ) + ) + + assert observed_message_counts == [1, 2] + assert len(index.projections) == 128 + assert observed_message_counts == [1, 2, 129] + assert len(index.projections) == 128 + assert len(index.roots) == 128 + assert observed_message_counts == [1, 2, 129] diff --git a/tests/test_incremental_contract.py b/tests/test_incremental_contract.py index dba9e71..d374c83 100644 --- a/tests/test_incremental_contract.py +++ b/tests/test_incremental_contract.py @@ -171,6 +171,43 @@ def test_structural_metadata_is_copied_while_payload_remains_caller_owned(): assert indexed.payload is message.payload +def test_public_roots_are_defensive_structural_copies(): + """Caller graph edits cannot corrupt the index's reusable internal forest.""" + payload = object() + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record( + "root", + message_id="root", + subject="Original", + payload=payload, + ), + _record( + "child", + message_id="child", + references=("root",), + ), + ), + ) + ) + + exposed = index.roots + exposed[0].message.subject = "Mutated" + exposed[0].children.clear() + + current = index.roots + assert current is not exposed + assert current[0] is not exposed[0] + assert current[0].message is not exposed[0].message + assert current[0].message.subject == "Original" + assert current[0].message.payload is payload + assert [child.message.payload for child in current[0].children] == ["child"] + assert index.projections[0].message_keys == ("root", "child") + + def test_reported_external_identity_is_immutable_on_replacement(): """RFC 8474 identity metadata cannot disappear or change after exposure.""" index = IncrementalThreadIndex() diff --git a/tests/test_incremental_parity.py b/tests/test_incremental_parity.py index 6c87474..2162dd1 100644 --- a/tests/test_incremental_parity.py +++ b/tests/test_incremental_parity.py @@ -2,6 +2,8 @@ from __future__ import annotations +import random + import pytest from threadweave import ( @@ -10,6 +12,7 @@ IndexedMessage, MailboxChangeSet, Message, + ThreadSerializationError, serialize_thread_response, thread_messages, ) @@ -105,6 +108,34 @@ def test_sent_date_order_matches_batch_across_unrelated_components(): ) +def test_implicit_sort_positions_do_not_leak_into_public_metadata(): + """Internal sort ranks must not become caller-visible IMAP identifiers.""" + records = ( + _record( + "earlier", + message_id="earlier", + sent_date="1 Jan 2026 00:00:00 +0000", + ), + _record( + "later", + message_id="later", + sent_date="2 Jan 2026 00:00:00 +0000", + ), + ) + index = IncrementalThreadIndex(sort_by_sent_date=True) + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + batch = thread_messages( + (record.message for record in records), + sort_by_sent_date=True, + ) + assert [root.message.sequence_number for root in index.roots] == [ + root.message.sequence_number for root in batch + ] == [None, None] + with pytest.raises(ThreadSerializationError, match="positive integer"): + serialize_thread_response(index.roots) + + def test_global_sequence_number_collision_fails_atomically(): """Separate components cannot hide duplicate RFC ordering tie-breakers.""" index = IncrementalThreadIndex(sort_by_sent_date=True) @@ -209,3 +240,132 @@ def test_implicit_sequence_positions_remain_stable_after_replacement(): ("a",), ("b",), ) + + +def test_missing_ancestor_creation_order_matches_canonical_batch_root_order(): + """A late missing-root placeholder cannot reorder an earlier independent root.""" + records = ( + _record( + "early_child", + message_id="child@example.test", + references=("late-parent@example.test",), + ), + _record("independent", message_id="independent@example.test"), + _record( + "late_parent", + message_id="late-parent@example.test", + references=("missing-ancestor@example.test",), + ), + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + batch = thread_messages(record.message for record in records) + + assert _preorder(index.roots) == _preorder(batch) == ( + ("independent",), + ("late_parent", "early_child"), + ) + + +@pytest.mark.parametrize("group_by_subject", [False, True]) +@pytest.mark.parametrize("sort_by_sent_date", [False, True]) +def test_bounded_randomized_change_stream_matches_full_batch_oracle( + group_by_subject: bool, + sort_by_sent_date: bool, +): + """Deterministic mixed mailbox changes preserve canonical forest parity.""" + option_code = int(group_by_subject) * 10 + int(sort_by_sent_date) + subjects = (None, "Topic", "Re: Topic", "Ｔｏｐｉｃ", "Other", "Fwd: Other") + dates = ( + None, + "1 Jan 2026 00:00:00 +0000", + "2 Jan 2026 09:00:00 +0900", + "3 Jan 2026 00:00:00 -0500", + ) + + for seed in range(4): + random_source = random.Random(10_000 + option_code * 100 + seed) + index = IncrementalThreadIndex( + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + ) + records: dict[str, IndexedMessage] = {} + ordered_keys: list[str] = [] + next_key = 0 + + def random_record(key: str) -> IndexedMessage: + """Build one deterministic adversarial record for this seed.""" + existing_ids = [ + record.message.message_id + for record in records.values() + if record.message.message_id is not None + ] + message_id_mode = random_source.randrange(5) + if message_id_mode == 0: + message_id = None + elif message_id_mode == 1 and existing_ids: + message_id = random_source.choice(existing_ids) + else: + message_id = f"id-{key}@example.test" + + reference_candidates = list(existing_ids) + reference_candidates.extend( + f"missing-{index}@example.test" for index in range(3) + ) + reference_count = random_source.randrange(3) + references = tuple( + random_source.choice(reference_candidates) + for _ in range(reference_count) + ) if reference_candidates else () + return _record( + key, + message_id=message_id, + references=references, + subject=random_source.choice(subjects), + sent_date=random_source.choice(dates), + ) + + for _step in range(24): + operation_roll = random_source.random() + if not records or operation_roll < 0.50: + key = f"seed_{seed}_message_{next_key}" + next_key += 1 + record = random_record(key) + changes = MailboxChangeSet( + expected_version=index.version, + additions=(record,), + ) + records[key] = record + ordered_keys.append(key) + elif operation_roll < 0.78: + key = random_source.choice(ordered_keys) + record = random_record(key) + changes = MailboxChangeSet( + expected_version=index.version, + replacements=(record,), + ) + records[key] = record + else: + key = random_source.choice(ordered_keys) + changes = MailboxChangeSet( + expected_version=index.version, + removals=(key,), + ) + del records[key] + ordered_keys.remove(key) + + index.apply(changes) + batch = thread_messages( + (records[key].message for key in ordered_keys), + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + ) + expected_shape = _preorder(batch) + assert _preorder(index.roots) == expected_shape + assert tuple( + projection.message_keys for projection in index.projections + ) == tuple( + tuple(key for key in root if key is not None) + for root in expected_shape + ) diff --git a/tests/test_incremental_private_graph.py b/tests/test_incremental_private_graph.py index 05efffe..325ce7a 100644 --- a/tests/test_incremental_private_graph.py +++ b/tests/test_incremental_private_graph.py @@ -85,16 +85,148 @@ def test_projection_defenses_cover_dummy_cycle_and_foreign_messages(): incremental._projection_for_root(child, {}, {"a": record}) -def test_empty_projection_has_an_earliest_safe_sent_date_key(): - """A malformed empty projection remains deterministically sortable.""" - key = incremental._root_order_key( - Container(), - ThreadProjection(()), - {}, + + +def test_thread_delta_scans_projection_sequences_a_bounded_number_of_times(): + """Large independent forests avoid pairwise projection comparisons.""" + + class CountingSequence: + """Wrap projections and record full-sequence operations.""" + + def __init__(self, values: tuple[ThreadProjection, ...]) -> None: + """Store values with zeroed iteration and membership counters.""" + self.values = values + self.iterations = 0 + self.membership_checks = 0 + + def __iter__(self): + """Iterate while recording one full-sequence pass.""" + self.iterations += 1 + return iter(self.values) + + def __len__(self) -> int: + """Return the wrapped projection count.""" + return len(self.values) + + def __getitem__(self, index: int) -> ThreadProjection: + """Return one wrapped projection by position.""" + return self.values[index] + + def __contains__(self, value: object) -> bool: + """Record whole-sequence membership checks.""" + self.membership_checks += 1 + return value in self.values + + projection_count = 128 + values = tuple( + ThreadProjection((f"message_{index}",)) + for index in range(projection_count) + ) + before = CountingSequence(values) + after = CountingSequence(values + (ThreadProjection(("new_message",)),)) + + delta = incremental._thread_delta(0, 1, ("new_message",), before, after) + + assert delta.added_threads == (ThreadProjection(("new_message",)),) + assert before.iterations <= 2 + assert after.iterations <= 2 + assert before.membership_checks == 0 + assert after.membership_checks == 0 + + +def test_projection_membership_rejects_duplicate_keys_between_roots(): + """Malformed projections cannot make merge/split classification ambiguous.""" + with pytest.raises(IncrementalThreadError, match="duplicate message_key"): + incremental._thread_delta( + 0, + 1, + (), + ( + ThreadProjection(("shared",)), + ThreadProjection(("shared",)), + ), + (), + ) + + +def test_reverse_token_buckets_are_copied_only_when_mutated(): + """Atomic changes preserve every shared pre-transaction bucket.""" + original = {"token": {"a", "b"}} + overlay = dict(original) + tokens_by_key = {"a": frozenset({"token"})} + copied_tokens: set[str] = set() + + incremental._remove_key_from_buckets( + "a", + tokens_by_key, + overlay, + copied_tokens, + ) + incremental._add_key_to_buckets( + "c", + frozenset({"token"}), + tokens_by_key, + overlay, + copied_tokens, + ) + + assert original == {"token": {"a", "b"}} + assert overlay == {"token": {"b", "c"}} + assert copied_tokens == {"token"} + + +def test_component_partition_reads_positions_linearly_for_disconnected_keys(): + """Disconnected mailboxes avoid a repeated-minimum quadratic scan.""" + + class CountingPositions(dict[str, int]): + """Count position lookups performed by the partitioner.""" + + def __init__(self, values: dict[str, int]) -> None: + """Create the mapping with a zeroed lookup counter.""" + super().__init__(values) + self.lookups = 0 + + def __getitem__(self, key: str) -> int: + """Return one position while recording algorithmic work.""" + self.lookups += 1 + return super().__getitem__(key) + + key_count = 128 + keys = {f"message_{index}" for index in range(key_count)} + positions = CountingPositions( + {key: index for index, key in enumerate(sorted(keys), start=1)} + ) + components = incremental._partition_components( + keys, + positions, + {key: frozenset() for key in keys}, {}, - sort_by_sent_date=True, ) - assert key == (incremental.normalize_sent_date(None), 0, 0) + + assert len(components) == key_count + assert positions.lookups <= key_count * 2 + + +def test_public_forest_copy_rejects_shared_and_cyclic_internal_nodes(): + """Defensive copies fail closed if derived graph invariants are corrupted.""" + index = IncrementalThreadIndex() + shared = Container(message=Message(message_id="shared")) + index._roots = (shared, shared) + with pytest.raises(IncrementalThreadError, match="shared or cyclic"): + _ = index.roots + + cyclic = Container(message=Message(message_id="cyclic")) + cyclic.children = [cyclic] + index._roots = (cyclic,) + with pytest.raises(IncrementalThreadError, match="shared or cyclic"): + _ = index.roots + + root = Container(message=Message(message_id="root")) + root.children = [Container(parent=root)] + index._roots = (root,) + copied = index.roots + assert copied[0].children[0].message is None + assert copied[0].children[0].parent is copied[0] def test_replacement_recovers_when_derived_component_mapping_is_missing(): diff --git a/tools/pr20-review-fixes.part-01 b/tools/pr20-review-fixes.part-01 deleted file mode 100644 index 453fc92..0000000 --- a/tools/pr20-review-fixes.part-01 +++ /dev/null @@ -1,173 +0,0 @@ -diff --git a/.github/workflows/incremental-benchmark.yml b/.github/workflows/incremental-benchmark.yml -new file mode 100644 -index 0000000000000000000000000000000000000000..7296aa1994c4115eb46834fe93eb6f9bd67541dc ---- /dev/null -+++ b/.github/workflows/incremental-benchmark.yml -@@ -0,0 +1,96 @@ -+name: Incremental Mailbox Benchmark -+ -+on: -+ workflow_dispatch: -+ inputs: -+ messages: -+ description: Number of existing mailbox messages -+ required: false -+ default: "100000" -+ type: string -+ schedule: -+ - cron: "17 3 * * 1" -+ -+permissions: -+ contents: read -+ -+concurrency: -+ group: incremental-mailbox-benchmark-${{ github.repository }} -+ cancel-in-progress: true -+ -+env: -+ FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true -+ PYTHONPATH: src -+ -+jobs: -+ benchmark: -+ runs-on: ubuntu-24.04 -+ timeout-minutes: 30 -+ steps: -+ - name: Harden runner and block undeclared egress -+ uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 -+ with: -+ egress-policy: block -+ disable-telemetry: true -+ allowed-endpoints: | -+ codeload.github.com:443 -+ files.pythonhosted.org:443 -+ github.com:443 -+ objects.githubusercontent.com:443 -+ pypi.org:443 -+ release-assets.githubusercontent.com:443 -+ results-receiver.actions.githubusercontent.com:443 -+ *.actions.githubusercontent.com:443 -+ *.blob.core.windows.net:443 -+ -+ - name: Check out the reviewed source without persisted credentials -+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 -+ with: -+ persist-credentials: false -+ -+ - name: Set up Python -+ uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 -+ with: -+ python-version: "3.13" -+ cache: pip -+ cache-dependency-path: requirements/ci.lock -+ -+ - name: Install the reviewed benchmark toolchain -+ run: python -m pip install --require-hashes -r requirements/ci.lock -+ -+ - name: Run the isolated mailbox benchmark -+ env: -+ BENCHMARK_MESSAGES: ${{ inputs.messages || '100000' }} -+ run: | -+ set -euo pipefail -+ python benchmarks/incremental_mailbox.py \ -+ --messages "$BENCHMARK_MESSAGES" \ -+ --thread-size 10 \ -+ --output incremental-benchmark.json -+ -+ - name: Enforce parity and the mailbox-scale delta target -+ run: | -+ python - <<'PY' -+ import json -+ from pathlib import Path -+ -+ result = json.loads( -+ Path("incremental-benchmark.json").read_text(encoding="utf-8") -+ ) -+ incremental = result["incremental"] -+ full_rebuild = result["full_rebuild"] -+ assert incremental["projection_sha256"] == full_rebuild["projection_sha256"] -+ assert incremental["affected_message_count"] == 21 -+ if result["message_count"] >= 100_001: -+ assert incremental["delta_apply_seconds"] < full_rebuild[ -+ "full_rebuild_seconds" -+ ] -+ PY -+ -+ - name: Upload benchmark evidence -+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 -+ with: -+ name: incremental-mailbox-benchmark-${{ github.run_id }}-${{ github.run_attempt }} -+ path: incremental-benchmark.json -+ if-no-files-found: error -+ retention-days: 90 -diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md -new file mode 100644 -index 0000000000000000000000000000000000000000..cfce2c17eb16877e7e25f99900753af932b0a287 ---- /dev/null -+++ b/ARCHITECTURE.md -@@ -0,0 +1,75 @@ -+# ThreadWeave Architecture -+ -+## Decision status -+ -+This document is the repository-level architecture decision record. `AGENTS.md` is -+the canonical operating policy; this file explains component boundaries and data -+flow for human reviewers and embedding services such as naruon. -+ -+## Architectural goal -+ -+ThreadWeave provides one standards-grounded threading kernel that works both as a -+zero-runtime-dependency Python package and as a module inside a larger mail or -+knowledge platform. Protocol, incremental state, automation, and release concerns -+remain separate from the canonical batch algorithm. -+ -+## Modules -+ -+| Boundary | Responsibility | Must not own | -+|---|---|---| -+| `headers`, `encoded_words`, `subject`, `collation`, `dates` | RFC parsing, normalization, and comparison primitives | graph state, sockets, databases | -+| `threading`, `container` | authoritative JWZ/RFC 5256 batch forest | mailbox sessions, persistence, IMAP framing | -+| `incremental` | caller keys, atomic change sets, component indexes, deltas, payload-free snapshots | a second threading algorithm, database/network state | -+| `imap` | non-mutating RFC 5256 `THREAD`/`UID THREAD` presentation | authentication, command parsing, mailbox storage | -+| stdlib adapters | conversion from Python `email` messages | transport sessions or durable state | -+| GitHub workflows and `scripts/ci` | review-first automation, NIM isolation, release evidence | runtime package behavior | -+ -+## Authoritative data flow -+ -+```text -+caller message metadata -+ -> RFC normalization -+ -> canonical batch thread_messages(component) -+ -> Container forest -+ -> optional incremental component cache and ThreadDelta -+ -> optional IMAP response projection -+``` -+ -+The incremental layer over-approximates connectivity with normalized message IDs, -+effective reference IDs, and optional RFC 5051 subject keys. Every affected -+component is still evaluated by `thread_messages`; no copy of the threading rules -+is maintained in incremental code. -+ -+## State and mutation policy -+ -+- `IndexedMessage.message_key` is caller-owned and immutable across revisions. -+- `apply` validates and computes on isolated transaction state, then commits once. -+- Reverse connectivity buckets use copy-on-write mutation. -+- Complete root/projection views are lazy caches invalidated by a successful change. -+- `roots` returns a defensive structural copy; payload references stay caller-owned. -+- Snapshot schema version 1 contains structural metadata only, never payload objects -+ or graph pointers. -+- RFC 8474 `EMAILID` and `THREADID` remain external identities. Structural merges -+ and splits are explicit transitions rather than silent identifier rewrites. -+ -+## Ordering and protocol policy -+ -+Implicit input positions may be used internally for RFC 5256 sent-date ordering, -+but they are not mailbox sequence numbers and are cleared before public roots are -+returned. IMAP serialization therefore fails closed unless callers supply valid -+sequence numbers or UIDs. -+ -+## Scale evidence -+ -+The deterministic benchmark runs incremental and full-rebuild workers in separate -+processes. It compares projection SHA-256 values and records initial-build time, diff --git a/tools/pr20-review-fixes.part-02 b/tools/pr20-review-fixes.part-02 deleted file mode 100644 index 6656421..0000000 --- a/tools/pr20-review-fixes.part-02 +++ /dev/null @@ -1,159 +0,0 @@ -+delta-application time, full-view materialization time, full-rebuild time, affected -+message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 -+existing messages. -+ -+## Integration policy -+ -+Naruon and other services should own persistence, tenancy, authentication, -+mailbox synchronization, and external stable-ID policy. They pass immutable caller -+keys and `Message` metadata into ThreadWeave and consume `ThreadDelta`, snapshots, -+or IMAP presentation output through public interfaces. -diff --git a/CHANGELOG.md b/CHANGELOG.md -index eb33f4732b41b1e84a8b3a8c25c4be2da11d1885..07cd9add2a0d9dbf0ce36e2c7ca9d903dcb37c27 100644 ---- a/CHANGELOG.md -+++ b/CHANGELOG.md -@@ -6,6 +6,17 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - - ## Unreleased - -+- Keep implicit RFC 5256 sent-date tie-break positions internal to the incremental -+ engine instead of exposing invented IMAP sequence numbers on public roots. -+- Return defensive structural copies from `IncrementalThreadIndex.roots` so caller -+ graph edits cannot corrupt reusable index state while payload objects remain -+ caller-owned references. -+- Replace quadratic disconnected-component partitioning and pairwise thread-delta -+ comparisons with bounded indexed passes, copy reverse-token buckets only when -+ touched, and defer complete forest materialization until a caller requests it. -+- Add deterministic 100,000-message incremental-versus-full-rebuild benchmark -+ evidence with projection parity, affected-message counts, wall time, and peak RSS. -+ - - Add an atomic `IncrementalThreadIndex` for mailbox additions, replacements, - and removals with stable caller message keys, affected-component - recomputation, batch-result parity, and explicit thread merge/split deltas. -diff --git a/CLAUDE.md b/CLAUDE.md -new file mode 100644 -index 0000000000000000000000000000000000000000..bf19006c7fd0bec0b0dea473c3dec2e02274d0b7 ---- /dev/null -+++ b/CLAUDE.md -@@ -0,0 +1,19 @@ -+# Claude / Coding-Agent Instructions -+ -+Read and follow [`AGENTS.md`](AGENTS.md) as the canonical repository policy. Do not -+create a second, conflicting rule set here. -+ -+For every change: -+ -+1. Preserve the batch threader as the correctness oracle. -+2. Work test-first and retain 100% production statement and branch coverage. -+3. Add beginner-readable docstrings to every authored production callable. -+4. Keep runtime dependencies at zero unless an approved architecture decision says -+ otherwise. -+5. Update `CHANGELOG.md`, user documentation, and standards references when public -+ behavior changes. -+6. Never bypass current-head CI, security scans, independent review, release -+ identity checks, or the release-blocker contract. -+7. GitHub product-development automation uses the isolated NVIDIA NIM/OpenCode -+ boundary. Do not introduce `COPILOT_GITHUB_TOKEN` or alter existing review-agent -+ credentials. -diff --git a/README.md b/README.md -index 416381a0f054801540ea68d9da1687d02914d898..8f839a24cb608dea83ac9197a669c3cff4f57c74 100644 ---- a/README.md -+++ b/README.md -@@ -217,7 +217,10 @@ assert IncrementalThreadIndex.restore(index.snapshot()).projections == ( - - Every affected component is recomputed through the canonical batch threader, and - full-rebuild parity is the correctness oracle. Structural merges and splits are --reported explicitly. Versioned snapshots omit arbitrary payloads and reject -+reported explicitly. `roots` returns a defensive structural copy, so callers may -+traverse or edit that graph without corrupting reusable index state; payload objects -+remain caller-owned references. Internal sent-date tie-break positions never become -+public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads and reject - unknown, malformed, or oversized input. See - [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, - identity, snapshot, complexity, and RFC boundaries. -@@ -265,6 +268,9 @@ identity, snapshot, complexity, and RFC boundaries. - - Graph operations, IMAP rendering, and incremental component traversal are - iterative and identity-guarded; deep or cyclic malformed input cannot recurse - indefinitely. -+- `benchmarks/incremental_mailbox.py` compares isolated incremental and full -+ rebuild processes at 100,000 messages, verifies an identical projection digest, -+ and records affected-message count, wall time, and peak RSS as JSON evidence. - - ## Reproducible CI supply chain - -diff --git a/benchmarks/incremental_mailbox.py b/benchmarks/incremental_mailbox.py -new file mode 100644 -index 0000000000000000000000000000000000000000..063aed022704c48cd22e64c236584145090634e9 ---- /dev/null -+++ b/benchmarks/incremental_mailbox.py -@@ -0,0 +1,232 @@ -+"""Deterministic mailbox-scale benchmark for the incremental threading layer. -+ -+The parent process runs incremental and full-rebuild workers separately so each -+scenario reports its own peak resident-set size. Both workers hash the same -+caller-key projection to prove structural parity without serializing a 100,000- -+message forest between processes. -+""" -+ -+from __future__ import annotations -+ -+import argparse -+import hashlib -+import json -+import resource -+import subprocess -+import sys -+from collections.abc import Iterable -+from time import perf_counter -+ -+from threadweave import ( -+ Container, -+ IncrementalThreadIndex, -+ IndexedMessage, -+ MailboxChangeSet, -+ Message, -+ thread_messages, -+) -+ -+ -+def _records(message_count: int, thread_size: int) -> tuple[IndexedMessage, ...]: -+ """Create deterministic thread chains with caller keys retained as payloads.""" -+ records: list[IndexedMessage] = [] -+ for index in range(message_count): -+ position = index % thread_size -+ references = () if position == 0 else (f"message_{index - 1}",) -+ records.append( -+ IndexedMessage( -+ f"key_{index}", -+ Message( -+ message_id=f"message_{index}", -+ references=references, -+ payload=f"key_{index}", -+ ), -+ ) -+ ) -+ return tuple(records) -+ -+ -+def _bridge_record(thread_size: int) -> IndexedMessage: -+ """Return one message that joins the first two deterministic components.""" -+ return IndexedMessage( -+ "bridge_key", -+ Message( -+ message_id="bridge_message", -+ references=( -+ f"message_{thread_size - 1}", -+ f"message_{(thread_size * 2) - 1}", -+ ), -+ payload="bridge_key", -+ ), -+ ) -+ -+ -+def _projection_digest(projections: Iterable[tuple[str, ...]]) -> str: -+ """Hash ordered root projections with unambiguous length framing.""" -+ digest = hashlib.sha256() -+ for projection in projections: -+ digest.update(len(projection).to_bytes(8, "big")) -+ for message_key in projection: -+ encoded = message_key.encode("utf-8") diff --git a/tools/pr20-review-fixes.part-03 b/tools/pr20-review-fixes.part-03 deleted file mode 100644 index 1a9c752..0000000 --- a/tools/pr20-review-fixes.part-03 +++ /dev/null @@ -1,173 +0,0 @@ -+ digest.update(len(encoded).to_bytes(8, "big")) -+ digest.update(encoded) -+ return digest.hexdigest() -+ -+ -+def _forest_projection(roots: Iterable[Container]) -> tuple[tuple[str, ...], ...]: -+ """Project a batch forest into iterative traversal-ordered payload keys.""" -+ result: list[tuple[str, ...]] = [] -+ for root in roots: -+ keys: list[str] = [] -+ seen: set[int] = set() -+ stack = [root] -+ while stack: -+ node = stack.pop() -+ if id(node) in seen: -+ continue -+ seen.add(id(node)) -+ if node.message is not None: -+ if not isinstance(node.message.payload, str): -+ raise RuntimeError("benchmark payload must be a caller key string") -+ keys.append(node.message.payload) -+ stack.extend(reversed(node.children)) -+ result.append(tuple(keys)) -+ return tuple(result) -+ -+ -+def _peak_rss_bytes() -> int: -+ """Return the process peak RSS in bytes on the Linux benchmark runner.""" -+ return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024 -+ -+ -+def _incremental_worker(message_count: int, thread_size: int) -> dict[str, object]: -+ """Build an index, apply one bridge delta, and materialize its final view.""" -+ records = _records(message_count, thread_size) -+ index = IncrementalThreadIndex() -+ started = perf_counter() -+ index.apply(MailboxChangeSet(expected_version=0, additions=records)) -+ initial_seconds = perf_counter() - started -+ -+ bridge = _bridge_record(thread_size) -+ started = perf_counter() -+ delta = index.apply( -+ MailboxChangeSet(expected_version=1, additions=(bridge,)) -+ ) -+ delta_seconds = perf_counter() - started -+ -+ started = perf_counter() -+ projections = tuple( -+ projection.message_keys for projection in index.projections -+ ) -+ materialize_seconds = perf_counter() - started -+ return { -+ "scenario": "incremental", -+ "message_count": message_count + 1, -+ "root_count": len(projections), -+ "affected_message_count": len(delta.affected_message_keys), -+ "initial_build_seconds": initial_seconds, -+ "delta_apply_seconds": delta_seconds, -+ "materialize_seconds": materialize_seconds, -+ "peak_rss_bytes": _peak_rss_bytes(), -+ "projection_sha256": _projection_digest(projections), -+ } -+ -+ -+def _full_worker(message_count: int, thread_size: int) -> dict[str, object]: -+ """Build the same final mailbox through the canonical batch oracle.""" -+ records = _records(message_count, thread_size) -+ messages = [record.message for record in records] -+ messages.append(_bridge_record(thread_size).message) -+ started = perf_counter() -+ roots = thread_messages(messages) -+ rebuild_seconds = perf_counter() - started -+ projections = _forest_projection(roots) -+ return { -+ "scenario": "full_rebuild", -+ "message_count": message_count + 1, -+ "root_count": len(projections), -+ "full_rebuild_seconds": rebuild_seconds, -+ "peak_rss_bytes": _peak_rss_bytes(), -+ "projection_sha256": _projection_digest(projections), -+ } -+ -+ -+def _run_worker(scenario: str, message_count: int, thread_size: int) -> dict[str, object]: -+ """Execute one isolated worker and decode its single JSON result.""" -+ completed = subprocess.run( -+ [ -+ sys.executable, -+ __file__, -+ "--worker", -+ scenario, -+ "--messages", -+ str(message_count), -+ "--thread-size", -+ str(thread_size), -+ ], -+ check=True, -+ capture_output=True, -+ text=True, -+ ) -+ return json.loads(completed.stdout) -+ -+ -+def _validated_positive(value: int, name: str) -> int: -+ """Require one positive non-boolean benchmark integer.""" -+ if isinstance(value, bool) or value <= 0: -+ raise ValueError(f"{name} must be a positive integer") -+ return value -+ -+ -+def run_benchmark(message_count: int, thread_size: int) -> dict[str, object]: -+ """Run isolated scenarios, require parity, and return one evidence record.""" -+ message_count = _validated_positive(message_count, "message_count") -+ thread_size = _validated_positive(thread_size, "thread_size") -+ if message_count < thread_size * 2: -+ raise ValueError("message_count must contain at least two complete threads") -+ incremental = _run_worker("incremental", message_count, thread_size) -+ full_rebuild = _run_worker("full_rebuild", message_count, thread_size) -+ if incremental["projection_sha256"] != full_rebuild["projection_sha256"]: -+ raise RuntimeError("incremental and full-rebuild projections disagree") -+ return { -+ "schema_version": 1, -+ "message_count": message_count + 1, -+ "thread_size": thread_size, -+ "incremental": incremental, -+ "full_rebuild": full_rebuild, -+ } -+ -+ -+def _parser() -> argparse.ArgumentParser: -+ """Build the command-line parser for parent and worker modes.""" -+ parser = argparse.ArgumentParser() -+ parser.add_argument("--messages", type=int, default=100_000) -+ parser.add_argument("--thread-size", type=int, default=10) -+ parser.add_argument( -+ "--worker", -+ choices=("incremental", "full_rebuild"), -+ ) -+ parser.add_argument("--output") -+ return parser -+ -+ -+def main(argv: list[str] | None = None) -> int: -+ """Run a worker or publish the combined deterministic benchmark JSON.""" -+ arguments = _parser().parse_args(argv) -+ if arguments.worker == "incremental": -+ result = _incremental_worker(arguments.messages, arguments.thread_size) -+ elif arguments.worker == "full_rebuild": -+ result = _full_worker(arguments.messages, arguments.thread_size) -+ else: -+ result = run_benchmark(arguments.messages, arguments.thread_size) -+ encoded = json.dumps(result, indent=2, sort_keys=True) + "\n" -+ if arguments.output: -+ with open(arguments.output, "w", encoding="utf-8") as output_file: -+ output_file.write(encoded) -+ else: -+ sys.stdout.write(encoded) -+ return 0 -+ -+ -+if __name__ == "__main__": -+ raise SystemExit(main()) -diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md -index b59ff27a0e683b88f359f786e21a123ee22a5340..6aa08c283a8bc71f25757b609d9c4b6dcb8d15f2 100644 ---- a/docs/incremental-threading.md -+++ b/docs/incremental-threading.md -@@ -118,10 +118,14 @@ old component; a new bridge message includes every old component touched by its - new tokens. The candidate region is repartitioned iteratively and each resulting - component is processed by `thread_messages`. - --Unchanged components retain their existing `Container` roots. Building the --ordered public root tuple still examines the component-root summaries so that --batch-compatible global ordering is preserved; it does not re-run threading or diff --git a/tools/pr20-review-fixes.part-04 b/tools/pr20-review-fixes.part-04 deleted file mode 100644 index 3ef9d3f..0000000 --- a/tools/pr20-review-fixes.part-04 +++ /dev/null @@ -1,153 +0,0 @@ --walk every message in unrelated components. -+Unchanged components retain their existing internal `Container` roots. Applying a -+change does not eagerly rebuild the complete public forest: only the affected old -+and new component views are composed for `ThreadDelta`. The complete ordered forest -+is materialized once, on demand, when `roots` or `projections` is requested. Public -+roots are defensive structural copies; editing their parent, child, or message -+metadata cannot corrupt internal state, while caller payload objects remain shared by -+reference. Internal implicit sent-date ranks are cleared before roots are exposed, so -+ordinary IMAP `THREAD` output still requires real caller-supplied sequence numbers. - - `ThreadDelta.affected_message_keys` reports the candidate region. The delta also - contains added, removed, and structurally updated projections. A metadata-only -@@ -179,11 +183,13 @@ It also covers RFC 5051 subject buckets, RFC 5256 sent-date ordering, ordinary a - UID THREAD output, duplicate and missing Message-ID values, deep chains, - optimistic conflicts, hostile snapshots, and payload omission. - --A separate scheduled/manual benchmark is required before this feature is released --as a performance claim. It must exercise at least 100,000 records and report wall --time, peak RSS, affected-message count, and a full-rebuild comparison. The --incremental contract promises that unrelated records are not passed to the batch --threader; it does not promise constant-time global root presentation. -+`benchmarks/incremental_mailbox.py` runs incremental and full-rebuild scenarios in -+separate processes, requires identical projection hashes, and reports wall time, peak -+RSS, affected-message count, and full-view materialization time. The scheduled/manual -+workflow defaults to 100,000 existing messages and stores the JSON evidence for 90 -+days. The update contract promises that unrelated records are not passed to the batch -+threader; full-view materialization remains proportional to the number of roots and is -+therefore reported separately from delta application. - - ## References - -diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py -index 9150bab749b0c1d87772feca12275d37ad765694..fe9adeaba587f08ccc7a4a33f75722bbbc70c2de 100644 ---- a/src/threadweave/incremental.py -+++ b/src/threadweave/incremental.py -@@ -369,22 +369,30 @@ def _connectivity_tokens( - return frozenset(tokens) - - --def _copy_token_buckets( -- buckets: Mapping[str, set[str]], --) -> dict[str, set[str]]: -- """Return independent mutable copies of reverse token buckets.""" -- return {token: set(keys) for token, keys in buckets.items()} -+def _writable_token_bucket( -+ token: str, -+ keys_by_token: dict[str, set[str]], -+ copied_tokens: set[str], -+) -> set[str]: -+ """Return one copy-on-write reverse bucket owned by the transaction.""" -+ if token not in copied_tokens: -+ keys_by_token[token] = set(keys_by_token.get(token, set())) -+ copied_tokens.add(token) -+ elif token not in keys_by_token: -+ keys_by_token[token] = set() -+ return keys_by_token[token] - - - def _remove_key_from_buckets( - key: str, - tokens_by_key: dict[str, frozenset[str]], - keys_by_token: dict[str, set[str]], -+ copied_tokens: set[str], - ) -> frozenset[str]: -- """Remove one key from copied token indexes and return its old tokens.""" -+ """Remove one key from transaction-owned token buckets.""" - old_tokens = tokens_by_key.pop(key, frozenset()) - for token in old_tokens: -- bucket = keys_by_token[token] -+ bucket = _writable_token_bucket(token, keys_by_token, copied_tokens) - bucket.discard(key) - if not bucket: - del keys_by_token[token] -@@ -396,11 +404,12 @@ def _add_key_to_buckets( - tokens: frozenset[str], - tokens_by_key: dict[str, frozenset[str]], - keys_by_token: dict[str, set[str]], -+ copied_tokens: set[str], - ) -> None: -- """Insert one key into copied forward and reverse token indexes.""" -+ """Insert one key through transaction-owned copy-on-write buckets.""" - tokens_by_key[key] = tokens - for token in tokens: -- keys_by_token.setdefault(token, set()).add(key) -+ _writable_token_bucket(token, keys_by_token, copied_tokens).add(key) - - - def _ordered_keys(keys: Iterable[str], positions: Mapping[str, int]) -> tuple[str, ...]: -@@ -497,8 +506,9 @@ def _partition_components( - """Partition candidate keys into deterministic current connectivity components.""" - remaining = set(keys) - components: list[tuple[str, ...]] = [] -- while remaining: -- seed = min(remaining, key=lambda key: (positions[key], key)) -+ for seed in _ordered_keys(keys, positions): -+ if seed not in remaining: -+ continue - component = {seed} - queue = [seed] - remaining.remove(seed) -@@ -531,6 +541,59 @@ def _message_for_batch(message: Message, sequence_number: int | None) -> Message - ) - - -+def _public_message_copy(message: Message) -> Message: -+ """Copy structural message metadata while retaining the caller payload.""" -+ return Message( -+ message_id=message.message_id, -+ in_reply_to=message.in_reply_to, -+ references=message.references, -+ subject=message.subject, -+ payload=message.payload, -+ sent_date=message.sent_date, -+ internal_date=message.internal_date, -+ sequence_number=message.sequence_number, -+ uid=message.uid, -+ ) -+ -+ -+def _public_forest_copy(roots: Iterable[Container]) -> tuple[Container, ...]: -+ """Return a loop-safe defensive copy of the index's internal forest.""" -+ copied_roots: list[Container] = [] -+ seen: set[int] = set() -+ for root in roots: -+ root_identity = id(root) -+ if root_identity in seen: -+ raise IncrementalThreadError( -+ "internal thread forest contains a shared or cyclic container" -+ ) -+ root_copy = Container( -+ message=None -+ if root.message is None -+ else _public_message_copy(root.message) -+ ) -+ copied_roots.append(root_copy) -+ seen.add(root_identity) -+ stack: list[tuple[Container, Container]] = [(root, root_copy)] -+ while stack: -+ source, target = stack.pop() -+ for child in source.children: -+ child_identity = id(child) -+ if child_identity in seen: -+ raise IncrementalThreadError( -+ "internal thread forest contains a shared or cyclic container" -+ ) -+ child_copy = Container( -+ message=None -+ if child.message is None -+ else _public_message_copy(child.message), -+ parent=target, -+ ) -+ target.children.append(child_copy) -+ seen.add(child_identity) diff --git a/tools/pr20-review-fixes.part-05 b/tools/pr20-review-fixes.part-05 deleted file mode 100644 index c168e82..0000000 --- a/tools/pr20-review-fixes.part-05 +++ /dev/null @@ -1,189 +0,0 @@ -+ stack.append((child, child_copy)) -+ return tuple(copied_roots) -+ -+ - def _projection_for_root( - root: Container, - key_by_message_identity: Mapping[int, str], -@@ -565,7 +628,7 @@ def _projection_for_root( - return ThreadProjection(tuple(message_keys), thread_ids) - - --def _build_component( -+def _build_forest( - keys: tuple[str, ...], - records: Mapping[str, IndexedMessage], - ranks: Mapping[str, int], -@@ -573,7 +636,7 @@ def _build_component( - group_by_subject: bool, - sort_by_sent_date: bool, - ) -> tuple[tuple[Container, ...], tuple[ThreadProjection, ...]]: -- """Run the canonical batch engine for one affected component.""" -+ """Run the canonical batch engine for one ordered record subset.""" - messages: list[Message] = [] - key_by_message_identity: dict[int, str] = {} - for key in keys: -@@ -593,74 +656,12 @@ def _build_component( - projections = tuple( - _projection_for_root(root, key_by_message_identity, records) for root in roots - ) -+ for key, message in zip(keys, messages): -+ if records[key].message.sequence_number is None: -+ message.sequence_number = None - return roots, projections - - --def _root_order_key( -- root: Container, -- projection: ThreadProjection, -- records: Mapping[str, IndexedMessage], -- ranks: Mapping[str, int], -- *, -- sort_by_sent_date: bool, --) -> tuple[object, ...]: -- """Return the same global root-order contract used by the batch algorithm.""" -- if not sort_by_sent_date: -- return (min(ranks[key] for key in projection.message_keys),) -- if not projection.message_keys: -- return (normalize_sent_date(None), 0, 0) -- first_key = projection.message_keys[0] -- message = records[first_key].message -- sequence_number = ( -- ranks[first_key] -- if message.sequence_number is None -- else message.sequence_number -- ) -- return ( -- normalize_sent_date(message.sent_date, message.internal_date), -- sequence_number, -- ranks[first_key], -- ) -- -- --def _compose_forest( -- roots_by_component: Mapping[str, tuple[Container, ...]], -- projections_by_component: Mapping[str, tuple[ThreadProjection, ...]], -- records: Mapping[str, IndexedMessage], -- ranks: Mapping[str, int], -- *, -- sort_by_sent_date: bool, --) -> tuple[tuple[Container, ...], tuple[ThreadProjection, ...]]: -- """Compose reusable component outputs into one deterministic global forest.""" -- entries: list[tuple[tuple[object, ...], Container, ThreadProjection]] = [] -- for component_id, roots in roots_by_component.items(): -- projections = projections_by_component[component_id] -- for root, projection in zip(roots, projections): -- entries.append( -- ( -- _root_order_key( -- root, -- projection, -- records, -- ranks, -- sort_by_sent_date=sort_by_sent_date, -- ), -- root, -- projection, -- ) -- ) -- entries.sort(key=lambda entry: entry[0]) -- return ( -- tuple(entry[1] for entry in entries), -- tuple(entry[2] for entry in entries), -- ) -- -- --def _projection_overlap(first: ThreadProjection, second: ThreadProjection) -> bool: -- """Return whether two projections share any immutable caller message key.""" -- return bool(set(first.message_keys) & set(second.message_keys)) -- -- - def _transition_thread_ids( - before: Iterable[ThreadProjection], - after: Iterable[ThreadProjection], -@@ -677,54 +678,85 @@ def _transition_thread_ids( - ) - - -+def _projection_membership( -+ projections: Sequence[ThreadProjection], -+ name: str, -+) -> dict[str, int]: -+ """Index each caller key once and reject overlapping root projections.""" -+ membership: dict[str, int] = {} -+ for projection_index, projection in enumerate(projections): -+ for message_key in projection.message_keys: -+ if message_key in membership: -+ raise IncrementalThreadError( -+ f"{name} projections contain duplicate message_key: {message_key}" -+ ) -+ membership[message_key] = projection_index -+ return membership -+ -+ - def _thread_delta( - previous_version: int, - version: int, - affected_message_keys: tuple[str, ...], -- before: tuple[ThreadProjection, ...], -- after: tuple[ThreadProjection, ...], -+ before: Sequence[ThreadProjection], -+ after: Sequence[ThreadProjection], - ) -> ThreadDelta: -- """Classify deterministic projection additions, removals, updates, and transitions.""" -+ """Classify projection changes and transitions in linear message-key work.""" -+ before_tuple = tuple(before) -+ after_tuple = tuple(after) -+ before_membership = _projection_membership(before_tuple, "before") -+ after_membership = _projection_membership(after_tuple, "after") -+ -+ before_by_after: list[set[int]] = [set() for _ in after_tuple] -+ after_by_before: list[set[int]] = [set() for _ in before_tuple] -+ for message_key, after_index in after_membership.items(): -+ before_index = before_membership.get(message_key) -+ if before_index is not None: -+ before_by_after[after_index].add(before_index) -+ after_by_before[before_index].add(after_index) -+ -+ before_values = set(before_tuple) - added = tuple( - projection -- for projection in after -- if not any(_projection_overlap(projection, old) for old in before) -+ for projection, overlaps in zip(after_tuple, before_by_after) -+ if not overlaps - ) - removed = tuple( - projection -- for projection in before -- if not any(_projection_overlap(projection, new) for new in after) -+ for projection, overlaps in zip(before_tuple, after_by_before) -+ if not overlaps - ) - updated = tuple( - projection -- for projection in after -- if projection not in before -- and any(_projection_overlap(projection, old) for old in before) -+ for projection, overlaps in zip(after_tuple, before_by_after) -+ if overlaps and projection not in before_values -+ ) -+ merges = tuple( -+ ThreadTransition( -+ "merge", -+ tuple(before_tuple[index] for index in sorted(overlaps)), -+ (projection,), -+ _transition_thread_ids( -+ tuple(before_tuple[index] for index in sorted(overlaps)), -+ (projection,), -+ ), -+ ) -+ for projection, overlaps in zip(after_tuple, before_by_after) -+ if len(overlaps) > 1 -+ ) -+ splits = tuple( -+ ThreadTransition( -+ "split", -+ (projection,), -+ tuple(after_tuple[index] for index in sorted(overlaps)), -+ _transition_thread_ids( diff --git a/tools/pr20-review-fixes.part-06 b/tools/pr20-review-fixes.part-06 deleted file mode 100644 index 39c453b..0000000 --- a/tools/pr20-review-fixes.part-06 +++ /dev/null @@ -1,191 +0,0 @@ -+ (projection,), -+ tuple(after_tuple[index] for index in sorted(overlaps)), -+ ), -+ ) -+ for projection, overlaps in zip(before_tuple, after_by_before) -+ if len(overlaps) > 1 - ) -- merges: list[ThreadTransition] = [] -- for projection in after: -- overlapping = tuple(old for old in before if _projection_overlap(projection, old)) -- if len(overlapping) > 1: -- merges.append( -- ThreadTransition( -- "merge", -- overlapping, -- (projection,), -- _transition_thread_ids(overlapping, (projection,)), -- ) -- ) -- splits: list[ThreadTransition] = [] -- for projection in before: -- overlapping = tuple(new for new in after if _projection_overlap(projection, new)) -- if len(overlapping) > 1: -- splits.append( -- ThreadTransition( -- "split", -- (projection,), -- overlapping, -- _transition_thread_ids((projection,), overlapping), -- ) -- ) - return ThreadDelta( - previous_version, - version, -@@ -732,8 +764,8 @@ def _thread_delta( - added, - removed, - updated, -- tuple(merges), -- tuple(splits), -+ merges, -+ splits, - ) - - -@@ -821,15 +853,32 @@ class IncrementalThreadIndex: - self._keys_by_token: dict[str, set[str]] = {} - self._component_by_key: dict[str, str] = {} - self._keys_by_component: dict[str, tuple[str, ...]] = {} -- self._roots_by_component: dict[str, tuple[Container, ...]] = {} -- self._projections_by_component: dict[str, tuple[ThreadProjection, ...]] = {} -- self._roots: tuple[Container, ...] = () -- self._projections: tuple[ThreadProjection, ...] = () -+ self._roots: tuple[Container, ...] | None = () -+ self._projections: tuple[ThreadProjection, ...] | None = () - - def __len__(self) -> int: - """Return the number of indexed caller message keys.""" - return len(self._records) - -+ def _materialize_forest(self) -> None: -+ """Build and cache the complete canonical forest only when requested.""" -+ if self._roots is not None and self._projections is not None: -+ return -+ ranks = ( -+ _current_ranks(self._positions) -+ if self._sort_by_sent_date -+ else self._positions -+ ) -+ roots, projections = _build_forest( -+ self.message_keys, -+ self._records, -+ ranks, -+ group_by_subject=self._group_by_subject, -+ sort_by_sent_date=self._sort_by_sent_date, -+ ) -+ self._roots = roots -+ self._projections = projections -+ - @property - def version(self) -> int: - """Return the optimistic mailbox-state version.""" -@@ -842,12 +891,16 @@ class IncrementalThreadIndex: - - @property - def roots(self) -> tuple[Container, ...]: -- """Return the current transport-neutral thread roots.""" -- return self._roots -+ """Return defensive transport-neutral copies of current thread roots.""" -+ self._materialize_forest() -+ assert self._roots is not None -+ return _public_forest_copy(self._roots) - - @property - def projections(self) -> tuple[ThreadProjection, ...]: - """Return deterministic caller-key projections for current roots.""" -+ self._materialize_forest() -+ assert self._projections is not None - return self._projections - - def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: -@@ -870,12 +923,15 @@ class IncrementalThreadIndex: - if not ( - change_set.additions or change_set.replacements or change_set.removals - ): -- return _thread_delta( -+ return ThreadDelta( - self._version, - self._version, - (), -- self._projections, -- self._projections, -+ (), -+ (), -+ (), -+ (), -+ (), - ) - - addition_keys = {record.message_key for record in change_set.additions} -@@ -913,7 +969,8 @@ class IncrementalThreadIndex: - records = dict(self._records) - positions = dict(self._positions) - tokens_by_key = dict(self._tokens_by_key) -- keys_by_token = _copy_token_buckets(self._keys_by_token) -+ keys_by_token = dict(self._keys_by_token) -+ copied_tokens: set[str] = set() - next_position = self._next_position - changed_existing_keys = replacement_keys | removal_keys - candidate_seeds: set[str] = set() -@@ -924,7 +981,12 @@ class IncrementalThreadIndex: - if component_id is not None: - candidate_seeds.update(self._keys_by_component[component_id]) - touched_tokens.update( -- _remove_key_from_buckets(key, tokens_by_key, keys_by_token) -+ _remove_key_from_buckets( -+ key, -+ tokens_by_key, -+ keys_by_token, -+ copied_tokens, -+ ) - ) - - for key in removal_keys: -@@ -942,6 +1004,7 @@ class IncrementalThreadIndex: - tokens, - tokens_by_key, - keys_by_token, -+ copied_tokens, - ) - candidate_seeds.add(replacement.message_key) - for addition in copied_additions: -@@ -958,6 +1021,7 @@ class IncrementalThreadIndex: - tokens, - tokens_by_key, - keys_by_token, -+ copied_tokens, - ) - candidate_seeds.add(addition.message_key) - -@@ -968,8 +1032,38 @@ class IncrementalThreadIndex: - if component_id is not None: - candidate_seeds.update(self._keys_by_component[component_id]) - -+ old_component_ids = { -+ self._component_by_key[key] -+ for key in candidate_seeds -+ if key in self._component_by_key -+ } -+ old_ranks = ( -+ _current_ranks(self._positions) -+ if self._sort_by_sent_date -+ else self._positions -+ ) -+ before_affected_keys = _ordered_keys( -+ { -+ key -+ for component_id in old_component_ids -+ for key in self._keys_by_component[component_id] -+ }, -+ self._positions, -+ ) -+ _, before_affected_projections = _build_forest( -+ before_affected_keys, -+ self._records, -+ old_ranks, -+ group_by_subject=self._group_by_subject, -+ sort_by_sent_date=self._sort_by_sent_date, -+ ) -+ - _validate_external_identities(records) diff --git a/tools/pr20-review-fixes.part-07 b/tools/pr20-review-fixes.part-07 deleted file mode 100644 index 779b3bf..0000000 --- a/tools/pr20-review-fixes.part-07 +++ /dev/null @@ -1,189 +0,0 @@ -- ranks = _current_ranks(positions) -+ ranks = ( -+ _current_ranks(positions) -+ if self._sort_by_sent_date -+ else positions -+ ) - if self._sort_by_sent_date: - _validate_effective_sequence_numbers(records, ranks) - -@@ -992,17 +1086,6 @@ class IncrementalThreadIndex: - for component_id, keys in self._keys_by_component.items() - if component_id in unaffected_component_ids - } -- roots_by_component = { -- component_id: roots -- for component_id, roots in self._roots_by_component.items() -- if component_id in unaffected_component_ids -- } -- projections_by_component = { -- component_id: projections -- for component_id, projections in self._projections_by_component.items() -- if component_id in unaffected_component_ids -- } -- - for keys in _partition_components( - current_candidate_keys, - positions, -@@ -1010,24 +1093,16 @@ class IncrementalThreadIndex: - keys_by_token, - ): - component_id = keys[0] -- roots, projections = _build_component( -- keys, -- records, -- ranks, -- group_by_subject=self._group_by_subject, -- sort_by_sent_date=self._sort_by_sent_date, -- ) - keys_by_component[component_id] = keys -- roots_by_component[component_id] = roots -- projections_by_component[component_id] = projections - for key in keys: - component_by_key[key] = component_id - -- roots, projections = _compose_forest( -- roots_by_component, -- projections_by_component, -+ after_affected_keys = _ordered_keys(current_candidate_keys, positions) -+ _, after_affected_projections = _build_forest( -+ after_affected_keys, - records, - ranks, -+ group_by_subject=self._group_by_subject, - sort_by_sent_date=self._sort_by_sent_date, - ) - affected_positions = { -@@ -1043,8 +1118,8 @@ class IncrementalThreadIndex: - previous_version, - version, - affected, -- self._projections, -- projections, -+ before_affected_projections, -+ after_affected_projections, - ) - - self._records = records -@@ -1054,10 +1129,8 @@ class IncrementalThreadIndex: - self._keys_by_token = keys_by_token - self._component_by_key = component_by_key - self._keys_by_component = keys_by_component -- self._roots_by_component = roots_by_component -- self._projections_by_component = projections_by_component -- self._roots = roots -- self._projections = projections -+ self._roots = None -+ self._projections = None - self._version = version - return delta - -diff --git a/tests/test_incremental_benchmark.py b/tests/test_incremental_benchmark.py -new file mode 100644 -index 0000000000000000000000000000000000000000..4dbe6eb248dba15d9c17b7bfc6f17c1a126f4d34 ---- /dev/null -+++ b/tests/test_incremental_benchmark.py -@@ -0,0 +1,65 @@ -+"""Contract tests for the mailbox-scale incremental benchmark.""" -+ -+from __future__ import annotations -+ -+import importlib.util -+import json -+import sys -+from pathlib import Path -+ -+import pytest -+ -+ROOT = Path(__file__).parents[1] -+BENCHMARK_PATH = ROOT / "benchmarks" / "incremental_mailbox.py" -+SPEC = importlib.util.spec_from_file_location( -+ "threadweave_incremental_benchmark", -+ BENCHMARK_PATH, -+) -+assert SPEC is not None and SPEC.loader is not None -+benchmark = importlib.util.module_from_spec(SPEC) -+sys.modules[SPEC.name] = benchmark -+SPEC.loader.exec_module(benchmark) -+ -+ -+def test_small_benchmark_proves_projection_parity_and_reports_resources(): -+ """The CI-sized run emits matching digests and complete evidence fields.""" -+ result = benchmark.run_benchmark(1_000, 10) -+ -+ assert result["schema_version"] == 1 -+ assert result["message_count"] == 1_001 -+ assert result["incremental"]["projection_sha256"] == ( -+ result["full_rebuild"]["projection_sha256"] -+ ) -+ assert result["incremental"]["affected_message_count"] == 21 -+ assert result["incremental"]["delta_apply_seconds"] >= 0 -+ assert result["incremental"]["peak_rss_bytes"] > 0 -+ assert result["full_rebuild"]["full_rebuild_seconds"] >= 0 -+ assert result["full_rebuild"]["peak_rss_bytes"] > 0 -+ -+ -+def test_benchmark_validates_size_contracts(): -+ """Invalid or undersized workloads fail before spawning workers.""" -+ with pytest.raises(ValueError, match="message_count"): -+ benchmark.run_benchmark(0, 10) -+ with pytest.raises(ValueError, match="thread_size"): -+ benchmark.run_benchmark(100, 0) -+ with pytest.raises(ValueError, match="two complete threads"): -+ benchmark.run_benchmark(10, 10) -+ -+ -+def test_main_writes_deterministic_json_shape(tmp_path: Path): -+ """The command-line entry point writes one parseable evidence document.""" -+ output = tmp_path / "benchmark.json" -+ assert benchmark.main( -+ [ -+ "--messages", -+ "1000", -+ "--thread-size", -+ "10", -+ "--output", -+ str(output), -+ ] -+ ) == 0 -+ parsed = json.loads(output.read_text(encoding="utf-8")) -+ assert parsed["schema_version"] == 1 -+ assert parsed["message_count"] == 1_001 -diff --git a/tests/test_incremental_benchmark_workflow.py b/tests/test_incremental_benchmark_workflow.py -new file mode 100644 -index 0000000000000000000000000000000000000000..3c9120d934c6d9a2f3d4834c452d78cf4438a9bf ---- /dev/null -+++ b/tests/test_incremental_benchmark_workflow.py -@@ -0,0 +1,56 @@ -+"""Governance contract for the mailbox-scale benchmark workflow.""" -+ -+from pathlib import Path -+ -+WORKFLOW = ( -+ Path(__file__).parents[1] -+ / ".github" -+ / "workflows" -+ / "incremental-benchmark.yml" -+) -+ -+ -+def _workflow() -> str: -+ """Return the workflow source without adding a YAML parser dependency.""" -+ return WORKFLOW.read_text(encoding="utf-8") -+ -+ -+def test_benchmark_is_manual_and_scheduled_with_a_100k_default(): -+ """Operators can reproduce evidence while a weekly run catches regressions.""" -+ workflow = _workflow() -+ assert "workflow_dispatch:" in workflow -+ assert 'default: "100000"' in workflow -+ assert 'cron: "17 3 * * 1"' in workflow -+ assert "cancel-in-progress: true" in workflow -+ -+ -+def test_benchmark_uses_immutable_actions_and_the_reviewed_dependency_lock(): -+ """Performance evidence runs through the same pinned supply-chain boundary.""" -+ workflow = _workflow() -+ required_actions = { -+ "step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920", -+ "actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1", diff --git a/tools/pr20-review-fixes.part-08 b/tools/pr20-review-fixes.part-08 deleted file mode 100644 index 18dd68f..0000000 --- a/tools/pr20-review-fixes.part-08 +++ /dev/null @@ -1,189 +0,0 @@ -+ "actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97", -+ "actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a", -+ } -+ observed_actions = { -+ line.strip().removeprefix("uses: ").split(" #", 1)[0] -+ for line in workflow.splitlines() -+ if line.strip().startswith("uses: ") -+ } -+ assert required_actions <= observed_actions -+ assert "pip install --require-hashes -r requirements/ci.lock" in workflow -+ assert "persist-credentials: false" in workflow -+ -+ -+def test_benchmark_requires_parity_bounded_impact_and_evidence_upload(): -+ """The workflow fails closed on incorrect output or a regressed 100k delta.""" -+ workflow = _workflow() -+ assert "incremental_mailbox.py" in workflow -+ assert 'incremental["projection_sha256"] == full_rebuild["projection_sha256"]' in workflow -+ assert 'incremental["affected_message_count"] == 21' in workflow -+ assert 'incremental["delta_apply_seconds"] < full_rebuild[' in workflow -+ assert "incremental-mailbox-benchmark-${{ github.run_id }}" in workflow -+ assert "retention-days: 90" in workflow -+ assert "NVIDIA_NIM_API_KEY" not in workflow -+ assert "COPILOT_GITHUB_TOKEN" not in workflow -diff --git a/tests/test_incremental_components.py b/tests/test_incremental_components.py -index 626a3f12604a8f926d69ee05ade5309d16086a82..5343c00195e02d93755f70673f3396e254af3aa6 100644 ---- a/tests/test_incremental_components.py -+++ b/tests/test_incremental_components.py -@@ -226,7 +226,7 @@ def test_unrelated_components_are_not_passed_to_the_batch_delegate(monkeypatch): - MailboxChangeSet(expected_version=1, replacements=(replacement,)) - ) - -- assert calls == [("a", "b")] -+ assert calls == [("a", "b"), ("a", "b")] - assert all("x" not in call and "y" not in call for call in calls) - - -@@ -243,3 +243,86 @@ def test_noop_change_set_advances_no_version_and_returns_empty_delta(): - assert delta.updated_threads == () - assert delta.merges == () - assert delta.splits == () -+ -+ -+def test_delta_classification_receives_only_affected_component_projections( -+ monkeypatch: pytest.MonkeyPatch, -+): -+ """Unrelated roots never enter one small change's delta classifier.""" -+ records = tuple( -+ _record(f"key_{index}", message_id=f"message_{index}") -+ for index in range(128) -+ ) -+ index = IncrementalThreadIndex() -+ index.apply(MailboxChangeSet(expected_version=0, additions=records)) -+ -+ original = incremental_module._thread_delta -+ observed_sizes: list[tuple[int, int]] = [] -+ -+ def recording_delta(*args): -+ """Record old/new projection counts before delegating.""" -+ observed_sizes.append((len(args[3]), len(args[4]))) -+ return original(*args) -+ -+ monkeypatch.setattr(incremental_module, "_thread_delta", recording_delta) -+ index.apply( -+ MailboxChangeSet( -+ expected_version=1, -+ additions=( -+ _record( -+ "new_key", -+ message_id="new_message", -+ references=("message_0",), -+ ), -+ ), -+ ) -+ ) -+ -+ assert observed_sizes == [(1, 1)] -+ -+ -+ -+def test_small_delta_defers_the_complete_canonical_batch_view( -+ monkeypatch: pytest.MonkeyPatch, -+): -+ """Apply batches only affected records until a full view is requested.""" -+ records = tuple( -+ _record(f"key_{index}", message_id=f"message_{index}") -+ for index in range(128) -+ ) -+ index = IncrementalThreadIndex() -+ index.apply(MailboxChangeSet(expected_version=0, additions=records)) -+ -+ original = incremental_module._batch_thread_messages -+ observed_message_counts: list[int] = [] -+ -+ def recording_batch(messages, **options): -+ """Record each canonical batch size before delegating.""" -+ materialized = tuple(messages) -+ observed_message_counts.append(len(materialized)) -+ return original(materialized, **options) -+ -+ monkeypatch.setattr( -+ incremental_module, -+ "_batch_thread_messages", -+ recording_batch, -+ ) -+ index.apply( -+ MailboxChangeSet( -+ expected_version=1, -+ additions=( -+ _record( -+ "new_key", -+ message_id="new_message", -+ references=("message_0",), -+ ), -+ ), -+ ) -+ ) -+ -+ assert observed_message_counts == [1, 2] -+ assert len(index.projections) == 128 -+ assert observed_message_counts == [1, 2, 129] -+ assert len(index.projections) == 128 -+ assert len(index.roots) == 128 -+ assert observed_message_counts == [1, 2, 129] -diff --git a/tests/test_incremental_contract.py b/tests/test_incremental_contract.py -index dba9e71d57036ac6705cf7d6000403ae6ce5c19d..d374c831a34fd5731162c8fcfde83303cb74fcff 100644 ---- a/tests/test_incremental_contract.py -+++ b/tests/test_incremental_contract.py -@@ -171,6 +171,43 @@ def test_structural_metadata_is_copied_while_payload_remains_caller_owned(): - assert indexed.payload is message.payload - - -+def test_public_roots_are_defensive_structural_copies(): -+ """Caller graph edits cannot corrupt the index's reusable internal forest.""" -+ payload = object() -+ index = IncrementalThreadIndex() -+ index.apply( -+ MailboxChangeSet( -+ expected_version=0, -+ additions=( -+ _record( -+ "root", -+ message_id="root", -+ subject="Original", -+ payload=payload, -+ ), -+ _record( -+ "child", -+ message_id="child", -+ references=("root",), -+ ), -+ ), -+ ) -+ ) -+ -+ exposed = index.roots -+ exposed[0].message.subject = "Mutated" -+ exposed[0].children.clear() -+ -+ current = index.roots -+ assert current is not exposed -+ assert current[0] is not exposed[0] -+ assert current[0].message is not exposed[0].message -+ assert current[0].message.subject == "Original" -+ assert current[0].message.payload is payload -+ assert [child.message.payload for child in current[0].children] == ["child"] -+ assert index.projections[0].message_keys == ("root", "child") -+ -+ - def test_reported_external_identity_is_immutable_on_replacement(): - """RFC 8474 identity metadata cannot disappear or change after exposure.""" - index = IncrementalThreadIndex() -diff --git a/tests/test_incremental_parity.py b/tests/test_incremental_parity.py -index 6c87474c4683a1a5bab17a9ff8d4678b5dbe2abc..2162dd143383842d651c6e2403955233999b498a 100644 ---- a/tests/test_incremental_parity.py -+++ b/tests/test_incremental_parity.py -@@ -2,6 +2,8 @@ - - from __future__ import annotations - -+import random -+ - import pytest - - from threadweave import ( -@@ -10,6 +12,7 @@ from threadweave import ( - IndexedMessage, - MailboxChangeSet, - Message, diff --git a/tools/pr20-review-fixes.part-09 b/tools/pr20-review-fixes.part-09 deleted file mode 100644 index 63452cd..0000000 --- a/tools/pr20-review-fixes.part-09 +++ /dev/null @@ -1,177 +0,0 @@ -+ ThreadSerializationError, - serialize_thread_response, - thread_messages, - ) -@@ -105,6 +108,34 @@ def test_sent_date_order_matches_batch_across_unrelated_components(): - ) - - -+def test_implicit_sort_positions_do_not_leak_into_public_metadata(): -+ """Internal sort ranks must not become caller-visible IMAP identifiers.""" -+ records = ( -+ _record( -+ "earlier", -+ message_id="earlier", -+ sent_date="1 Jan 2026 00:00:00 +0000", -+ ), -+ _record( -+ "later", -+ message_id="later", -+ sent_date="2 Jan 2026 00:00:00 +0000", -+ ), -+ ) -+ index = IncrementalThreadIndex(sort_by_sent_date=True) -+ index.apply(MailboxChangeSet(expected_version=0, additions=records)) -+ -+ batch = thread_messages( -+ (record.message for record in records), -+ sort_by_sent_date=True, -+ ) -+ assert [root.message.sequence_number for root in index.roots] == [ -+ root.message.sequence_number for root in batch -+ ] == [None, None] -+ with pytest.raises(ThreadSerializationError, match="positive integer"): -+ serialize_thread_response(index.roots) -+ -+ - def test_global_sequence_number_collision_fails_atomically(): - """Separate components cannot hide duplicate RFC ordering tie-breakers.""" - index = IncrementalThreadIndex(sort_by_sent_date=True) -@@ -209,3 +240,132 @@ def test_implicit_sequence_positions_remain_stable_after_replacement(): - ("a",), - ("b",), - ) -+ -+ -+def test_missing_ancestor_creation_order_matches_canonical_batch_root_order(): -+ """A late missing-root placeholder cannot reorder an earlier independent root.""" -+ records = ( -+ _record( -+ "early_child", -+ message_id="child@example.test", -+ references=("late-parent@example.test",), -+ ), -+ _record("independent", message_id="independent@example.test"), -+ _record( -+ "late_parent", -+ message_id="late-parent@example.test", -+ references=("missing-ancestor@example.test",), -+ ), -+ ) -+ index = IncrementalThreadIndex() -+ index.apply(MailboxChangeSet(expected_version=0, additions=records)) -+ -+ batch = thread_messages(record.message for record in records) -+ -+ assert _preorder(index.roots) == _preorder(batch) == ( -+ ("independent",), -+ ("late_parent", "early_child"), -+ ) -+ -+ -+@pytest.mark.parametrize("group_by_subject", [False, True]) -+@pytest.mark.parametrize("sort_by_sent_date", [False, True]) -+def test_bounded_randomized_change_stream_matches_full_batch_oracle( -+ group_by_subject: bool, -+ sort_by_sent_date: bool, -+): -+ """Deterministic mixed mailbox changes preserve canonical forest parity.""" -+ option_code = int(group_by_subject) * 10 + int(sort_by_sent_date) -+ subjects = (None, "Topic", "Re: Topic", "Ｔｏｐｉｃ", "Other", "Fwd: Other") -+ dates = ( -+ None, -+ "1 Jan 2026 00:00:00 +0000", -+ "2 Jan 2026 09:00:00 +0900", -+ "3 Jan 2026 00:00:00 -0500", -+ ) -+ -+ for seed in range(4): -+ random_source = random.Random(10_000 + option_code * 100 + seed) -+ index = IncrementalThreadIndex( -+ group_by_subject=group_by_subject, -+ sort_by_sent_date=sort_by_sent_date, -+ ) -+ records: dict[str, IndexedMessage] = {} -+ ordered_keys: list[str] = [] -+ next_key = 0 -+ -+ def random_record(key: str) -> IndexedMessage: -+ """Build one deterministic adversarial record for this seed.""" -+ existing_ids = [ -+ record.message.message_id -+ for record in records.values() -+ if record.message.message_id is not None -+ ] -+ message_id_mode = random_source.randrange(5) -+ if message_id_mode == 0: -+ message_id = None -+ elif message_id_mode == 1 and existing_ids: -+ message_id = random_source.choice(existing_ids) -+ else: -+ message_id = f"id-{key}@example.test" -+ -+ reference_candidates = list(existing_ids) -+ reference_candidates.extend( -+ f"missing-{index}@example.test" for index in range(3) -+ ) -+ reference_count = random_source.randrange(3) -+ references = tuple( -+ random_source.choice(reference_candidates) -+ for _ in range(reference_count) -+ ) if reference_candidates else () -+ return _record( -+ key, -+ message_id=message_id, -+ references=references, -+ subject=random_source.choice(subjects), -+ sent_date=random_source.choice(dates), -+ ) -+ -+ for _step in range(24): -+ operation_roll = random_source.random() -+ if not records or operation_roll < 0.50: -+ key = f"seed_{seed}_message_{next_key}" -+ next_key += 1 -+ record = random_record(key) -+ changes = MailboxChangeSet( -+ expected_version=index.version, -+ additions=(record,), -+ ) -+ records[key] = record -+ ordered_keys.append(key) -+ elif operation_roll < 0.78: -+ key = random_source.choice(ordered_keys) -+ record = random_record(key) -+ changes = MailboxChangeSet( -+ expected_version=index.version, -+ replacements=(record,), -+ ) -+ records[key] = record -+ else: -+ key = random_source.choice(ordered_keys) -+ changes = MailboxChangeSet( -+ expected_version=index.version, -+ removals=(key,), -+ ) -+ del records[key] -+ ordered_keys.remove(key) -+ -+ index.apply(changes) -+ batch = thread_messages( -+ (records[key].message for key in ordered_keys), -+ group_by_subject=group_by_subject, -+ sort_by_sent_date=sort_by_sent_date, -+ ) -+ expected_shape = _preorder(batch) -+ assert _preorder(index.roots) == expected_shape -+ assert tuple( -+ projection.message_keys for projection in index.projections -+ ) == tuple( -+ tuple(key for key in root if key is not None) -+ for root in expected_shape -+ ) -diff --git a/tests/test_incremental_private_graph.py b/tests/test_incremental_private_graph.py -index 05efffe7ae5217f742c584f6bdc63366da9cf2f6..325ce7a1b07a7f0cf6674bcde93e03d5aa6e5fd6 100644 ---- a/tests/test_incremental_private_graph.py -+++ b/tests/test_incremental_private_graph.py -@@ -85,16 +85,148 @@ def test_projection_defenses_cover_dummy_cycle_and_foreign_messages(): diff --git a/tools/pr20-review-fixes.part-10 b/tools/pr20-review-fixes.part-10 deleted file mode 100644 index 17aa57e..0000000 --- a/tools/pr20-review-fixes.part-10 +++ /dev/null @@ -1,156 +0,0 @@ - incremental._projection_for_root(child, {}, {"a": record}) - - --def test_empty_projection_has_an_earliest_safe_sent_date_key(): -- """A malformed empty projection remains deterministically sortable.""" -- key = incremental._root_order_key( -- Container(), -- ThreadProjection(()), -- {}, -+ -+ -+def test_thread_delta_scans_projection_sequences_a_bounded_number_of_times(): -+ """Large independent forests avoid pairwise projection comparisons.""" -+ -+ class CountingSequence: -+ """Wrap projections and record full-sequence operations.""" -+ -+ def __init__(self, values: tuple[ThreadProjection, ...]) -> None: -+ """Store values with zeroed iteration and membership counters.""" -+ self.values = values -+ self.iterations = 0 -+ self.membership_checks = 0 -+ -+ def __iter__(self): -+ """Iterate while recording one full-sequence pass.""" -+ self.iterations += 1 -+ return iter(self.values) -+ -+ def __len__(self) -> int: -+ """Return the wrapped projection count.""" -+ return len(self.values) -+ -+ def __getitem__(self, index: int) -> ThreadProjection: -+ """Return one wrapped projection by position.""" -+ return self.values[index] -+ -+ def __contains__(self, value: object) -> bool: -+ """Record whole-sequence membership checks.""" -+ self.membership_checks += 1 -+ return value in self.values -+ -+ projection_count = 128 -+ values = tuple( -+ ThreadProjection((f"message_{index}",)) -+ for index in range(projection_count) -+ ) -+ before = CountingSequence(values) -+ after = CountingSequence(values + (ThreadProjection(("new_message",)),)) -+ -+ delta = incremental._thread_delta(0, 1, ("new_message",), before, after) -+ -+ assert delta.added_threads == (ThreadProjection(("new_message",)),) -+ assert before.iterations <= 2 -+ assert after.iterations <= 2 -+ assert before.membership_checks == 0 -+ assert after.membership_checks == 0 -+ -+ -+def test_projection_membership_rejects_duplicate_keys_between_roots(): -+ """Malformed projections cannot make merge/split classification ambiguous.""" -+ with pytest.raises(IncrementalThreadError, match="duplicate message_key"): -+ incremental._thread_delta( -+ 0, -+ 1, -+ (), -+ ( -+ ThreadProjection(("shared",)), -+ ThreadProjection(("shared",)), -+ ), -+ (), -+ ) -+ -+ -+def test_reverse_token_buckets_are_copied_only_when_mutated(): -+ """Atomic changes preserve every shared pre-transaction bucket.""" -+ original = {"token": {"a", "b"}} -+ overlay = dict(original) -+ tokens_by_key = {"a": frozenset({"token"})} -+ copied_tokens: set[str] = set() -+ -+ incremental._remove_key_from_buckets( -+ "a", -+ tokens_by_key, -+ overlay, -+ copied_tokens, -+ ) -+ incremental._add_key_to_buckets( -+ "c", -+ frozenset({"token"}), -+ tokens_by_key, -+ overlay, -+ copied_tokens, -+ ) -+ -+ assert original == {"token": {"a", "b"}} -+ assert overlay == {"token": {"b", "c"}} -+ assert copied_tokens == {"token"} -+ -+ -+def test_component_partition_reads_positions_linearly_for_disconnected_keys(): -+ """Disconnected mailboxes avoid a repeated-minimum quadratic scan.""" -+ -+ class CountingPositions(dict[str, int]): -+ """Count position lookups performed by the partitioner.""" -+ -+ def __init__(self, values: dict[str, int]) -> None: -+ """Create the mapping with a zeroed lookup counter.""" -+ super().__init__(values) -+ self.lookups = 0 -+ -+ def __getitem__(self, key: str) -> int: -+ """Return one position while recording algorithmic work.""" -+ self.lookups += 1 -+ return super().__getitem__(key) -+ -+ key_count = 128 -+ keys = {f"message_{index}" for index in range(key_count)} -+ positions = CountingPositions( -+ {key: index for index, key in enumerate(sorted(keys), start=1)} -+ ) -+ components = incremental._partition_components( -+ keys, -+ positions, -+ {key: frozenset() for key in keys}, - {}, -- sort_by_sent_date=True, - ) -- assert key == (incremental.normalize_sent_date(None), 0, 0) -+ -+ assert len(components) == key_count -+ assert positions.lookups <= key_count * 2 -+ -+ -+def test_public_forest_copy_rejects_shared_and_cyclic_internal_nodes(): -+ """Defensive copies fail closed if derived graph invariants are corrupted.""" -+ index = IncrementalThreadIndex() -+ shared = Container(message=Message(message_id="shared")) -+ index._roots = (shared, shared) -+ with pytest.raises(IncrementalThreadError, match="shared or cyclic"): -+ _ = index.roots -+ -+ cyclic = Container(message=Message(message_id="cyclic")) -+ cyclic.children = [cyclic] -+ index._roots = (cyclic,) -+ with pytest.raises(IncrementalThreadError, match="shared or cyclic"): -+ _ = index.roots -+ -+ root = Container(message=Message(message_id="root")) -+ root.children = [Container(parent=root)] -+ index._roots = (root,) -+ copied = index.roots -+ assert copied[0].children[0].message is None -+ assert copied[0].children[0].parent is copied[0] - - - def test_replacement_recovers_when_derived_component_mapping_is_missing(): diff --git a/tools/pr20-update.zip b/tools/pr20-update.zip deleted file mode 100644 index bfa40ba28d1ac6b77da14e6e11e5c16b5c160d22..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 12921 zcmaKz18`-}*62^{2_`zR?POw4tjWa5iJeTWiEZ1qjfrjBwr$Lt`@MJd|L%QVwRczT zUF+AYu@|~}%S%H*q5}W`*gqGP*6*?{Mh7SWU<3sKp#AfN!NknL)KQ=Dr?uS=V+-q_ z_Kap$26jf4Mph1b7WDc?RtBb)dUih;Tr4g2RV}So*-<_&bcCFtS}?Ln)Nf9Uj7yq~ z1$VL~;VrO<iBf#7_l4HXCO<*2GhMQQ}nXF;E+mblq5})9-YGkqRy(Qut*}7m})27viHInv@k>1 z7+aWsUv@KEGiq^Y8yJK+DN9bomZ9Qm*|*pmLpS;EB4MgvEHb^4z{nndj1u+eoWm89 zrHf>(NK~^t5hq1PyXIk|!0iByHz=4k?y|EVv3@<-`a@s>EPf1R*(iv=7H@eU3GT;t zs;Sv?Cct(`1h&>%S3XIGvK7GR3g!E`Ddn|ZRhj{CE)mA^VxOde5^26lYvYWL}jyv(s*k3(ArK1=H==0l0!9vL`2uh0s9~f8(ZeuyVX9@GlaRRh zodsz?=uB>>B+2>0{b2hs_?pw!1y5#Ooi}3OHb&Z$sEO2MkK3xuJq*}NKb`P^9v)oa$D6Lh?c?> zpoj}buaCPN(=h%KGSnBT#DUyLFin()+!aw7LDW-P_{33gi~sUQj{vWyi$gVwlUa@g zHb643_Kep@PZealik<2DHea{T9D)f2HG%1tvt~{Sp*V7Gduxa^B$Cg<{DbHUh_tSX zG+ir?_h732YS(xEo%eYq%7Y+WcMO|Q+~Oe8B+quNFs{qKnd4SWQ9$3Gg#XbbE{5ZQIQU?8V7T{QfjYlwVX-#iCGrrN2Rxma4Iwtn-7AKD5pDB(o8Mws zVbv;ye=vLzT}QjxH`WMz8hY0JEfoMJh*DB)B)%%;R}h$6U#?T!RD>fWNUOkaW`#tH zwSC|;wv_VuiXdLXsQ}?Iksbyk_q~hB^y?9&e&>D6J0ed7y9I_W-{mS-?+y)md|=CY zb2EW$FxdP`Syku(7f4&1;CBn+HECujPO z9lV|JjNg8%J+R#cMT?1Ex&zE!--TWTNw-SjP|yW{GP-DP~)t? z$DHW3SF9orTp$wfra;NM&|OV=X0~U#eDD-YKez>pWHr(y9}e(D?Nj zUJs88Z|;w)WJEDTvGLb+3I6UN3cSXD|Ji()b`daC8Xzi=59p=c`;_C61%zOjF zFUzyzmGDuaoSZ~|nT0P?ZDxKz%m174POVVFF+>Fb1StRjxIe~QNL*A-Ns+;uq zJxkeC_PKgHl`m2@4ejmTaGd(fNcl#l)`q(pd&2YL?~hr;)(y=0JbD6)L@mcgM(D+j zl-6nG1q%=BP}?h6=A{wzpN;swE_d&II@k34PThBnx7|n`n~Zf9d?0H}&Dr_Jn5w{x zFJ8}2n}v7h^1l`y3a=9?#`AEU4qnDXo*(v`HdF5Jm(%J_>qg`1kWXcjc2v6AMR|U) z(A~{jYj?0!BMVq+!TEWlS!Yzx&SH3U!Mv=D?y7F$t|l%Ojp#WhC}VobOd1W!qFN9P zj59Xkg}?LOi4NcI;0|tfFbOn8N+;K51M%gibaAn+6b^VgH0Ofu!OKTzpGI}Rt_rN5 zy{(_Lt8czJcrjWr1k9ngsdIrBP$_?&3I$gK4odu$bOdC4X#zhGIASgAE4##hln5e z+KgE0*jFyos>GeA?@K*#=7c1tUv1J>QoGZ%a-2;=aiY!WIL-$2%&e-Pbfx&4y1c(V zb;*rdb!sQypZQ-W!4Y3yVy7RQX@zC&U0PIhOquYMTf9DDRcP`+hI^%wY`M4YbU*Wp z2YahN>iR5Jb+8)mY$Zau34K;_kEvr(}m5btcIW%kD=@6+4)@jiT&>(@aoPz3;=AN%F6y= zS7FWsDhQ)zxZJ06QbRy}rX0HQyzZCABe&lYNfk1F7+qPvLqrL-z0Wa&7rw<#!jh%> zi=KcBQpp5a4g$gVs-sTGe`vg#S~!Ypn};mKmJCcn3XPs&MDYA*(aR#_Y&soe*Dw+A zJ2qWwu;ityaaW#(1O!5SNAR1fodUD#g4Em%y7TdlMRa!dGE?zOUUFMf?(M$^AKXz4 z;qUysCub~)YC7O*1BhSo{>-W!V5Uh_a9E2iS-cCT!F2N@P6iCUPcW-Pg`*j44E-C8 z1x%B>bd!6qdlr??2Pf0|UWy`ZpTImC*z%9Fb&mWWE+p8S#NEO{#qAUrJzi-2@2D(K zsM0+!!ZLT@?y{WHJ*l$i98Zq#7O!f2I`-80xn)IkI=}IeJ=sS0ZwPDBje$DP@uXSnE|Br6XzvJV*3dPb9#=LYlO+pDy}T zu{;~>>fkW#vsmdJ+gO7XO=nMXUiTL%wx#(fR)4lLHyr z=<;Ni1x!9EmYztYY6dWvmDepd!>~&`$@rsAYAAMC1d{PMzAX#e_NRsR^-lKwM6v{Q z8L4QfSg=!?(K@-ZaGWD_i7@xG#O24k&DLTX-*+}CF!Si*D-)r~(dpwR*1|rSRYcC9 zm{gcZr>EG!5JWQV!iJTaF z+pkJI<-dBIgISeJyR_g3eN=(i1EO(8eFr@iP@z_R`##OdW{T6|r5m>~lm-zV2oR({ z5q<|X9`#H>6fPt0@EcKr@sdM#`G9`~-z3gRt`PqHd0driB0fsq1NP!2qcS}!3;y9@ zu3B$r2q-ji9BuH$goO^{dg zVBu%zp`p~=xB7R=I}Gy}WE;6foKq3`eOoo>$QN3vir*lIT}y&TKMTSpMv-qBl5&xL z#*ia=O>B@Pw+kO_1XS$C=klm3Pk(<5!K>y=Agh-BNh;|wf;)36KW}AF4*OfUv&+q6 zk{9FS-atnxQEr;LXK#I+|GXu+5w7G~-8;klA>gTm2FHEG*GV;Ap-<7Tzp+|shs~nZk1@f$%ObYw}y3ON>{~5}30}`R8Wp!9b8mo)m4s%eMss#Ms=J1`RfU`GuL0 z8BxVia#0MPlqbJ4*x=x3N~q}6_-Vr!9m~t>pEf_qe66()aI2?eAPCkHPAU{`t>>in zOm&Ob#>0&hP#09bFnTXx^o1SW5aCg=FP;^d_^_SE#Z&w+g_h345S~Z#t&B)OAvc~J zi>>_VCRmWJpmeaB4h}FfYb7iGR>`X?NJ{k`+B4qAC1cIPUD ze%z`2M@eplZx%gB4wlgc7rR4z0bGyLxJJO$08~8+ahW?q`d~)T9232#Jlg~ZW`eZ9^s(m# zByqD18))#sX@VkFWLp*`>daTsI~)w171DhDVby=;&3;;Dm@(`S%{3cDhc0Cav6R`a zX}8`sbL^UHgRn1gWr&T>k7b`r1RIGf5Yliw7nvo4^*k5);xSD-Op*IgdWFSmW|NHI z{P2^Y5}{e*2}tHJQv?3)jhK&OhHfEVX(~2xdD@;mVyloGK;LA_z;6?$K~VaY$0*(*oY~@o8^um)ZpKj0s=iC|5g)$? zVux3|z;aF>LqLj7d!#P&$Lbd}0|K(#Fgfe#xyx{JRSKF-N5thyb=g`Bb4c-V_3kTw zMhIp!EUyov1UT}0`g_Kua({`~<9u8mVH4Ge7`Y!g)k}G!QZ^@9rL^aNgZd(-PF@Es z2@pBH)`L(Qq;aTiHZ9(hj@GwxgT_@z~(CMZ=+XtBB1Mnu&Ymcz`jb45U_{bi-9l zWB)<}Yh`7jmRif$3p*k0TZ97=t?dSivNU^LD_sg1a$^UuVLvF?r{#8L;T-opP{i{@ z=XKHM$$ZsAa#CpXVbk`FDvBZJv$>f)xjA3%pI5l_nAIT-zXq z>^AStMl-39%g0ah$#US*A3YA6uHZ9uwtdmK_z(Z8s~$+oi7eClt0>5$ajL%9D_hvT z{POsY_`S~oAurJ#VRh2+0F2n!$g@-%@Aip6!$4%_P0YuK#0E2bGfO=8Q(S?WilZD4 zO0quG@-eHbE%WN+AyinNmf~WwbcuqB-D>g)aMick}Of z_RrT}71pJLn~A8YGV$kwfw)eJKZ>@z=ob`o1c@cS74+T=vVYp5C8Cpn7Siw7b#oXA z+ui*w;Rf#!D?~O`@7{Z1QB<6CK=I(mlC^N%5D74*CDI~(@D|u82!0|lHbRrjG!_ig zxD<*P@l0ZwF^qFGTx-!U_jRzlqP50C=YFIA384e6ixKlo9&W1`CXm9~Y1WB5>}Gr~&kG9Vl~WLrmaN%>!U>{^1B+q2^He%8FDt3HAnL!Mo^=*Q z_y01hUP>KMuiBrptMtJA{X35TwQ=`7!uVLa@4jSMJy_Y9+EqTQt3sfXC4AJNQerdh z&8CFNtj&OPt>L_1MRh0I>1H-r<0~DgYLVzpYr(8}7^soddPmpY@FVEHj4{gi=eYQ-TepBa zu@?Hrc5m~0cQnlg z4b%QUXz%5H3u7E;-ueAIHSPfM2Q7Z@)It_)^Ep}>%hZKl}iSquLg0lkDC(~g*^=7a%|sFalEYUjln$upTHTl)UbEyVr5G)5fimjIPQlpxfOT4*hq@O%OQnh5~_ zNX3%fuz%s{H z?hA`zLb_SbXv&0rhP2|aR5PZ&Z9^%MB+qd}D{~6LD60u>+A{d2b5!fQ#6jtI)__iN z;Ra`13ZwExiG_)cWV}PxB74_tcswF}U0c0H@(xmZTzdTF0G`!>T}Y)_k{4aNzC3Ze zTH7L;y&mCM6u10niv1`5@>O-cH#8OZr|T@9!%e)Q!`De2!9|4_mCSXs5sg+ZvSD&w zbNqUGGA>8Dhrt_A#gCHh)do#YWk#c2bhno1UmM-kX4++NK)p z3_r;!cQjT?(yX)R+rh?)Y`O2QghO!rY$T(EN?I_KnyurP+ncjuG8(o7n76~>RgqlFY`&OEo#8fWIj!9F-18doCU(NA!WYi!aOU>N@r$Ub$Hp&Pwg=Qg$Dim0^JsUmCl`({qlKJVH8D$_z zc`a}P08T-MwVk&>eoK8_7K-+7Ns8gkz16bEHym037Kn(kx#5#~Mj4LYeU{o8$n$mU zq;F@V=gl{s6IyV{2!OB=b%}rF7NlwoR71@J)7cLA>bZS%(xrlKwuZ=Py6Ime{&tsE zc6_}jqun7Okcwy3{S>#gxS@sL8=LZu=MxxyhUn_`WgYNEX_#97o@>bK+vQpbS3%hX zjHj43`YDeK=hXH4fj=*fd!{y2O;C#|`s=uA>4P)^hOaAOrDxL)6glPgmyfqd0k7UH zfvZEaw{RxKJ3?DFr2WqZ!B*w^*h|#HWGoTW1dFHW{makEH0xncaifc~@9)V4w`$7! zYeyt7Q$%FoTd4pP(VTiAfARR3#y}zoyN91y?IPMcmM3Tq^?nFh;8Gq*1vzHFc#k!v zFyW01jBsM&10j8p@l?S^h|x>HH_GN)s%enfms9T|MH2<(cN|Y}cDE`GD%?-mxjq@N z88Js2N_~Y|tQ&P7<*ptGROm1yyEDp6l}FRb#R<~DmZF}>?ZC{bu1&(5Kb{zDoqRMj z->9dmxiXHWKEZONQFX;%L)3D#nIqyoChm|`-*#}db8+xwd3NS(=9Oxexs!4n5CI`y zB;0g3hGiScuyoKZ+Pgp0THrFMdJGbOy|1ZuU$0RX1EQ1~F2x=`73uHu^!NZKbxyE+< zJ^`0SV1UMFQXzvfo{MaO*UdM#?RXe&2p+P_ zqV-rn8dfj{$UH>Ns9=~ly2h$f6de1nu%>5HK@$(^pYuYYSdJ0MRG(|D(uVpPImz;K zx)rh^{7zfl{LV-yJ3kDbdp9?JxAFa{Hoe-&SGeee7R7R=3W6l;=2)Oq=~^zy;BQys z#TY#MG*@VC3m$IrNLJ?MtII@W|$t&wA%dPr|N{yyTcooilp}r$wm-L3iAMl&mnYO^L&`6gwG3N7M;|A00d?_3G!wm@M~EVXY{j-vh{#q($sy039-Hw!S+w8uCQ(8t4J?Sjn8 ze@0sc;u4v&2cxzzl_M^NaA2fluzm^7&WYMLq}7EMHXK&z7!1_Sj-xZJ7uj_+@4~58 zUnY|js~F6wceGqATJKY};Mg(1vPa0OQ7@nQ*xi(^zU|GO+h1O`@}alFA@|;VKj}dU zaW5w86eTiln_`t-iLl;4xAzp*NX=KP8;+l#f@f1CNaZUf$c_2-IOidS{-u$d-H&Em z4t%7e3mGO)T>eqW30SLT+JKO$VK3r&|1EH&ve%oq4<$&WLpLBH@!2>+sh$N?R4K0e zYg|Gs2s@BDj{Cy?BCdAj%B9ZR$L#`UO47#5-|{*2ot&$>nZ^twOKgjkZws*>ML}@N ziM4*28OgunN{n<@5eSdQvPrhjmJKV%FoC}TA-rg`}hJ8ss82pJ()Lo_8;bCiz3BS8*stjG{i>dZH?xO=#AFf_1i3H6!s;30a674f^X>9eZ$fV9K$y z?1uD&=Uh{YEAQ3DU<>>}=YMPf8tCyoF z`2wP*SynbI)8iO+nt70@+%{zM+%5J^+WNXK)_QBE3T*r<-^)9G#U4BVI2Tj$&FG=d#j79?WSi_$eiYiz{z)v;EzWXSQ(3rBT&gcxf8NkmVp={MIF z@|k|!M<#prYX}4(W?xA2@_zPEI^y)D+JWYS0;0EAs|emTuM2iJ(dN;;CVSk(U2~vj z_eUA4S8&U%I$`SEqe_|1Bb4j8+|$-K-BEHWEg7~5S`9;uc=!ptE#(d9PyI63Z2())T#J=P0 z<{AwP-+Mbsf(^C2*YbESrEKtLay!>q+S*7iO|cSSAhHMUrbI#)rqq=}wn}_7A}-## zk?e_H+|RC&ynD^;OD4X4I^nNUgj-7-y~xg~|57;czEa27G?9Nz{VQnXDzL7+$ucCi z8W7wUxFn_QU(dlDr?}gm^vkJYq8Zt3z-R}eA(7C__3>F5G2}H*ZRid5tRA~kX|T6W zlRT|Hu78>7`T2$gr598yftm@!8*R;&i&J?jJ>dC|*B04I zubSKnE5fjpEP)5tIDU50Qs?M>uw`y@Phcu#hCB9$p(w5%GyGoo)rJn@Ae;Lo+$^G! zOM-NQiGQ^DF8PPzDD1eXm6QLq8CeQTw}=5$PH72&g|pExhl6Voq`4Z65t++|{Qw7*J%D!P&TOjtlhUD_8v*Gab&Npl!GR)j!8y9Ix*>}uXiG#{x2 zGYve`RFi}Ot6mn9^KkJ(I0v%?^3?9{QsNDOCEnYUKh!cjt+^*DN>m&@N(j2*3>E$uHSi>IfX zsb;@_7jLRPj^8}oEd4o%U=oE)AXj=5t}QQhmRQ!B@1j}ewWl((ff;VzW#eV=i!g4D zV7j{Ay)48Vbri88ME@t7affdDu;5xdl9HmT1^Qe6RnT52NnIVsxrpnQS{ER|T*!GS zxyzegdV_q=+^t|WPZ7ZpU%C8YN#sF{NxLHzv9m!u$xu}>5NPpd+e&-$pHHS z@MwC@aby)^H_X3Q9COhEN8`4~oED34D9 znooxj%OxPje4k(bv6ea1mJuQd*ba>0K1gtqWAWD)7ff~SLk;dwYssp3xE-1vL zzss;7{K>lK3m8FMUHP;{k(lpI=#2{(dyyE@&>p2-qSMQZ;tm~WNwt$Sk1MRe>j#NA zZF!u!R*RDj-VD}UZN-_vMtDzzD?~|puD>I#(C;M3$T8((ZwFD#f@)I#%NKq`Mj>N| zXJ3IW$bPOH19xVdwk!8Q=k@|jE$XT-e_8xPC&_r0u%Q;f;7N6&f~UZW=*W}C95r@T zvlHlVT|M9gAt^V6{qpSOV6|YlJAdWb{YqBgPvJ}2GlBONiQbQnoUVkc3IJ_i)=VB| z5*Y)O_V#j_UPnN{hnG$7XVOORAH}3e(}MQ~s=7g5gN_>G;^e7k7n#r|hp_h4wRm1L z30{TpzSJ|`e4WmFv_!?^cfc|Bba`&+`?iNz zHV$fhoSCpyKeAl8#9tiE}8Mv`>QX6&xi08|NIt|z2X6L?1)nVG zc3wBgro*(TbUzvY`Ys%fT)HzP_{1gd&k+YvVRS_;ZM7lNe@!r0M;xPnGY3 z)^0(5=-=Il@9wPTTiy?&9|p9*_qB$(HGA#P&O0I{84WHT%McbU?zvieBQ1fu zv?8W-OAZ+eQ{2Z}8@&v(+Yh4Z>$@ehsyt63Fk*ZaIg7ej8mg|qNA=4E?B}yJjyyw# zjubQtJiQ-#iNLvK)F)|(ac(QGx;$x(>T0-eTwvP51;N#I4pa`64oI~qp9+c(E{JAr zUi=Z7ET%wXcOSnOpEy5&Cfu+L6Rc2Dt6SK%Gn2XJ@^X>ChgNxmTegzb!uu|0e=!VD zw$JvyvTL~v@bp*uP+$qA$nA^?UeSSMv%mpSQy@Qx=~>d%c7(NK&ckCSOPp)2p);>o zYjn*}x?kW1{9qdr*w^+NNMQ1@%O%mDq2s@Rx5S)z?CLpBf-twCMkqL+cE%jnvQE&} zKXjoDe&z%r)n!O~-hB2x+C6oaWKtdzmYA)m#bD zz?_(l1OYnl4jOU^E5n*PKk)L5l#^`*%u}H{ONBF(9a!=wLIZc6+JI@!#0zp+UP4;@ zCE;+CPrdhPd-6C3SB2jt+`Vj=J-&L(?;Qwu|GNt`w*Ee&)YSz5f_npC{@n$xt*fP` zsI6nI^)(T)nSOjNF6`e8g9`GG(; zHG+X@QORvExC_3|6Kr8!q1IRI>_0)=P~O?z8xNN1&vn`y<(EpL$V$B-+mR=ta`dd# ziO>_s6I))%UPDf>S$f90I}Wz0G_^Vbs3!J}+%OzpV4-@R$$JosR-T@~RBDF@$j1xH zuFvaVgkb;=hbi%mwDnHyYCK~q_$V*8+%$l9bc*zKVkKSj74mrEY$zH^7_@O%C=FtD zsdTV#QLdwX(NtJn)9<@aCRwiv+uQXz#IfmIAo}s1x|AZ95kHz10hwQ3B@K&hYm`z zY2sE03{>lx)Q)mz*cEgD;yH+v*&mMV1Adh<`)*N53A*`=Xt^%?t=utS)xy)(3F?~J zoz&gvNonFrr?27ceg8CdFYWLyf#ytwBMo8manv89%`olmFDdY1HjUpbIA{UDFqH4%qetwNe0%+evP)g2!F z^i4s$-SA4{la8?;e>dYwEWtb+*x zSo{J16O{gW@jr!{NbpaMkg}qMgRd60zPgsWoQAu;Hu^7-DuP-HD0u}}O*3^g69hv4 z3)r|!89D`05P$;b4FLZe>;DyMB5R>PPrpwqCZLa|ssB|>biSWNuGbsDAl?Ol==!@5 z06)*s; zy#YYnrM1x|b-b`KL_|bswPY>q6$H>)vf3(8T0LzoOyi$HKZ4djtlt0bmDeZIE)aw_ zfHnjGfcZDw?B@QM(!VwGue0+1Lz!LOM9c>Y0vIB@0Dp$_ABCDoGx`7NyOyH4tfrZ~ zf-v{rI<1C6c-HV|W-+J!wb{7C|X)RR~A8&N&?~CK#w8L6e(*R+oZ6Kr* zDE(!@d~4(*S{d#FP*(l%4c`Ay?a$GJD5Q$Lmg@V5_J{blza{=& zsEOp+yZ#&TAF1VE_4|6zitQyN2tW<^L+tuH>u(>RE9ZZw;iYONYXMc4*OE1H{=?k< zHTC^FmYPVa<-d;Ne@cY^s_kC`ihqKo-T!Sg|5JFOgw`$O{YL^)5K<5VOc226@i+1R E1LDjZ7ytkO From 10a02a3abdf038c0b87397f7d34f9c7285691118 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:10:50 +0900 Subject: [PATCH 38/97] ci: schedule reproducible incremental mailbox benchmarks --- .github/workflows/incremental-benchmark.yml | 96 +++++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 .github/workflows/incremental-benchmark.yml diff --git a/.github/workflows/incremental-benchmark.yml b/.github/workflows/incremental-benchmark.yml new file mode 100644 index 0000000..7296aa1 --- /dev/null +++ b/.github/workflows/incremental-benchmark.yml @@ -0,0 +1,96 @@ +name: Incremental Mailbox Benchmark + +on: + workflow_dispatch: + inputs: + messages: + description: Number of existing mailbox messages + required: false + default: "100000" + type: string + schedule: + - cron: "17 3 * * 1" + +permissions: + contents: read + +concurrency: + group: incremental-mailbox-benchmark-${{ github.repository }} + cancel-in-progress: true + +env: + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true + PYTHONPATH: src + +jobs: + benchmark: + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Harden runner and block undeclared egress + uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0 + with: + egress-policy: block + disable-telemetry: true + allowed-endpoints: | + codeload.github.com:443 + files.pythonhosted.org:443 + github.com:443 + objects.githubusercontent.com:443 + pypi.org:443 + release-assets.githubusercontent.com:443 + results-receiver.actions.githubusercontent.com:443 + *.actions.githubusercontent.com:443 + *.blob.core.windows.net:443 + + - name: Check out the reviewed source without persisted credentials + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed benchmark toolchain + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Run the isolated mailbox benchmark + env: + BENCHMARK_MESSAGES: ${{ inputs.messages || '100000' }} + run: | + set -euo pipefail + python benchmarks/incremental_mailbox.py \ + --messages "$BENCHMARK_MESSAGES" \ + --thread-size 10 \ + --output incremental-benchmark.json + + - name: Enforce parity and the mailbox-scale delta target + run: | + python - <<'PY' + import json + from pathlib import Path + + result = json.loads( + Path("incremental-benchmark.json").read_text(encoding="utf-8") + ) + incremental = result["incremental"] + full_rebuild = result["full_rebuild"] + assert incremental["projection_sha256"] == full_rebuild["projection_sha256"] + assert incremental["affected_message_count"] == 21 + if result["message_count"] >= 100_001: + assert incremental["delta_apply_seconds"] < full_rebuild[ + "full_rebuild_seconds" + ] + PY + + - name: Upload benchmark evidence + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: incremental-mailbox-benchmark-${{ github.run_id }}-${{ github.run_attempt }} + path: incremental-benchmark.json + if-no-files-found: error + retention-days: 90 From f9728b35dff77631c500c7a2fd6b3b92b92b66d4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:11:59 +0900 Subject: [PATCH 39/97] ci: remove the completed PR 20 bootstrap job --- .github/workflows/ci.yml | 107 +-------------------------------------- 1 file changed, 1 insertion(+), 106 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3e608cf..6273f8b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,111 +16,6 @@ env: PYTHONPATH: src jobs: - # BEGIN PR20 SEALED UPDATE - apply-pr20-update: - if: >- - github.event_name == 'pull_request' && - github.event.pull_request.number == 20 && - github.event.pull_request.head.repo.full_name == github.repository && - github.event.pull_request.head.ref == 'feature/incremental-thread-index' - permissions: - contents: write - runs-on: ubuntu-24.04 - timeout-minutes: 10 - steps: - - name: Check out the exact pull-request head - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Verify and extract the sealed update - shell: bash - run: | - set -euo pipefail - expected_blob='bfa40ba28d1ac6b77da14e6e11e5c16b5c160d22' - actual_blob="$(git hash-object tools/pr20-update.zip)" - test "$actual_blob" = "$expected_blob" - python - <<'PY' - from pathlib import Path, PurePosixPath - import shutil - import stat - import tempfile - import zipfile - - expected = { - '.github/workflows/incremental-benchmark.yml', - 'AGENTS.md', - 'ARCHITECTURE.md', - 'CHANGELOG.md', - 'README.md', - 'docs/adr/0001-batch-oracle-for-incremental-threading.md', - 'docs/incremental-threading.md', - 'docs/research/README.md', - 'scripts/benchmarks/incremental_mailbox.py', - 'src/threadweave/incremental.py', - 'src/threadweave/threading.py', - 'tests/test_incremental_benchmark.py', - 'tests/test_incremental_components.py', - 'tests/test_incremental_parity.py', - 'tests/test_incremental_private_graph.py', - 'tests/test_incremental_randomized_parity.py', - 'tests/test_subject_threading.py', - } - archive = Path('tools/pr20-update.zip') - with zipfile.ZipFile(archive) as handle: - infos = handle.infolist() - names = {info.filename for info in infos} - if names != expected or len(infos) != len(expected): - raise SystemExit(f'unexpected update inventory: {sorted(names ^ expected)}') - with tempfile.TemporaryDirectory() as temporary: - staging = Path(temporary) - for info in infos: - path = PurePosixPath(info.filename) - if path.is_absolute() or '..' in path.parts or info.is_dir(): - raise SystemExit(f'unsafe archive path: {info.filename}') - mode = info.external_attr >> 16 - if stat.S_IFMT(mode) not in {0, stat.S_IFREG}: - raise SystemExit(f'non-regular archive entry: {info.filename}') - target = staging.joinpath(*path.parts) - target.parent.mkdir(parents=True, exist_ok=True) - target.write_bytes(handle.read(info)) - for name in sorted(expected): - source = staging.joinpath(*PurePosixPath(name).parts) - destination = Path(name) - destination.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(source, destination) - Path('scripts/benchmarks/incremental_mailbox.py').chmod(0o755) - PY - - - name: Commit the extracted update and remove bootstrap material - shell: bash - run: | - set -euo pipefail - python - <<'PY' - from pathlib import Path - - path = Path('.github/workflows/ci.yml') - text = path.read_text(encoding='utf-8') - start = ' # BEGIN PR20 SEALED UPDATE\n' - end = ' # END PR20 SEALED UPDATE\n' - before, remainder = text.split(start, 1) - _, after = remainder.split(end, 1) - path.write_text(before + after, encoding='utf-8') - PY - rm -f \ - tools/pr20-update.zip \ - .github/workflows/apply-pr20-update.yml \ - .github/workflows/apply-pr20-update-pr.yml - rmdir tools 2>/dev/null || true - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add -A - git diff --cached --check - git commit -m 'feat: harden incremental parity and benchmark evidence' - git push origin HEAD:feature/incremental-thread-index - # END PR20 SEALED UPDATE - lock-integrity: runs-on: ubuntu-24.04 timeout-minutes: 10 @@ -161,7 +56,7 @@ jobs: - name: Install the reviewed CI lock run: python -m pip install --require-hashes -r requirements/ci.lock - run: ruff check . - - run: python -m compileall -q src tests scripts + - run: python -m compileall -q src tests scripts benchmarks - name: Run production doctests run: | python -m doctest \ From 49fbe0b7851ada44ea802fc6b224b42f34f55ddc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:12:33 +0900 Subject: [PATCH 40/97] ci: remove the completed push bootstrap workflow --- .github/workflows/apply-pr20-update.yml | 150 ------------------------ 1 file changed, 150 deletions(-) delete mode 100644 .github/workflows/apply-pr20-update.yml diff --git a/.github/workflows/apply-pr20-update.yml b/.github/workflows/apply-pr20-update.yml deleted file mode 100644 index a1fe9e0..0000000 --- a/.github/workflows/apply-pr20-update.yml +++ /dev/null @@ -1,150 +0,0 @@ -name: Apply PR 20 verified update - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - tools/pr20-update.zip - - .github/workflows/apply-pr20-update.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-update - cancel-in-progress: false - -jobs: - apply-and-verify: - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the update branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Verify and extract the bounded update archive - shell: bash - run: | - set -euo pipefail - printf '%s %s\n' \ - '2b18426b1afa0b8001753e1e3c372f9ec33af1ba793d7052fee07ec0aa174523' \ - 'tools/pr20-update.zip' | sha256sum -c - - python - <<'PY' - from pathlib import Path, PurePosixPath - import shutil - import stat - import tempfile - import zipfile - - expected = { - '.github/workflows/incremental-benchmark.yml', - 'AGENTS.md', - 'ARCHITECTURE.md', - 'CHANGELOG.md', - 'README.md', - 'docs/adr/0001-batch-oracle-for-incremental-threading.md', - 'docs/incremental-threading.md', - 'docs/research/README.md', - 'scripts/benchmarks/incremental_mailbox.py', - 'src/threadweave/incremental.py', - 'src/threadweave/threading.py', - 'tests/test_incremental_benchmark.py', - 'tests/test_incremental_components.py', - 'tests/test_incremental_parity.py', - 'tests/test_incremental_private_graph.py', - 'tests/test_incremental_randomized_parity.py', - 'tests/test_subject_threading.py', - } - archive = Path('tools/pr20-update.zip') - with zipfile.ZipFile(archive) as handle: - infos = handle.infolist() - names = {info.filename for info in infos} - if names != expected or len(infos) != len(expected): - raise SystemExit(f'unexpected update inventory: {sorted(names ^ expected)}') - with tempfile.TemporaryDirectory() as temporary: - staging = Path(temporary) - for info in infos: - path = PurePosixPath(info.filename) - if path.is_absolute() or '..' in path.parts or info.is_dir(): - raise SystemExit(f'unsafe archive path: {info.filename}') - mode = info.external_attr >> 16 - file_type = stat.S_IFMT(mode) - if file_type not in {0, stat.S_IFREG}: - raise SystemExit(f'non-regular archive entry: {info.filename}') - target = staging.joinpath(*path.parts) - target.parent.mkdir(parents=True, exist_ok=True) - target.write_bytes(handle.read(info)) - for name in sorted(expected): - source = staging.joinpath(*PurePosixPath(name).parts) - destination = Path(name) - destination.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(source, destination) - Path('scripts/benchmarks/incremental_mailbox.py').chmod(0o755) - PY - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Run repository verification - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - python scripts/benchmarks/incremental_mailbox.py \ - --messages 10000 \ - --component-size 100 \ - --minimum-speedup 0 \ - --output /tmp/incremental-benchmark.json - - - name: Commit the verified update and remove bootstrap files - shell: bash - run: | - set -euo pipefail - rm -f tools/pr20-update.zip .github/workflows/apply-pr20-update.yml - rmdir tools 2>/dev/null || true - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add -A - git diff --cached --check - git commit -m 'feat: harden incremental parity and benchmark evidence' - git push origin HEAD:feature/incremental-thread-index From 6cbfa46e12286d3c8a72ae7b6bce5586d17542cf Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 17:12:46 +0900 Subject: [PATCH 41/97] ci: remove the completed pull-request bootstrap workflow --- .github/workflows/apply-pr20-update-pr.yml | 133 --------------------- 1 file changed, 133 deletions(-) delete mode 100644 .github/workflows/apply-pr20-update-pr.yml diff --git a/.github/workflows/apply-pr20-update-pr.yml b/.github/workflows/apply-pr20-update-pr.yml deleted file mode 100644 index 7de358c..0000000 --- a/.github/workflows/apply-pr20-update-pr.yml +++ /dev/null @@ -1,133 +0,0 @@ -name: Apply PR 20 verified update - -on: - pull_request: - types: [synchronize] - paths: - - .github/workflows/apply-pr20-update-pr.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-update-pr - cancel-in-progress: false - -jobs: - apply: - if: >- - github.event.pull_request.number == 20 && - github.event.pull_request.head.repo.full_name == github.repository && - github.event.pull_request.head.ref == 'feature/incremental-thread-index' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact pull-request head - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Reassemble, verify, apply, and repair the reviewed patch - shell: bash - env: - PR20_PATCH: ${{ runner.temp }}/pr20-review-fixes.patch - run: | - set -euo pipefail - cat tools/pr20-review-fixes.part-* >"$PR20_PATCH" - printf '%s %s\n' \ - '14216e3eac0bee274542e487caf06984f36741ceb81bf8bfcd5428a9dbd69c41' \ - "$PR20_PATCH" | sha256sum -c - - git apply --check "$PR20_PATCH" - git apply "$PR20_PATCH" - python - <<'PY' - from pathlib import Path - - source = Path('src/threadweave/incremental.py') - source_text = source.read_text(encoding='utf-8') - obsolete = 'from threadweave.dates import normalize_sent_date\n' - if source_text.count(obsolete) != 1: - raise SystemExit('unexpected normalize_sent_date import state') - source.write_text(source_text.replace(obsolete, ''), encoding='utf-8') - - tests = Path('tests/test_incremental_components.py') - test_text = tests.read_text(encoding='utf-8') - anchor = 'from __future__ import annotations\n\n' - if 'import pytest\n' not in test_text: - if test_text.count(anchor) != 1: - raise SystemExit('unexpected incremental component import state') - test_text = test_text.replace(anchor, anchor + 'import pytest\n\n') - tests.write_text(test_text, encoding='utf-8') - PY - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Verify the patched repository state - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - python benchmarks/incremental_mailbox.py \ - --messages 10000 \ - --thread-size 10 \ - --output "$RUNNER_TEMP/incremental-benchmark.json" - - - name: Commit the verified product update - shell: bash - run: | - set -euo pipefail - rm -f .github/workflows/incremental-benchmark.yml - rm -f tools/pr20-update.zip - rm -f tools/pr20-review-fixes.part-01 tools/pr20-review-fixes.part-02 - rm -f tools/pr20-review-fixes.part-03 tools/pr20-review-fixes.part-04 - rm -f tools/pr20-review-fixes.part-05 tools/pr20-review-fixes.part-06 - rm -f tools/pr20-review-fixes.part-07 tools/pr20-review-fixes.part-08 - rm -f tools/pr20-review-fixes.part-09 tools/pr20-review-fixes.part-10 - rmdir tools 2>/dev/null || true - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add -A - if git diff --cached --name-only | grep -q '^\.github/workflows/'; then - echo 'Refusing to push workflow changes with the repository token.' >&2 - exit 1 - fi - git diff --cached --check - git commit -m 'perf: harden incremental mailbox updates' - git push origin HEAD:feature/incremental-thread-index From 6f00b1a0595d4c13a06d4c8b7798082ed76f06a4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:48:10 +0900 Subject: [PATCH 42/97] test: stage verified incremental concurrency fix --- tools/pr20-concurrency-fix.patch | 255 +++++++++++++++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 tools/pr20-concurrency-fix.patch diff --git a/tools/pr20-concurrency-fix.patch b/tools/pr20-concurrency-fix.patch new file mode 100644 index 0000000..fabe3e2 --- /dev/null +++ b/tools/pr20-concurrency-fix.patch @@ -0,0 +1,255 @@ +diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md +index cfce2c1..e7c1bad 100644 +--- a/ARCHITECTURE.md ++++ b/ARCHITECTURE.md +@@ -43,6 +43,11 @@ is maintained in incremental code. + ## State and mutation policy + + - `IndexedMessage.message_key` is caller-owned and immutable across revisions. ++- Public reads, snapshots, and `apply` calls on one index are serialized by an ++ in-process reentrant lock. Two transactions targeting one version cannot both ++ commit; the later transaction observes the new version and fails explicitly. ++- The process-local lock is not a distributed lock. Naruon or another host must ++ serialize durable writes across processes and persist the optimistic version. + - `apply` validates and computes on isolated transaction state, then commits once. + - Reverse connectivity buckets use copy-on-write mutation. + - Complete root/projection views are lazy caches invalidated by a successful change. +@@ -70,6 +75,7 @@ existing messages. + ## Integration policy + + Naruon and other services should own persistence, tenancy, authentication, +-mailbox synchronization, and external stable-ID policy. They pass immutable caller +-keys and `Message` metadata into ThreadWeave and consume `ThreadDelta`, snapshots, +-or IMAP presentation output through public interfaces. ++mailbox synchronization, distributed write serialization, and external stable-ID ++policy. They pass immutable caller keys and `Message` metadata into ThreadWeave and ++consume `ThreadDelta`, snapshots, or IMAP presentation output through public ++interfaces. +diff --git a/CHANGELOG.md b/CHANGELOG.md +index 07cd9ad..4a4197f 100644 +--- a/CHANGELOG.md ++++ b/CHANGELOG.md +@@ -6,6 +6,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + + ## Unreleased + ++- Serialize in-process incremental readers and writers with a reentrant state lock, ++ so concurrent transactions targeting the same optimistic version produce one ++ commit and one explicit `VersionConflictError` instead of a silent lost update. + - Keep implicit RFC 5256 sent-date tie-break positions internal to the incremental + engine instead of exposing invented IMAP sequence numbers on public roots. + - Return defensive structural copies from `IncrementalThreadIndex.roots` so caller +diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md +index 6aa08c2..4462f05 100644 +--- a/docs/incremental-threading.md ++++ b/docs/incremental-threading.md +@@ -102,6 +102,13 @@ assert updated.version == 2 + with the old version therefore fails explicitly instead of duplicating records or + edges. An empty change set is idempotent and does not advance the version. + ++One `IncrementalThreadIndex` serializes public reads, snapshots, and mutations with ++an in-process reentrant lock. Two threads that submit the same expected version ++therefore cannot both commit: the first successful transaction advances the ++version and the second receives `VersionConflictError`. This lock is process-local; ++services that share mailbox state across processes or hosts must still serialize ++durable writes and persist the resulting version in their own storage layer. ++ + ## Affected-component recomputation + + Each record contributes connectivity tokens for: +diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py +index b7fed9f..7fec91e 100644 +--- a/src/threadweave/incremental.py ++++ b/src/threadweave/incremental.py +@@ -13,6 +13,7 @@ import json + from collections.abc import Iterable, Mapping, Sequence + from dataclasses import dataclass + from datetime import datetime ++from threading import RLock + from typing import Literal + + from threadweave.collation import unicode_casemap_key +@@ -834,6 +835,7 @@ class IncrementalThreadIndex: + raise IncrementalThreadError("group_by_subject must be a boolean") + if not isinstance(sort_by_sent_date, bool): + raise IncrementalThreadError("sort_by_sent_date must be a boolean") ++ self._state_lock = RLock() + self._group_by_subject = group_by_subject + self._sort_by_sent_date = sort_by_sent_date + self._max_snapshot_records = _validated_positive_limit( +@@ -857,7 +859,8 @@ class IncrementalThreadIndex: + + def __len__(self) -> int: + """Return the number of indexed caller message keys.""" +- return len(self._records) ++ with self._state_lock: ++ return len(self._records) + + def _materialize_forest(self) -> None: + """Build and cache the complete canonical forest only when requested.""" +@@ -881,30 +884,37 @@ class IncrementalThreadIndex: + @property + def version(self) -> int: + """Return the optimistic mailbox-state version.""" +- return self._version ++ with self._state_lock: ++ return self._version + + @property + def message_keys(self) -> tuple[str, ...]: + """Return current caller keys in stable batch input order.""" +- return _ordered_keys(self._records, self._positions) ++ with self._state_lock: ++ return _ordered_keys(self._records, self._positions) + + @property + def roots(self) -> tuple[Container, ...]: + """Return defensive transport-neutral copies of current thread roots.""" +- self._materialize_forest() +- assert self._roots is not None +- return _public_forest_copy(self._roots) ++ with self._state_lock: ++ self._materialize_forest() ++ assert self._roots is not None ++ return _public_forest_copy(self._roots) + + @property + def projections(self) -> tuple[ThreadProjection, ...]: + """Return deterministic caller-key projections for current roots.""" +- self._materialize_forest() +- assert self._projections is not None +- return self._projections ++ with self._state_lock: ++ self._materialize_forest() ++ assert self._projections is not None ++ return self._projections + + def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: + """Atomically apply one optimistic mailbox change set. + ++ Concurrent callers are serialized so two transactions targeting the same ++ version cannot both commit. ++ + Raises: + VersionConflictError: ``expected_version`` is stale. + IncrementalThreadError: Key ownership, metadata, or graph processing +@@ -912,6 +922,11 @@ class IncrementalThreadIndex: + ExternalIdentityError: Reported EMAILID/THREADID metadata changes or + conflicts across equal EMAILID values. + """ ++ with self._state_lock: ++ return self._apply_locked(change_set) ++ ++ def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: ++ """Apply one change while the index state lock is held.""" + if not isinstance(change_set, MailboxChangeSet): + raise IncrementalThreadError("change_set must be a MailboxChangeSet") + if change_set.expected_version != self._version: +@@ -1135,6 +1150,11 @@ class IncrementalThreadIndex: + + def snapshot(self) -> dict[str, object]: + """Return deterministic versioned JSON-safe state without payload objects.""" ++ with self._state_lock: ++ return self._snapshot_locked() ++ ++ def _snapshot_locked(self) -> dict[str, object]: ++ """Build one snapshot while the index state lock is held.""" + if len(self._records) > self._max_snapshot_records: + raise IncrementalThreadError( + "snapshot exceeds max_snapshot_records" +diff --git a/tests/test_incremental_concurrency.py b/tests/test_incremental_concurrency.py +new file mode 100644 +index 0000000..395446d +--- /dev/null ++++ b/tests/test_incremental_concurrency.py +@@ -0,0 +1,90 @@ ++"""Concurrency contracts for the incremental mailbox index.""" ++ ++from __future__ import annotations ++ ++import threading ++ ++import threadweave.incremental as incremental_module ++from threadweave import ( ++ IncrementalThreadIndex, ++ IndexedMessage, ++ MailboxChangeSet, ++ Message, ++ VersionConflictError, ++) ++ ++ ++def _record(message_key: str) -> IndexedMessage: ++ """Return one independent indexed message for a concurrent update.""" ++ return IndexedMessage(message_key, Message(message_id=message_key)) ++ ++ ++def test_concurrent_writers_are_serialized_by_optimistic_version(monkeypatch): ++ """Two writers targeting one version yield one commit and one conflict.""" ++ index = IncrementalThreadIndex() ++ real_copy = incremental_module._copied_indexed_message ++ first_entered = threading.Event() ++ second_entered = threading.Event() ++ release_first = threading.Event() ++ activity_lock = threading.Lock() ++ active_calls = 0 ++ maximum_active_calls = 0 ++ ++ def controlled_copy(record: IndexedMessage) -> IndexedMessage: ++ """Expose whether two transactions copy records at the same time.""" ++ nonlocal active_calls, maximum_active_calls ++ with activity_lock: ++ active_calls += 1 ++ maximum_active_calls = max(maximum_active_calls, active_calls) ++ is_first_call = active_calls == 1 ++ if is_first_call: ++ first_entered.set() ++ else: ++ second_entered.set() ++ release_first.set() ++ if is_first_call: ++ release_first.wait(timeout=1) ++ try: ++ return real_copy(record) ++ finally: ++ with activity_lock: ++ active_calls -= 1 ++ ++ monkeypatch.setattr( ++ incremental_module, ++ "_copied_indexed_message", ++ controlled_copy, ++ ) ++ ++ outcomes: list[tuple[str, object]] = [] ++ ++ def apply_record(message_key: str) -> None: ++ """Apply one expected-version-zero transaction and capture its outcome.""" ++ try: ++ delta = index.apply( ++ MailboxChangeSet( ++ expected_version=0, ++ additions=(_record(message_key),), ++ ) ++ ) ++ except VersionConflictError as error: ++ outcomes.append(("conflict", error)) ++ else: ++ outcomes.append(("committed", delta)) ++ ++ first = threading.Thread(target=apply_record, args=("first",)) ++ second = threading.Thread(target=apply_record, args=("second",)) ++ first.start() ++ assert first_entered.wait(timeout=1) ++ second.start() ++ if not second_entered.wait(timeout=0.1): ++ release_first.set() ++ first.join(timeout=2) ++ second.join(timeout=2) ++ ++ assert not first.is_alive() ++ assert not second.is_alive() ++ assert maximum_active_calls == 1 ++ assert sorted(kind for kind, _ in outcomes) == ["committed", "conflict"] ++ assert index.version == 1 ++ assert len(index.message_keys) == 1 From c6b52a38c9daa9af1d6e1d7ae332d68ac3794b03 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:49:03 +0900 Subject: [PATCH 43/97] ci: verify and apply incremental concurrency fix --- .../workflows/apply-pr20-concurrency-fix.yml | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 .github/workflows/apply-pr20-concurrency-fix.yml diff --git a/.github/workflows/apply-pr20-concurrency-fix.yml b/.github/workflows/apply-pr20-concurrency-fix.yml new file mode 100644 index 0000000..ad0f57c --- /dev/null +++ b/.github/workflows/apply-pr20-concurrency-fix.yml @@ -0,0 +1,103 @@ +name: Apply PR 20 concurrency fix + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/apply-pr20-concurrency-fix.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-concurrency-fix + cancel-in-progress: false + +jobs: + apply-and-verify: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Verify and apply the reviewed patch + shell: bash + run: | + set -euo pipefail + printf '%s %s\n' \ + '62798654e77832717e98f1153306afce24df107c4cadb993087b0897bd8b9973' \ + 'tools/pr20-concurrency-fix.patch' | sha256sum -c - + git apply --check tools/pr20-concurrency-fix.patch + git apply tools/pr20-concurrency-fix.patch + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Verify the patched repository state + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + python benchmarks/incremental_mailbox.py \ + --messages 10000 \ + --thread-size 10 \ + --output "$RUNNER_TEMP/incremental-concurrency-benchmark.json" + + - name: Commit the verified product update + shell: bash + run: | + set -euo pipefail + rm tools/pr20-concurrency-fix.patch + rmdir tools 2>/dev/null || true + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add -A + if git diff --cached --name-only | grep -q '^\.github/workflows/'; then + echo 'Refusing to push workflow changes with the repository token.' >&2 + exit 1 + fi + git diff --cached --check + git commit -m 'fix: serialize incremental index state access' + git push origin HEAD:feature/incremental-thread-index From bd9b6200fb87d393fc157ee301b0187a36b6efed Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 20:56:39 +0900 Subject: [PATCH 44/97] ci: trigger verified concurrency fix from PR synchronization --- .github/workflows/apply-pr20-concurrency-fix.yml | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/.github/workflows/apply-pr20-concurrency-fix.yml b/.github/workflows/apply-pr20-concurrency-fix.yml index ad0f57c..8dda17d 100644 --- a/.github/workflows/apply-pr20-concurrency-fix.yml +++ b/.github/workflows/apply-pr20-concurrency-fix.yml @@ -1,9 +1,8 @@ name: Apply PR 20 concurrency fix on: - push: - branches: - - feature/incremental-thread-index + pull_request: + types: [synchronize] paths: - .github/workflows/apply-pr20-concurrency-fix.yml @@ -16,7 +15,10 @@ concurrency: jobs: apply-and-verify: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' + if: >- + github.event.pull_request.number == 20 && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.head.ref == 'feature/incremental-thread-index' runs-on: ubuntu-24.04 timeout-minutes: 30 steps: From 15c2c9c6a5856456b801a44f60f3b6573285417b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:10:40 +0900 Subject: [PATCH 45/97] test: update verified incremental concurrency fix --- tools/pr20-concurrency-fix.patch | 149 ++++++++++++++++++++++++++----- 1 file changed, 126 insertions(+), 23 deletions(-) diff --git a/tools/pr20-concurrency-fix.patch b/tools/pr20-concurrency-fix.patch index fabe3e2..09311da 100644 --- a/tools/pr20-concurrency-fix.patch +++ b/tools/pr20-concurrency-fix.patch @@ -26,46 +26,64 @@ index cfce2c1..e7c1bad 100644 +consume `ThreadDelta`, snapshots, or IMAP presentation output through public +interfaces. diff --git a/CHANGELOG.md b/CHANGELOG.md -index 07cd9ad..4a4197f 100644 +index 07cd9ad..a32d0b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased -+- Serialize in-process incremental readers and writers with a reentrant state lock, -+ so concurrent transactions targeting the same optimistic version produce one -+ commit and one explicit `VersionConflictError` instead of a silent lost update. ++- Serialize every read and write on one `IncrementalThreadIndex` with a ++ process-local reentrant lock so same-version concurrent writers yield one ++ commit and one explicit conflict, while readers observe only committed state. - Keep implicit RFC 5256 sent-date tie-break positions internal to the incremental engine instead of exposing invented IMAP sequence numbers on public roots. - Return defensive structural copies from `IncrementalThreadIndex.roots` so caller diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md -index 6aa08c2..4462f05 100644 +index 6aa08c2..3889a62 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md -@@ -102,6 +102,13 @@ assert updated.version == 2 +@@ -102,6 +102,31 @@ assert updated.version == 2 with the old version therefore fails explicitly instead of duplicating records or edges. An empty change set is idempotent and does not advance the version. -+One `IncrementalThreadIndex` serializes public reads, snapshots, and mutations with -+an in-process reentrant lock. Two threads that submit the same expected version -+therefore cannot both commit: the first successful transaction advances the -+version and the second receives `VersionConflictError`. This lock is process-local; -+services that share mailbox state across processes or hosts must still serialize -+durable writes and persist the resulting version in their own storage layer. ++### Concurrent access ++ ++All public reads and writes on one index acquire the same process-local reentrant ++lock. A transaction owns that lock from optimistic-version validation through the ++single state publication point. A second writer using the same version therefore ++waits, observes the committed version, and raises `VersionConflictError`; a reader ++waits and sees either the complete old state or the complete new state. ++ ++```mermaid ++flowchart LR ++ W1[Writer A: expected version n] --> L[Per-index reentrant lock] ++ W2[Writer B: expected version n] --> L ++ R[Reader: roots, projections, snapshot] --> L ++ L --> V{Validate current version} ++ V -->|Writer A| C[Compute isolated transaction] ++ C --> P[Publish one committed state] ++ P --> O[Release lock] ++ V -->|Writer B after A| X[Explicit version conflict] ++ V -->|Reader| S[Return one committed snapshot] ++``` ++ ++The lock coordinates threads inside one Python process only. A host such as naruon ++must still serialize durable writes across workers or replicas and persist the ++optimistic version beside its mailbox state. + ## Affected-component recomputation Each record contributes connectivity tokens for: diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py -index b7fed9f..7fec91e 100644 +index b7fed9f..4be1c57 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -13,6 +13,7 @@ import json from collections.abc import Iterable, Mapping, Sequence from dataclasses import dataclass from datetime import datetime -+from threading import RLock ++from _thread import RLock from typing import Literal from threadweave.collation import unicode_casemap_key @@ -87,7 +105,7 @@ index b7fed9f..7fec91e 100644 def _materialize_forest(self) -> None: """Build and cache the complete canonical forest only when requested.""" -@@ -881,30 +884,37 @@ class IncrementalThreadIndex: +@@ -881,30 +884,38 @@ class IncrementalThreadIndex: @property def version(self) -> int: """Return the optimistic mailbox-state version.""" @@ -127,13 +145,14 @@ index b7fed9f..7fec91e 100644 def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: """Atomically apply one optimistic mailbox change set. -+ Concurrent callers are serialized so two transactions targeting the same -+ version cannot both commit. ++ Concurrent callers are serialized. A second writer using the same ++ ``expected_version`` observes the first commit and raises ++ :class:`VersionConflictError` instead of interleaving copied state. + Raises: VersionConflictError: ``expected_version`` is stale. IncrementalThreadError: Key ownership, metadata, or graph processing -@@ -912,6 +922,11 @@ class IncrementalThreadIndex: +@@ -912,6 +923,11 @@ class IncrementalThreadIndex: ExternalIdentityError: Reported EMAILID/THREADID metadata changes or conflicts across equal EMAILID values. """ @@ -141,11 +160,11 @@ index b7fed9f..7fec91e 100644 + return self._apply_locked(change_set) + + def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: -+ """Apply one change while the index state lock is held.""" ++ """Apply one change while ``_state_lock`` protects every state field.""" if not isinstance(change_set, MailboxChangeSet): raise IncrementalThreadError("change_set must be a MailboxChangeSet") if change_set.expected_version != self._version: -@@ -1135,6 +1150,11 @@ class IncrementalThreadIndex: +@@ -1135,6 +1151,11 @@ class IncrementalThreadIndex: def snapshot(self) -> dict[str, object]: """Return deterministic versioned JSON-safe state without payload objects.""" @@ -153,21 +172,25 @@ index b7fed9f..7fec91e 100644 + return self._snapshot_locked() + + def _snapshot_locked(self) -> dict[str, object]: -+ """Build one snapshot while the index state lock is held.""" ++ """Build one snapshot while ``_state_lock`` protects current state.""" if len(self._records) > self._max_snapshot_records: raise IncrementalThreadError( "snapshot exceeds max_snapshot_records" diff --git a/tests/test_incremental_concurrency.py b/tests/test_incremental_concurrency.py new file mode 100644 -index 0000000..395446d +index 0000000..922a84a --- /dev/null +++ b/tests/test_incremental_concurrency.py -@@ -0,0 +1,90 @@ +@@ -0,0 +1,174 @@ +"""Concurrency contracts for the incremental mailbox index.""" + +from __future__ import annotations + ++import os ++import subprocess ++import sys +import threading ++from pathlib import Path + +import threadweave.incremental as incremental_module +from threadweave import ( @@ -253,3 +276,83 @@ index 0000000..395446d + assert sorted(kind for kind, _ in outcomes) == ["committed", "conflict"] + assert index.version == 1 + assert len(index.message_keys) == 1 ++ ++ ++def test_readers_observe_only_committed_versions(monkeypatch): ++ """A reader blocks behind a writer and never observes partial transaction state.""" ++ index = IncrementalThreadIndex() ++ real_copy = incremental_module._copied_indexed_message ++ writer_entered = threading.Event() ++ release_writer = threading.Event() ++ reader_started = threading.Event() ++ reader_finished = threading.Event() ++ observed: list[dict[str, object]] = [] ++ ++ def controlled_copy(record: IndexedMessage) -> IndexedMessage: ++ """Pause one writer after it owns the state lock but before commit.""" ++ writer_entered.set() ++ assert release_writer.wait(timeout=2) ++ return real_copy(record) ++ ++ monkeypatch.setattr( ++ incremental_module, ++ "_copied_indexed_message", ++ controlled_copy, ++ ) ++ ++ def write() -> None: ++ """Apply one transaction while the test controls its commit point.""" ++ index.apply( ++ MailboxChangeSet( ++ expected_version=0, ++ additions=(_record("message"),), ++ ) ++ ) ++ ++ def read() -> None: ++ """Capture one snapshot after acquiring the same state lock.""" ++ reader_started.set() ++ observed.append(index.snapshot()) ++ reader_finished.set() ++ ++ writer = threading.Thread(target=write) ++ reader = threading.Thread(target=read) ++ writer.start() ++ assert writer_entered.wait(timeout=1) ++ reader.start() ++ assert reader_started.wait(timeout=1) ++ assert not reader_finished.wait(timeout=0.05) ++ ++ release_writer.set() ++ writer.join(timeout=2) ++ reader.join(timeout=2) ++ ++ assert not writer.is_alive() ++ assert not reader.is_alive() ++ assert reader_finished.is_set() ++ assert observed[0]["version"] == 1 ++ records = observed[0]["records"] ++ assert isinstance(records, list) ++ assert [record["message_key"] for record in records] == ["message"] ++ ++ ++def test_package_import_survives_a_top_level_threading_name_collision(): ++ """The built-in lock remains importable beside ``threadweave/threading.py``.""" ++ repository_root = Path(__file__).resolve().parents[1] ++ import_script = ( ++ "import sys; " ++ "sys.path.insert(0, 'src/threadweave'); " ++ "sys.path.insert(1, 'src'); " ++ "import subject" ++ ) ++ ++ result = subprocess.run( ++ [sys.executable, "-S", "-c", import_script], ++ cwd=repository_root, ++ env=os.environ.copy(), ++ capture_output=True, ++ text=True, ++ check=False, ++ ) ++ ++ assert result.returncode == 0, result.stderr From d8d9592b94c91335e5eee4d2af1db8f4af318a42 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:11:51 +0900 Subject: [PATCH 46/97] ci: verify the corrected concurrency patch --- .github/workflows/apply-pr20-concurrency-fix.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/apply-pr20-concurrency-fix.yml b/.github/workflows/apply-pr20-concurrency-fix.yml index 8dda17d..9e04ab4 100644 --- a/.github/workflows/apply-pr20-concurrency-fix.yml +++ b/.github/workflows/apply-pr20-concurrency-fix.yml @@ -40,7 +40,7 @@ jobs: run: | set -euo pipefail printf '%s %s\n' \ - '62798654e77832717e98f1153306afce24df107c4cadb993087b0897bd8b9973' \ + 'b4938487f6d8e37157ef6bbfe59f1b52b7db27ce782147053918068640fbbc8e' \ 'tools/pr20-concurrency-fix.patch' | sha256sum -c - git apply --check tools/pr20-concurrency-fix.patch git apply tools/pr20-concurrency-fix.patch From 54e44349698fae523614ef9a425a07ec871eb549 Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 5 Aug 2026 12:12:34 +0000 Subject: [PATCH 47/97] fix: serialize incremental index state access --- ARCHITECTURE.md | 12 +- CHANGELOG.md | 3 + docs/incremental-threading.md | 25 ++ src/threadweave/incremental.py | 39 ++- tests/test_incremental_concurrency.py | 174 +++++++++++++ tools/pr20-concurrency-fix.patch | 358 -------------------------- 6 files changed, 241 insertions(+), 370 deletions(-) create mode 100644 tests/test_incremental_concurrency.py delete mode 100644 tools/pr20-concurrency-fix.patch diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index cfce2c1..e7c1bad 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -43,6 +43,11 @@ is maintained in incremental code. ## State and mutation policy - `IndexedMessage.message_key` is caller-owned and immutable across revisions. +- Public reads, snapshots, and `apply` calls on one index are serialized by an + in-process reentrant lock. Two transactions targeting one version cannot both + commit; the later transaction observes the new version and fails explicitly. +- The process-local lock is not a distributed lock. Naruon or another host must + serialize durable writes across processes and persist the optimistic version. - `apply` validates and computes on isolated transaction state, then commits once. - Reverse connectivity buckets use copy-on-write mutation. - Complete root/projection views are lazy caches invalidated by a successful change. @@ -70,6 +75,7 @@ existing messages. ## Integration policy Naruon and other services should own persistence, tenancy, authentication, -mailbox synchronization, and external stable-ID policy. They pass immutable caller -keys and `Message` metadata into ThreadWeave and consume `ThreadDelta`, snapshots, -or IMAP presentation output through public interfaces. +mailbox synchronization, distributed write serialization, and external stable-ID +policy. They pass immutable caller keys and `Message` metadata into ThreadWeave and +consume `ThreadDelta`, snapshots, or IMAP presentation output through public +interfaces. diff --git a/CHANGELOG.md b/CHANGELOG.md index 07cd9ad..a32d0b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Serialize every read and write on one `IncrementalThreadIndex` with a + process-local reentrant lock so same-version concurrent writers yield one + commit and one explicit conflict, while readers observe only committed state. - Keep implicit RFC 5256 sent-date tie-break positions internal to the incremental engine instead of exposing invented IMAP sequence numbers on public roots. - Return defensive structural copies from `IncrementalThreadIndex.roots` so caller diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index 6aa08c2..3889a62 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -102,6 +102,31 @@ assert updated.version == 2 with the old version therefore fails explicitly instead of duplicating records or edges. An empty change set is idempotent and does not advance the version. +### Concurrent access + +All public reads and writes on one index acquire the same process-local reentrant +lock. A transaction owns that lock from optimistic-version validation through the +single state publication point. A second writer using the same version therefore +waits, observes the committed version, and raises `VersionConflictError`; a reader +waits and sees either the complete old state or the complete new state. + +```mermaid +flowchart LR + W1[Writer A: expected version n] --> L[Per-index reentrant lock] + W2[Writer B: expected version n] --> L + R[Reader: roots, projections, snapshot] --> L + L --> V{Validate current version} + V -->|Writer A| C[Compute isolated transaction] + C --> P[Publish one committed state] + P --> O[Release lock] + V -->|Writer B after A| X[Explicit version conflict] + V -->|Reader| S[Return one committed snapshot] +``` + +The lock coordinates threads inside one Python process only. A host such as naruon +must still serialize durable writes across workers or replicas and persist the +optimistic version beside its mailbox state. + ## Affected-component recomputation Each record contributes connectivity tokens for: diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index b7fed9f..4be1c57 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -13,6 +13,7 @@ from collections.abc import Iterable, Mapping, Sequence from dataclasses import dataclass from datetime import datetime +from _thread import RLock from typing import Literal from threadweave.collation import unicode_casemap_key @@ -834,6 +835,7 @@ def __init__( raise IncrementalThreadError("group_by_subject must be a boolean") if not isinstance(sort_by_sent_date, bool): raise IncrementalThreadError("sort_by_sent_date must be a boolean") + self._state_lock = RLock() self._group_by_subject = group_by_subject self._sort_by_sent_date = sort_by_sent_date self._max_snapshot_records = _validated_positive_limit( @@ -857,7 +859,8 @@ def __init__( def __len__(self) -> int: """Return the number of indexed caller message keys.""" - return len(self._records) + with self._state_lock: + return len(self._records) def _materialize_forest(self) -> None: """Build and cache the complete canonical forest only when requested.""" @@ -881,30 +884,38 @@ def _materialize_forest(self) -> None: @property def version(self) -> int: """Return the optimistic mailbox-state version.""" - return self._version + with self._state_lock: + return self._version @property def message_keys(self) -> tuple[str, ...]: """Return current caller keys in stable batch input order.""" - return _ordered_keys(self._records, self._positions) + with self._state_lock: + return _ordered_keys(self._records, self._positions) @property def roots(self) -> tuple[Container, ...]: """Return defensive transport-neutral copies of current thread roots.""" - self._materialize_forest() - assert self._roots is not None - return _public_forest_copy(self._roots) + with self._state_lock: + self._materialize_forest() + assert self._roots is not None + return _public_forest_copy(self._roots) @property def projections(self) -> tuple[ThreadProjection, ...]: """Return deterministic caller-key projections for current roots.""" - self._materialize_forest() - assert self._projections is not None - return self._projections + with self._state_lock: + self._materialize_forest() + assert self._projections is not None + return self._projections def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: """Atomically apply one optimistic mailbox change set. + Concurrent callers are serialized. A second writer using the same + ``expected_version`` observes the first commit and raises + :class:`VersionConflictError` instead of interleaving copied state. + Raises: VersionConflictError: ``expected_version`` is stale. IncrementalThreadError: Key ownership, metadata, or graph processing @@ -912,6 +923,11 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: ExternalIdentityError: Reported EMAILID/THREADID metadata changes or conflicts across equal EMAILID values. """ + with self._state_lock: + return self._apply_locked(change_set) + + def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: + """Apply one change while ``_state_lock`` protects every state field.""" if not isinstance(change_set, MailboxChangeSet): raise IncrementalThreadError("change_set must be a MailboxChangeSet") if change_set.expected_version != self._version: @@ -1135,6 +1151,11 @@ def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: def snapshot(self) -> dict[str, object]: """Return deterministic versioned JSON-safe state without payload objects.""" + with self._state_lock: + return self._snapshot_locked() + + def _snapshot_locked(self) -> dict[str, object]: + """Build one snapshot while ``_state_lock`` protects current state.""" if len(self._records) > self._max_snapshot_records: raise IncrementalThreadError( "snapshot exceeds max_snapshot_records" diff --git a/tests/test_incremental_concurrency.py b/tests/test_incremental_concurrency.py new file mode 100644 index 0000000..922a84a --- /dev/null +++ b/tests/test_incremental_concurrency.py @@ -0,0 +1,174 @@ +"""Concurrency contracts for the incremental mailbox index.""" + +from __future__ import annotations + +import os +import subprocess +import sys +import threading +from pathlib import Path + +import threadweave.incremental as incremental_module +from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + VersionConflictError, +) + + +def _record(message_key: str) -> IndexedMessage: + """Return one independent indexed message for a concurrent update.""" + return IndexedMessage(message_key, Message(message_id=message_key)) + + +def test_concurrent_writers_are_serialized_by_optimistic_version(monkeypatch): + """Two writers targeting one version yield one commit and one conflict.""" + index = IncrementalThreadIndex() + real_copy = incremental_module._copied_indexed_message + first_entered = threading.Event() + second_entered = threading.Event() + release_first = threading.Event() + activity_lock = threading.Lock() + active_calls = 0 + maximum_active_calls = 0 + + def controlled_copy(record: IndexedMessage) -> IndexedMessage: + """Expose whether two transactions copy records at the same time.""" + nonlocal active_calls, maximum_active_calls + with activity_lock: + active_calls += 1 + maximum_active_calls = max(maximum_active_calls, active_calls) + is_first_call = active_calls == 1 + if is_first_call: + first_entered.set() + else: + second_entered.set() + release_first.set() + if is_first_call: + release_first.wait(timeout=1) + try: + return real_copy(record) + finally: + with activity_lock: + active_calls -= 1 + + monkeypatch.setattr( + incremental_module, + "_copied_indexed_message", + controlled_copy, + ) + + outcomes: list[tuple[str, object]] = [] + + def apply_record(message_key: str) -> None: + """Apply one expected-version-zero transaction and capture its outcome.""" + try: + delta = index.apply( + MailboxChangeSet( + expected_version=0, + additions=(_record(message_key),), + ) + ) + except VersionConflictError as error: + outcomes.append(("conflict", error)) + else: + outcomes.append(("committed", delta)) + + first = threading.Thread(target=apply_record, args=("first",)) + second = threading.Thread(target=apply_record, args=("second",)) + first.start() + assert first_entered.wait(timeout=1) + second.start() + if not second_entered.wait(timeout=0.1): + release_first.set() + first.join(timeout=2) + second.join(timeout=2) + + assert not first.is_alive() + assert not second.is_alive() + assert maximum_active_calls == 1 + assert sorted(kind for kind, _ in outcomes) == ["committed", "conflict"] + assert index.version == 1 + assert len(index.message_keys) == 1 + + +def test_readers_observe_only_committed_versions(monkeypatch): + """A reader blocks behind a writer and never observes partial transaction state.""" + index = IncrementalThreadIndex() + real_copy = incremental_module._copied_indexed_message + writer_entered = threading.Event() + release_writer = threading.Event() + reader_started = threading.Event() + reader_finished = threading.Event() + observed: list[dict[str, object]] = [] + + def controlled_copy(record: IndexedMessage) -> IndexedMessage: + """Pause one writer after it owns the state lock but before commit.""" + writer_entered.set() + assert release_writer.wait(timeout=2) + return real_copy(record) + + monkeypatch.setattr( + incremental_module, + "_copied_indexed_message", + controlled_copy, + ) + + def write() -> None: + """Apply one transaction while the test controls its commit point.""" + index.apply( + MailboxChangeSet( + expected_version=0, + additions=(_record("message"),), + ) + ) + + def read() -> None: + """Capture one snapshot after acquiring the same state lock.""" + reader_started.set() + observed.append(index.snapshot()) + reader_finished.set() + + writer = threading.Thread(target=write) + reader = threading.Thread(target=read) + writer.start() + assert writer_entered.wait(timeout=1) + reader.start() + assert reader_started.wait(timeout=1) + assert not reader_finished.wait(timeout=0.05) + + release_writer.set() + writer.join(timeout=2) + reader.join(timeout=2) + + assert not writer.is_alive() + assert not reader.is_alive() + assert reader_finished.is_set() + assert observed[0]["version"] == 1 + records = observed[0]["records"] + assert isinstance(records, list) + assert [record["message_key"] for record in records] == ["message"] + + +def test_package_import_survives_a_top_level_threading_name_collision(): + """The built-in lock remains importable beside ``threadweave/threading.py``.""" + repository_root = Path(__file__).resolve().parents[1] + import_script = ( + "import sys; " + "sys.path.insert(0, 'src/threadweave'); " + "sys.path.insert(1, 'src'); " + "import subject" + ) + + result = subprocess.run( + [sys.executable, "-S", "-c", import_script], + cwd=repository_root, + env=os.environ.copy(), + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode == 0, result.stderr diff --git a/tools/pr20-concurrency-fix.patch b/tools/pr20-concurrency-fix.patch deleted file mode 100644 index 09311da..0000000 --- a/tools/pr20-concurrency-fix.patch +++ /dev/null @@ -1,358 +0,0 @@ -diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md -index cfce2c1..e7c1bad 100644 ---- a/ARCHITECTURE.md -+++ b/ARCHITECTURE.md -@@ -43,6 +43,11 @@ is maintained in incremental code. - ## State and mutation policy - - - `IndexedMessage.message_key` is caller-owned and immutable across revisions. -+- Public reads, snapshots, and `apply` calls on one index are serialized by an -+ in-process reentrant lock. Two transactions targeting one version cannot both -+ commit; the later transaction observes the new version and fails explicitly. -+- The process-local lock is not a distributed lock. Naruon or another host must -+ serialize durable writes across processes and persist the optimistic version. - - `apply` validates and computes on isolated transaction state, then commits once. - - Reverse connectivity buckets use copy-on-write mutation. - - Complete root/projection views are lazy caches invalidated by a successful change. -@@ -70,6 +75,7 @@ existing messages. - ## Integration policy - - Naruon and other services should own persistence, tenancy, authentication, --mailbox synchronization, and external stable-ID policy. They pass immutable caller --keys and `Message` metadata into ThreadWeave and consume `ThreadDelta`, snapshots, --or IMAP presentation output through public interfaces. -+mailbox synchronization, distributed write serialization, and external stable-ID -+policy. They pass immutable caller keys and `Message` metadata into ThreadWeave and -+consume `ThreadDelta`, snapshots, or IMAP presentation output through public -+interfaces. -diff --git a/CHANGELOG.md b/CHANGELOG.md -index 07cd9ad..a32d0b0 100644 ---- a/CHANGELOG.md -+++ b/CHANGELOG.md -@@ -6,6 +6,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - - ## Unreleased - -+- Serialize every read and write on one `IncrementalThreadIndex` with a -+ process-local reentrant lock so same-version concurrent writers yield one -+ commit and one explicit conflict, while readers observe only committed state. - - Keep implicit RFC 5256 sent-date tie-break positions internal to the incremental - engine instead of exposing invented IMAP sequence numbers on public roots. - - Return defensive structural copies from `IncrementalThreadIndex.roots` so caller -diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md -index 6aa08c2..3889a62 100644 ---- a/docs/incremental-threading.md -+++ b/docs/incremental-threading.md -@@ -102,6 +102,31 @@ assert updated.version == 2 - with the old version therefore fails explicitly instead of duplicating records or - edges. An empty change set is idempotent and does not advance the version. - -+### Concurrent access -+ -+All public reads and writes on one index acquire the same process-local reentrant -+lock. A transaction owns that lock from optimistic-version validation through the -+single state publication point. A second writer using the same version therefore -+waits, observes the committed version, and raises `VersionConflictError`; a reader -+waits and sees either the complete old state or the complete new state. -+ -+```mermaid -+flowchart LR -+ W1[Writer A: expected version n] --> L[Per-index reentrant lock] -+ W2[Writer B: expected version n] --> L -+ R[Reader: roots, projections, snapshot] --> L -+ L --> V{Validate current version} -+ V -->|Writer A| C[Compute isolated transaction] -+ C --> P[Publish one committed state] -+ P --> O[Release lock] -+ V -->|Writer B after A| X[Explicit version conflict] -+ V -->|Reader| S[Return one committed snapshot] -+``` -+ -+The lock coordinates threads inside one Python process only. A host such as naruon -+must still serialize durable writes across workers or replicas and persist the -+optimistic version beside its mailbox state. -+ - ## Affected-component recomputation - - Each record contributes connectivity tokens for: -diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py -index b7fed9f..4be1c57 100644 ---- a/src/threadweave/incremental.py -+++ b/src/threadweave/incremental.py -@@ -13,6 +13,7 @@ import json - from collections.abc import Iterable, Mapping, Sequence - from dataclasses import dataclass - from datetime import datetime -+from _thread import RLock - from typing import Literal - - from threadweave.collation import unicode_casemap_key -@@ -834,6 +835,7 @@ class IncrementalThreadIndex: - raise IncrementalThreadError("group_by_subject must be a boolean") - if not isinstance(sort_by_sent_date, bool): - raise IncrementalThreadError("sort_by_sent_date must be a boolean") -+ self._state_lock = RLock() - self._group_by_subject = group_by_subject - self._sort_by_sent_date = sort_by_sent_date - self._max_snapshot_records = _validated_positive_limit( -@@ -857,7 +859,8 @@ class IncrementalThreadIndex: - - def __len__(self) -> int: - """Return the number of indexed caller message keys.""" -- return len(self._records) -+ with self._state_lock: -+ return len(self._records) - - def _materialize_forest(self) -> None: - """Build and cache the complete canonical forest only when requested.""" -@@ -881,30 +884,38 @@ class IncrementalThreadIndex: - @property - def version(self) -> int: - """Return the optimistic mailbox-state version.""" -- return self._version -+ with self._state_lock: -+ return self._version - - @property - def message_keys(self) -> tuple[str, ...]: - """Return current caller keys in stable batch input order.""" -- return _ordered_keys(self._records, self._positions) -+ with self._state_lock: -+ return _ordered_keys(self._records, self._positions) - - @property - def roots(self) -> tuple[Container, ...]: - """Return defensive transport-neutral copies of current thread roots.""" -- self._materialize_forest() -- assert self._roots is not None -- return _public_forest_copy(self._roots) -+ with self._state_lock: -+ self._materialize_forest() -+ assert self._roots is not None -+ return _public_forest_copy(self._roots) - - @property - def projections(self) -> tuple[ThreadProjection, ...]: - """Return deterministic caller-key projections for current roots.""" -- self._materialize_forest() -- assert self._projections is not None -- return self._projections -+ with self._state_lock: -+ self._materialize_forest() -+ assert self._projections is not None -+ return self._projections - - def apply(self, change_set: MailboxChangeSet) -> ThreadDelta: - """Atomically apply one optimistic mailbox change set. - -+ Concurrent callers are serialized. A second writer using the same -+ ``expected_version`` observes the first commit and raises -+ :class:`VersionConflictError` instead of interleaving copied state. -+ - Raises: - VersionConflictError: ``expected_version`` is stale. - IncrementalThreadError: Key ownership, metadata, or graph processing -@@ -912,6 +923,11 @@ class IncrementalThreadIndex: - ExternalIdentityError: Reported EMAILID/THREADID metadata changes or - conflicts across equal EMAILID values. - """ -+ with self._state_lock: -+ return self._apply_locked(change_set) -+ -+ def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: -+ """Apply one change while ``_state_lock`` protects every state field.""" - if not isinstance(change_set, MailboxChangeSet): - raise IncrementalThreadError("change_set must be a MailboxChangeSet") - if change_set.expected_version != self._version: -@@ -1135,6 +1151,11 @@ class IncrementalThreadIndex: - - def snapshot(self) -> dict[str, object]: - """Return deterministic versioned JSON-safe state without payload objects.""" -+ with self._state_lock: -+ return self._snapshot_locked() -+ -+ def _snapshot_locked(self) -> dict[str, object]: -+ """Build one snapshot while ``_state_lock`` protects current state.""" - if len(self._records) > self._max_snapshot_records: - raise IncrementalThreadError( - "snapshot exceeds max_snapshot_records" -diff --git a/tests/test_incremental_concurrency.py b/tests/test_incremental_concurrency.py -new file mode 100644 -index 0000000..922a84a ---- /dev/null -+++ b/tests/test_incremental_concurrency.py -@@ -0,0 +1,174 @@ -+"""Concurrency contracts for the incremental mailbox index.""" -+ -+from __future__ import annotations -+ -+import os -+import subprocess -+import sys -+import threading -+from pathlib import Path -+ -+import threadweave.incremental as incremental_module -+from threadweave import ( -+ IncrementalThreadIndex, -+ IndexedMessage, -+ MailboxChangeSet, -+ Message, -+ VersionConflictError, -+) -+ -+ -+def _record(message_key: str) -> IndexedMessage: -+ """Return one independent indexed message for a concurrent update.""" -+ return IndexedMessage(message_key, Message(message_id=message_key)) -+ -+ -+def test_concurrent_writers_are_serialized_by_optimistic_version(monkeypatch): -+ """Two writers targeting one version yield one commit and one conflict.""" -+ index = IncrementalThreadIndex() -+ real_copy = incremental_module._copied_indexed_message -+ first_entered = threading.Event() -+ second_entered = threading.Event() -+ release_first = threading.Event() -+ activity_lock = threading.Lock() -+ active_calls = 0 -+ maximum_active_calls = 0 -+ -+ def controlled_copy(record: IndexedMessage) -> IndexedMessage: -+ """Expose whether two transactions copy records at the same time.""" -+ nonlocal active_calls, maximum_active_calls -+ with activity_lock: -+ active_calls += 1 -+ maximum_active_calls = max(maximum_active_calls, active_calls) -+ is_first_call = active_calls == 1 -+ if is_first_call: -+ first_entered.set() -+ else: -+ second_entered.set() -+ release_first.set() -+ if is_first_call: -+ release_first.wait(timeout=1) -+ try: -+ return real_copy(record) -+ finally: -+ with activity_lock: -+ active_calls -= 1 -+ -+ monkeypatch.setattr( -+ incremental_module, -+ "_copied_indexed_message", -+ controlled_copy, -+ ) -+ -+ outcomes: list[tuple[str, object]] = [] -+ -+ def apply_record(message_key: str) -> None: -+ """Apply one expected-version-zero transaction and capture its outcome.""" -+ try: -+ delta = index.apply( -+ MailboxChangeSet( -+ expected_version=0, -+ additions=(_record(message_key),), -+ ) -+ ) -+ except VersionConflictError as error: -+ outcomes.append(("conflict", error)) -+ else: -+ outcomes.append(("committed", delta)) -+ -+ first = threading.Thread(target=apply_record, args=("first",)) -+ second = threading.Thread(target=apply_record, args=("second",)) -+ first.start() -+ assert first_entered.wait(timeout=1) -+ second.start() -+ if not second_entered.wait(timeout=0.1): -+ release_first.set() -+ first.join(timeout=2) -+ second.join(timeout=2) -+ -+ assert not first.is_alive() -+ assert not second.is_alive() -+ assert maximum_active_calls == 1 -+ assert sorted(kind for kind, _ in outcomes) == ["committed", "conflict"] -+ assert index.version == 1 -+ assert len(index.message_keys) == 1 -+ -+ -+def test_readers_observe_only_committed_versions(monkeypatch): -+ """A reader blocks behind a writer and never observes partial transaction state.""" -+ index = IncrementalThreadIndex() -+ real_copy = incremental_module._copied_indexed_message -+ writer_entered = threading.Event() -+ release_writer = threading.Event() -+ reader_started = threading.Event() -+ reader_finished = threading.Event() -+ observed: list[dict[str, object]] = [] -+ -+ def controlled_copy(record: IndexedMessage) -> IndexedMessage: -+ """Pause one writer after it owns the state lock but before commit.""" -+ writer_entered.set() -+ assert release_writer.wait(timeout=2) -+ return real_copy(record) -+ -+ monkeypatch.setattr( -+ incremental_module, -+ "_copied_indexed_message", -+ controlled_copy, -+ ) -+ -+ def write() -> None: -+ """Apply one transaction while the test controls its commit point.""" -+ index.apply( -+ MailboxChangeSet( -+ expected_version=0, -+ additions=(_record("message"),), -+ ) -+ ) -+ -+ def read() -> None: -+ """Capture one snapshot after acquiring the same state lock.""" -+ reader_started.set() -+ observed.append(index.snapshot()) -+ reader_finished.set() -+ -+ writer = threading.Thread(target=write) -+ reader = threading.Thread(target=read) -+ writer.start() -+ assert writer_entered.wait(timeout=1) -+ reader.start() -+ assert reader_started.wait(timeout=1) -+ assert not reader_finished.wait(timeout=0.05) -+ -+ release_writer.set() -+ writer.join(timeout=2) -+ reader.join(timeout=2) -+ -+ assert not writer.is_alive() -+ assert not reader.is_alive() -+ assert reader_finished.is_set() -+ assert observed[0]["version"] == 1 -+ records = observed[0]["records"] -+ assert isinstance(records, list) -+ assert [record["message_key"] for record in records] == ["message"] -+ -+ -+def test_package_import_survives_a_top_level_threading_name_collision(): -+ """The built-in lock remains importable beside ``threadweave/threading.py``.""" -+ repository_root = Path(__file__).resolve().parents[1] -+ import_script = ( -+ "import sys; " -+ "sys.path.insert(0, 'src/threadweave'); " -+ "sys.path.insert(1, 'src'); " -+ "import subject" -+ ) -+ -+ result = subprocess.run( -+ [sys.executable, "-S", "-c", import_script], -+ cwd=repository_root, -+ env=os.environ.copy(), -+ capture_output=True, -+ text=True, -+ check=False, -+ ) -+ -+ assert result.returncode == 0, result.stderr From 19ca8551fdb4d4a12f98edb9055b468d953b6edc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:14:12 +0900 Subject: [PATCH 48/97] ci: remove completed concurrency-fix bootstrap --- .../workflows/apply-pr20-concurrency-fix.yml | 105 ------------------ 1 file changed, 105 deletions(-) delete mode 100644 .github/workflows/apply-pr20-concurrency-fix.yml diff --git a/.github/workflows/apply-pr20-concurrency-fix.yml b/.github/workflows/apply-pr20-concurrency-fix.yml deleted file mode 100644 index 9e04ab4..0000000 --- a/.github/workflows/apply-pr20-concurrency-fix.yml +++ /dev/null @@ -1,105 +0,0 @@ -name: Apply PR 20 concurrency fix - -on: - pull_request: - types: [synchronize] - paths: - - .github/workflows/apply-pr20-concurrency-fix.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-concurrency-fix - cancel-in-progress: false - -jobs: - apply-and-verify: - if: >- - github.event.pull_request.number == 20 && - github.event.pull_request.head.repo.full_name == github.repository && - github.event.pull_request.head.ref == 'feature/incremental-thread-index' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Verify and apply the reviewed patch - shell: bash - run: | - set -euo pipefail - printf '%s %s\n' \ - 'b4938487f6d8e37157ef6bbfe59f1b52b7db27ce782147053918068640fbbc8e' \ - 'tools/pr20-concurrency-fix.patch' | sha256sum -c - - git apply --check tools/pr20-concurrency-fix.patch - git apply tools/pr20-concurrency-fix.patch - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Verify the patched repository state - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - python benchmarks/incremental_mailbox.py \ - --messages 10000 \ - --thread-size 10 \ - --output "$RUNNER_TEMP/incremental-concurrency-benchmark.json" - - - name: Commit the verified product update - shell: bash - run: | - set -euo pipefail - rm tools/pr20-concurrency-fix.patch - rmdir tools 2>/dev/null || true - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add -A - if git diff --cached --name-only | grep -q '^\.github/workflows/'; then - echo 'Refusing to push workflow changes with the repository token.' >&2 - exit 1 - fi - git diff --cached --check - git commit -m 'fix: serialize incremental index state access' - git push origin HEAD:feature/incremental-thread-index From b62ba55c2003c78a717d1b45276817f683174054 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:16:18 +0900 Subject: [PATCH 49/97] ci: verify incremental snapshot hardening test-first --- .../workflows/apply-snapshot-hardening.yml | 278 ++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 .github/workflows/apply-snapshot-hardening.yml diff --git a/.github/workflows/apply-snapshot-hardening.yml b/.github/workflows/apply-snapshot-hardening.yml new file mode 100644 index 0000000..e92f3ba --- /dev/null +++ b/.github/workflows/apply-snapshot-hardening.yml @@ -0,0 +1,278 @@ +name: Apply incremental snapshot hardening + +on: + pull_request: + types: [synchronize] + paths: + - .github/workflows/apply-snapshot-hardening.yml + +permissions: + contents: write + +concurrency: + group: apply-incremental-snapshot-hardening + cancel-in-progress: false + +jobs: + apply: + if: >- + github.event.pull_request.number == 20 && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.head.ref == 'feature/incremental-thread-index' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the exact pull-request head + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Add focused snapshot regressions and prove RED + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PY' + from pathlib import Path + + path = Path('tests/test_incremental_snapshot.py') + text = path.read_text(encoding='utf-8') + if 'import sys\n' not in text: + text = text.replace('import json\n', 'import json\nimport sys\n', 1) + anchor = ''' with pytest.raises(IncrementalThreadError, match="snapshot fields"): + IncrementalThreadIndex.restore(missing) + + + def test_restore_rejects_malformed_record_fields_and_duplicate_keys(): + ''' + addition = ''' with pytest.raises(IncrementalThreadError, match="snapshot fields"): + IncrementalThreadIndex.restore(missing) + + + @pytest.mark.parametrize("invalid_schema_version", [True, 1.0]) + def test_restore_requires_an_exact_integer_schema_version( + invalid_schema_version: object, + ): + """Boolean and floating-point lookalikes cannot select a snapshot schema.""" + snapshot = _index().snapshot() + snapshot["schema_version"] = invalid_schema_version + + with pytest.raises(IncrementalThreadError, match="schema_version"): + IncrementalThreadIndex.restore(snapshot) + + + def test_snapshot_reports_unencodable_unicode_as_a_domain_error(): + """Lone surrogates fail closed instead of leaking a codec exception.""" + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + IndexedMessage( + "surrogate_key", + Message(message_id="surrogate", subject="bad\\ud800subject"), + ), + ), + ) + ) + + with pytest.raises(IncrementalThreadError, match="JSON-safe"): + index.snapshot() + + + def test_restore_reports_excessive_json_nesting_as_a_domain_error(): + """Hostile nesting cannot escape the snapshot boundary as RecursionError.""" + nested: object = [] + for _ in range(sys.getrecursionlimit() * 20): + nested = [nested] + snapshot = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [], + "unexpected": nested, + } + + with pytest.raises(IncrementalThreadError, match="JSON-safe"): + IncrementalThreadIndex.restore(snapshot) + + + def test_restore_rejects_malformed_record_fields_and_duplicate_keys(): + ''' + if anchor not in text: + raise SystemExit('snapshot regression insertion anchor changed') + path.write_text(text.replace(anchor, addition, 1), encoding='utf-8') + PY + + set +e + python -m pytest -q tests/test_incremental_snapshot.py \ + -k 'exact_integer_schema_version or unencodable_unicode or excessive_json_nesting' \ + >"$RUNNER_TEMP/snapshot-red.log" 2>&1 + status=$? + set -e + cat "$RUNNER_TEMP/snapshot-red.log" + if [ "$status" -eq 0 ]; then + echo 'The snapshot regressions unexpectedly passed before implementation.' >&2 + exit 1 + fi + + - name: Implement the fail-closed snapshot boundary and documentation + shell: bash + run: | + set -euo pipefail + python - <<'PY' + from pathlib import Path + + source = Path('src/threadweave/incremental.py') + text = source.read_text(encoding='utf-8') + old = '''def _snapshot_json_bytes(value: object) -> bytes: + """Serialize a snapshot canonically or raise a bounded domain error.""" + try: + encoded = json.dumps( + value, + ensure_ascii=False, + allow_nan=False, + sort_keys=True, + separators=(",", ":"), + ) + except (TypeError, ValueError) as error: + raise IncrementalThreadError("snapshot must contain only JSON-safe values") from error + return encoded.encode("utf-8") + ''' + new = '''def _snapshot_json_bytes(value: object) -> bytes: + """Serialize a snapshot canonically or raise a bounded domain error.""" + try: + encoded = json.dumps( + value, + ensure_ascii=False, + allow_nan=False, + sort_keys=True, + separators=(",", ":"), + ) + return encoded.encode("utf-8") + except (RecursionError, TypeError, UnicodeError, ValueError) as error: + raise IncrementalThreadError( + "snapshot must contain only JSON-safe values" + ) from error + ''' + if text.count(old) != 1: + raise SystemExit('snapshot serializer anchor changed') + text = text.replace(old, new, 1) + old = ''' if snapshot["schema_version"] != _SNAPSHOT_SCHEMA_VERSION: + raise IncrementalThreadError("unsupported snapshot schema_version") + ''' + new = ''' schema_version = snapshot["schema_version"] + if ( + isinstance(schema_version, bool) + or not isinstance(schema_version, int) + or schema_version != _SNAPSHOT_SCHEMA_VERSION + ): + raise IncrementalThreadError("unsupported snapshot schema_version") + ''' + if text.count(old) != 1: + raise SystemExit('schema-version anchor changed') + source.write_text(text.replace(old, new, 1), encoding='utf-8') + + changelog = Path('CHANGELOG.md') + text = changelog.read_text(encoding='utf-8') + anchor = ( + '- Add deterministic 100,000-message incremental-versus-full-rebuild benchmark\n' + ' evidence with projection parity, affected-message counts, wall time, and peak RSS.\n' + ) + addition = ( + anchor + + '- Harden incremental snapshot publication and restore so schema versions require\n' + + ' exact non-boolean integers and hostile nesting or unencodable Unicode fails\n' + + ' closed with `IncrementalThreadError` instead of leaking runtime exceptions.\n' + ) + if text.count(anchor) != 1: + raise SystemExit('CHANGELOG insertion anchor changed') + changelog.write_text(text.replace(anchor, addition, 1), encoding='utf-8') + + docs = Path('docs/incremental-threading.md') + text = docs.read_text(encoding='utf-8') + anchor = ( + 'Payload objects and derived graph pointers are never serialized. Restored\n' + 'messages therefore have `payload=None`. Date values use an explicit tagged text\n' + 'or ISO-8601 datetime representation. Restore rejects unknown schema versions,\n' + 'extra or missing fields, duplicate keys, malformed types, invalid external IDs,\n' + 'and configured record or byte limits before publishing state.\n' + ) + replacement = ( + 'Payload objects and derived graph pointers are never serialized. Restored\n' + 'messages therefore have `payload=None`. Date values use an explicit tagged text\n' + 'or ISO-8601 datetime representation. Schema versions must be exact non-boolean\n' + 'integers. Restore rejects unknown versions, extra or missing fields, duplicate\n' + 'keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode,\n' + 'and configured record or byte limits through `IncrementalThreadError` before\n' + 'publishing state.\n' + ) + if text.count(anchor) != 1: + raise SystemExit('snapshot documentation anchor changed') + docs.write_text(text.replace(anchor, replacement, 1), encoding='utf-8') + PY + + - name: Verify the complete repository + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit the verified product fix + shell: bash + run: | + set -euo pipefail + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add \ + src/threadweave/incremental.py \ + tests/test_incremental_snapshot.py \ + docs/incremental-threading.md \ + CHANGELOG.md + git diff --cached --check + git commit -m 'fix: harden incremental snapshot validation' + git push origin HEAD:feature/incremental-thread-index From 5d12348f2ae7fcb30cae7c938ee9d7c3beefed15 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:18:53 +0900 Subject: [PATCH 50/97] ci: make snapshot hardening verifier anchor-safe --- .../workflows/apply-snapshot-hardening.yml | 103 ++++++++---------- 1 file changed, 47 insertions(+), 56 deletions(-) diff --git a/.github/workflows/apply-snapshot-hardening.yml b/.github/workflows/apply-snapshot-hardening.yml index e92f3ba..548e4e8 100644 --- a/.github/workflows/apply-snapshot-hardening.yml +++ b/.github/workflows/apply-snapshot-hardening.yml @@ -51,16 +51,8 @@ jobs: text = path.read_text(encoding='utf-8') if 'import sys\n' not in text: text = text.replace('import json\n', 'import json\nimport sys\n', 1) - anchor = ''' with pytest.raises(IncrementalThreadError, match="snapshot fields"): - IncrementalThreadIndex.restore(missing) - - - def test_restore_rejects_malformed_record_fields_and_duplicate_keys(): - ''' - addition = ''' with pytest.raises(IncrementalThreadError, match="snapshot fields"): - IncrementalThreadIndex.restore(missing) - - + marker = '\ndef test_restore_rejects_malformed_record_fields_and_duplicate_keys():\n' + block = ''' @pytest.mark.parametrize("invalid_schema_version", [True, 1.0]) def test_restore_requires_an_exact_integer_schema_version( invalid_schema_version: object, @@ -111,12 +103,10 @@ jobs: with pytest.raises(IncrementalThreadError, match="JSON-safe"): IncrementalThreadIndex.restore(snapshot) - - def test_restore_rejects_malformed_record_fields_and_duplicate_keys(): ''' - if anchor not in text: - raise SystemExit('snapshot regression insertion anchor changed') - path.write_text(text.replace(anchor, addition, 1), encoding='utf-8') + if text.count(marker) != 1: + raise SystemExit('snapshot regression insertion marker changed') + path.write_text(text.replace(marker, block + marker, 1), encoding='utf-8') PY set +e @@ -140,50 +130,51 @@ jobs: source = Path('src/threadweave/incremental.py') text = source.read_text(encoding='utf-8') - old = '''def _snapshot_json_bytes(value: object) -> bytes: - """Serialize a snapshot canonically or raise a bounded domain error.""" - try: - encoded = json.dumps( - value, - ensure_ascii=False, - allow_nan=False, - sort_keys=True, - separators=(",", ":"), - ) - except (TypeError, ValueError) as error: - raise IncrementalThreadError("snapshot must contain only JSON-safe values") from error - return encoded.encode("utf-8") - ''' - new = '''def _snapshot_json_bytes(value: object) -> bytes: - """Serialize a snapshot canonically or raise a bounded domain error.""" - try: - encoded = json.dumps( - value, - ensure_ascii=False, - allow_nan=False, - sort_keys=True, - separators=(",", ":"), - ) - return encoded.encode("utf-8") - except (RecursionError, TypeError, UnicodeError, ValueError) as error: - raise IncrementalThreadError( - "snapshot must contain only JSON-safe values" - ) from error - ''' + old = ( + ' try:\n' + ' encoded = json.dumps(\n' + ' value,\n' + ' ensure_ascii=False,\n' + ' allow_nan=False,\n' + ' sort_keys=True,\n' + ' separators=(",", ":"),\n' + ' )\n' + ' except (TypeError, ValueError) as error:\n' + ' raise IncrementalThreadError(' + '"snapshot must contain only JSON-safe values") from error\n' + ' return encoded.encode("utf-8")\n' + ) + new = ( + ' try:\n' + ' encoded = json.dumps(\n' + ' value,\n' + ' ensure_ascii=False,\n' + ' allow_nan=False,\n' + ' sort_keys=True,\n' + ' separators=(",", ":"),\n' + ' )\n' + ' return encoded.encode("utf-8")\n' + ' except (RecursionError, TypeError, UnicodeError, ValueError) as error:\n' + ' raise IncrementalThreadError(\n' + ' "snapshot must contain only JSON-safe values"\n' + ' ) from error\n' + ) if text.count(old) != 1: raise SystemExit('snapshot serializer anchor changed') text = text.replace(old, new, 1) - old = ''' if snapshot["schema_version"] != _SNAPSHOT_SCHEMA_VERSION: - raise IncrementalThreadError("unsupported snapshot schema_version") - ''' - new = ''' schema_version = snapshot["schema_version"] - if ( - isinstance(schema_version, bool) - or not isinstance(schema_version, int) - or schema_version != _SNAPSHOT_SCHEMA_VERSION - ): - raise IncrementalThreadError("unsupported snapshot schema_version") - ''' + old = ( + ' if snapshot["schema_version"] != _SNAPSHOT_SCHEMA_VERSION:\n' + ' raise IncrementalThreadError("unsupported snapshot schema_version")\n' + ) + new = ( + ' schema_version = snapshot["schema_version"]\n' + ' if (\n' + ' isinstance(schema_version, bool)\n' + ' or not isinstance(schema_version, int)\n' + ' or schema_version != _SNAPSHOT_SCHEMA_VERSION\n' + ' ):\n' + ' raise IncrementalThreadError("unsupported snapshot schema_version")\n' + ) if text.count(old) != 1: raise SystemExit('schema-version anchor changed') source.write_text(text.replace(old, new, 1), encoding='utf-8') From 4520933e33f56a9b06c662444beb35199a6f749c Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 5 Aug 2026 12:19:38 +0000 Subject: [PATCH 51/97] fix: harden incremental snapshot validation --- CHANGELOG.md | 3 ++ docs/incremental-threading.md | 8 +++-- src/threadweave/incremental.py | 15 ++++++--- tests/test_incremental_snapshot.py | 52 ++++++++++++++++++++++++++++++ 4 files changed, 71 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a32d0b0..2c55587 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). touched, and defer complete forest materialization until a caller requests it. - Add deterministic 100,000-message incremental-versus-full-rebuild benchmark evidence with projection parity, affected-message counts, wall time, and peak RSS. +- Harden incremental snapshot publication and restore so schema versions require + exact non-boolean integers and hostile nesting or unencodable Unicode fails + closed with `IncrementalThreadError` instead of leaking runtime exceptions. - Add an atomic `IncrementalThreadIndex` for mailbox additions, replacements, and removals with stable caller message keys, affected-component diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index 3889a62..0aba4b3 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -196,9 +196,11 @@ Schema version 1 stores: Payload objects and derived graph pointers are never serialized. Restored messages therefore have `payload=None`. Date values use an explicit tagged text -or ISO-8601 datetime representation. Restore rejects unknown schema versions, -extra or missing fields, duplicate keys, malformed types, invalid external IDs, -and configured record or byte limits before publishing state. +or ISO-8601 datetime representation. Schema versions must be exact non-boolean +integers. Restore rejects unknown versions, extra or missing fields, duplicate +keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, +and configured record or byte limits through `IncrementalThreadError` before +publishing state. ## Correctness and operational boundaries diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index 4be1c57..83e3773 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -808,9 +808,11 @@ def _snapshot_json_bytes(value: object) -> bytes: sort_keys=True, separators=(",", ":"), ) - except (TypeError, ValueError) as error: - raise IncrementalThreadError("snapshot must contain only JSON-safe values") from error - return encoded.encode("utf-8") + return encoded.encode("utf-8") + except (RecursionError, TypeError, UnicodeError, ValueError) as error: + raise IncrementalThreadError( + "snapshot must contain only JSON-safe values" + ) from error def _required_fields(value: Mapping[str, object], expected: set[str], name: str) -> None: @@ -1220,7 +1222,12 @@ def restore( {"schema_version", "version", "options", "records"}, "snapshot", ) - if snapshot["schema_version"] != _SNAPSHOT_SCHEMA_VERSION: + schema_version = snapshot["schema_version"] + if ( + isinstance(schema_version, bool) + or not isinstance(schema_version, int) + or schema_version != _SNAPSHOT_SCHEMA_VERSION + ): raise IncrementalThreadError("unsupported snapshot schema_version") version = _validated_nonnegative_integer(snapshot["version"], "version") options = snapshot["options"] diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py index a511ea9..e7d73d0 100644 --- a/tests/test_incremental_snapshot.py +++ b/tests/test_incremental_snapshot.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import sys from copy import deepcopy from datetime import datetime, timezone @@ -149,6 +150,57 @@ def test_restore_rejects_unknown_root_fields_and_schema_versions(): IncrementalThreadIndex.restore(missing) +@pytest.mark.parametrize("invalid_schema_version", [True, 1.0]) +def test_restore_requires_an_exact_integer_schema_version( + invalid_schema_version: object, +): + """Boolean and floating-point lookalikes cannot select a snapshot schema.""" + snapshot = _index().snapshot() + snapshot["schema_version"] = invalid_schema_version + + with pytest.raises(IncrementalThreadError, match="schema_version"): + IncrementalThreadIndex.restore(snapshot) + + +def test_snapshot_reports_unencodable_unicode_as_a_domain_error(): + """Lone surrogates fail closed instead of leaking a codec exception.""" + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + IndexedMessage( + "surrogate_key", + Message(message_id="surrogate", subject="bad\ud800subject"), + ), + ), + ) + ) + + with pytest.raises(IncrementalThreadError, match="JSON-safe"): + index.snapshot() + + +def test_restore_reports_excessive_json_nesting_as_a_domain_error(): + """Hostile nesting cannot escape the snapshot boundary as RecursionError.""" + nested: object = [] + for _ in range(sys.getrecursionlimit() * 20): + nested = [nested] + snapshot = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [], + "unexpected": nested, + } + + with pytest.raises(IncrementalThreadError, match="JSON-safe"): + IncrementalThreadIndex.restore(snapshot) + + def test_restore_rejects_malformed_record_fields_and_duplicate_keys(): """Record decoding rejects ambiguity before constructing any graph state.""" snapshot = _index().snapshot() From fb48d5e9b0de07b24e2dd70abf9f6d7a1d243cb2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:21:02 +0900 Subject: [PATCH 52/97] ci: remove completed snapshot hardening verifier --- .../workflows/apply-snapshot-hardening.yml | 269 ------------------ 1 file changed, 269 deletions(-) delete mode 100644 .github/workflows/apply-snapshot-hardening.yml diff --git a/.github/workflows/apply-snapshot-hardening.yml b/.github/workflows/apply-snapshot-hardening.yml deleted file mode 100644 index 548e4e8..0000000 --- a/.github/workflows/apply-snapshot-hardening.yml +++ /dev/null @@ -1,269 +0,0 @@ -name: Apply incremental snapshot hardening - -on: - pull_request: - types: [synchronize] - paths: - - .github/workflows/apply-snapshot-hardening.yml - -permissions: - contents: write - -concurrency: - group: apply-incremental-snapshot-hardening - cancel-in-progress: false - -jobs: - apply: - if: >- - github.event.pull_request.number == 20 && - github.event.pull_request.head.repo.full_name == github.repository && - github.event.pull_request.head.ref == 'feature/incremental-thread-index' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact pull-request head - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Add focused snapshot regressions and prove RED - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PY' - from pathlib import Path - - path = Path('tests/test_incremental_snapshot.py') - text = path.read_text(encoding='utf-8') - if 'import sys\n' not in text: - text = text.replace('import json\n', 'import json\nimport sys\n', 1) - marker = '\ndef test_restore_rejects_malformed_record_fields_and_duplicate_keys():\n' - block = ''' - @pytest.mark.parametrize("invalid_schema_version", [True, 1.0]) - def test_restore_requires_an_exact_integer_schema_version( - invalid_schema_version: object, - ): - """Boolean and floating-point lookalikes cannot select a snapshot schema.""" - snapshot = _index().snapshot() - snapshot["schema_version"] = invalid_schema_version - - with pytest.raises(IncrementalThreadError, match="schema_version"): - IncrementalThreadIndex.restore(snapshot) - - - def test_snapshot_reports_unencodable_unicode_as_a_domain_error(): - """Lone surrogates fail closed instead of leaking a codec exception.""" - index = IncrementalThreadIndex() - index.apply( - MailboxChangeSet( - expected_version=0, - additions=( - IndexedMessage( - "surrogate_key", - Message(message_id="surrogate", subject="bad\\ud800subject"), - ), - ), - ) - ) - - with pytest.raises(IncrementalThreadError, match="JSON-safe"): - index.snapshot() - - - def test_restore_reports_excessive_json_nesting_as_a_domain_error(): - """Hostile nesting cannot escape the snapshot boundary as RecursionError.""" - nested: object = [] - for _ in range(sys.getrecursionlimit() * 20): - nested = [nested] - snapshot = { - "schema_version": 1, - "version": 0, - "options": { - "group_by_subject": False, - "sort_by_sent_date": False, - }, - "records": [], - "unexpected": nested, - } - - with pytest.raises(IncrementalThreadError, match="JSON-safe"): - IncrementalThreadIndex.restore(snapshot) - - ''' - if text.count(marker) != 1: - raise SystemExit('snapshot regression insertion marker changed') - path.write_text(text.replace(marker, block + marker, 1), encoding='utf-8') - PY - - set +e - python -m pytest -q tests/test_incremental_snapshot.py \ - -k 'exact_integer_schema_version or unencodable_unicode or excessive_json_nesting' \ - >"$RUNNER_TEMP/snapshot-red.log" 2>&1 - status=$? - set -e - cat "$RUNNER_TEMP/snapshot-red.log" - if [ "$status" -eq 0 ]; then - echo 'The snapshot regressions unexpectedly passed before implementation.' >&2 - exit 1 - fi - - - name: Implement the fail-closed snapshot boundary and documentation - shell: bash - run: | - set -euo pipefail - python - <<'PY' - from pathlib import Path - - source = Path('src/threadweave/incremental.py') - text = source.read_text(encoding='utf-8') - old = ( - ' try:\n' - ' encoded = json.dumps(\n' - ' value,\n' - ' ensure_ascii=False,\n' - ' allow_nan=False,\n' - ' sort_keys=True,\n' - ' separators=(",", ":"),\n' - ' )\n' - ' except (TypeError, ValueError) as error:\n' - ' raise IncrementalThreadError(' - '"snapshot must contain only JSON-safe values") from error\n' - ' return encoded.encode("utf-8")\n' - ) - new = ( - ' try:\n' - ' encoded = json.dumps(\n' - ' value,\n' - ' ensure_ascii=False,\n' - ' allow_nan=False,\n' - ' sort_keys=True,\n' - ' separators=(",", ":"),\n' - ' )\n' - ' return encoded.encode("utf-8")\n' - ' except (RecursionError, TypeError, UnicodeError, ValueError) as error:\n' - ' raise IncrementalThreadError(\n' - ' "snapshot must contain only JSON-safe values"\n' - ' ) from error\n' - ) - if text.count(old) != 1: - raise SystemExit('snapshot serializer anchor changed') - text = text.replace(old, new, 1) - old = ( - ' if snapshot["schema_version"] != _SNAPSHOT_SCHEMA_VERSION:\n' - ' raise IncrementalThreadError("unsupported snapshot schema_version")\n' - ) - new = ( - ' schema_version = snapshot["schema_version"]\n' - ' if (\n' - ' isinstance(schema_version, bool)\n' - ' or not isinstance(schema_version, int)\n' - ' or schema_version != _SNAPSHOT_SCHEMA_VERSION\n' - ' ):\n' - ' raise IncrementalThreadError("unsupported snapshot schema_version")\n' - ) - if text.count(old) != 1: - raise SystemExit('schema-version anchor changed') - source.write_text(text.replace(old, new, 1), encoding='utf-8') - - changelog = Path('CHANGELOG.md') - text = changelog.read_text(encoding='utf-8') - anchor = ( - '- Add deterministic 100,000-message incremental-versus-full-rebuild benchmark\n' - ' evidence with projection parity, affected-message counts, wall time, and peak RSS.\n' - ) - addition = ( - anchor - + '- Harden incremental snapshot publication and restore so schema versions require\n' - + ' exact non-boolean integers and hostile nesting or unencodable Unicode fails\n' - + ' closed with `IncrementalThreadError` instead of leaking runtime exceptions.\n' - ) - if text.count(anchor) != 1: - raise SystemExit('CHANGELOG insertion anchor changed') - changelog.write_text(text.replace(anchor, addition, 1), encoding='utf-8') - - docs = Path('docs/incremental-threading.md') - text = docs.read_text(encoding='utf-8') - anchor = ( - 'Payload objects and derived graph pointers are never serialized. Restored\n' - 'messages therefore have `payload=None`. Date values use an explicit tagged text\n' - 'or ISO-8601 datetime representation. Restore rejects unknown schema versions,\n' - 'extra or missing fields, duplicate keys, malformed types, invalid external IDs,\n' - 'and configured record or byte limits before publishing state.\n' - ) - replacement = ( - 'Payload objects and derived graph pointers are never serialized. Restored\n' - 'messages therefore have `payload=None`. Date values use an explicit tagged text\n' - 'or ISO-8601 datetime representation. Schema versions must be exact non-boolean\n' - 'integers. Restore rejects unknown versions, extra or missing fields, duplicate\n' - 'keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode,\n' - 'and configured record or byte limits through `IncrementalThreadError` before\n' - 'publishing state.\n' - ) - if text.count(anchor) != 1: - raise SystemExit('snapshot documentation anchor changed') - docs.write_text(text.replace(anchor, replacement, 1), encoding='utf-8') - PY - - - name: Verify the complete repository - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit the verified product fix - shell: bash - run: | - set -euo pipefail - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add \ - src/threadweave/incremental.py \ - tests/test_incremental_snapshot.py \ - docs/incremental-threading.md \ - CHANGELOG.md - git diff --cached --check - git commit -m 'fix: harden incremental snapshot validation' - git push origin HEAD:feature/incremental-thread-index From 6a7f75e024d938f8f5f48f3ed57eb0363106aa10 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:54:01 +0900 Subject: [PATCH 53/97] test: stage plain-container snapshot hardening --- tools/pr20-plain-json-hardening.patch | 122 ++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 tools/pr20-plain-json-hardening.patch diff --git a/tools/pr20-plain-json-hardening.patch b/tools/pr20-plain-json-hardening.patch new file mode 100644 index 0000000..1b56505 --- /dev/null +++ b/tools/pr20-plain-json-hardening.patch @@ -0,0 +1,122 @@ +diff --git a/CHANGELOG.md b/CHANGELOG.md +index 2c55587..a93e06d 100644 +--- a/CHANGELOG.md ++++ b/CHANGELOG.md +@@ -6,6 +6,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + + ## Unreleased + ++- Reject dictionary and list subclasses at the incremental snapshot restore ++ boundary before JSON encoding can invoke untrusted iterator overrides. + - Serialize every read and write on one `IncrementalThreadIndex` with a + process-local reentrant lock so same-version concurrent writers yield one + commit and one explicit conflict, while readers observe only committed state. +diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md +index 0aba4b3..d72b48d 100644 +--- a/docs/incremental-threading.md ++++ b/docs/incremental-threading.md +@@ -200,7 +200,9 @@ or ISO-8601 datetime representation. Schema versions must be exact non-boolean + integers. Restore rejects unknown versions, extra or missing fields, duplicate + keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, + and configured record or byte limits through `IncrementalThreadError` before +-publishing state. ++publishing state. Only built-in JSON dictionaries and lists are accepted; container ++subclasses are rejected before serialization so hostile ``items`` or iterator ++overrides cannot execute inside the restore boundary. + + ## Correctness and operational boundaries + +diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py +index 83e3773..d5a851b 100644 +--- a/src/threadweave/incremental.py ++++ b/src/threadweave/incremental.py +@@ -798,6 +798,28 @@ def _decoded_date(value: object, name: str) -> str | datetime | None: + raise IncrementalThreadError(f"{name} date kind is unsupported") + + ++def _require_plain_json_containers(value: object) -> None: ++ """Reject executable container subclasses before JSON serialization. ++ ++ JSON-decoded state consists of built-in dictionaries, lists, and scalar ++ values. Requiring exact container types prevents untrusted ``items`` or ++ iterator overrides from executing inside the restore boundary. ++ """ ++ pending = [value] ++ while pending: ++ current = pending.pop() ++ if type(current) is dict: ++ pending.extend(dict.values(current)) ++ elif type(current) is list: ++ pending.extend(current) ++ elif current is None or isinstance(current, (str, int, float, bool)): ++ continue ++ else: ++ raise IncrementalThreadError( ++ "snapshot must contain only plain JSON containers and scalar values" ++ ) ++ ++ + def _snapshot_json_bytes(value: object) -> bytes: + """Serialize a snapshot canonically or raise a bounded domain error.""" + try: +@@ -1215,6 +1237,7 @@ class IncrementalThreadIndex: + ) + if not isinstance(snapshot, Mapping): + raise IncrementalThreadError("snapshot must be a mapping") ++ _require_plain_json_containers(snapshot) + if len(_snapshot_json_bytes(snapshot)) > max_bytes: + raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") + _required_fields( +diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py +index e7d73d0..9020bd2 100644 +--- a/tests/test_incremental_snapshot.py ++++ b/tests/test_incremental_snapshot.py +@@ -282,3 +282,48 @@ def test_restore_rejects_non_mapping_and_non_json_safe_values(): + invalid["records"][0]["email_id"] = object() + with pytest.raises(IncrementalThreadError, match="JSON"): + IncrementalThreadIndex.restore(invalid) ++ ++ ++class _HostileDictionary(dict): ++ """Dictionary subclass whose iteration must never cross the snapshot boundary.""" ++ ++ def items(self): ++ """Raise if generic JSON encoding invokes subclass behavior.""" ++ raise RuntimeError("hostile dictionary iteration") ++ ++ ++class _HostileList(list): ++ """List subclass whose iteration must never cross the snapshot boundary.""" ++ ++ def __iter__(self): ++ """Raise if generic JSON encoding invokes subclass behavior.""" ++ raise RuntimeError("hostile list iteration") ++ ++ ++def test_restore_rejects_container_subclasses_before_they_execute(): ++ """Only plain JSON containers may enter the untrusted snapshot decoder.""" ++ root = _HostileDictionary( ++ { ++ "schema_version": 1, ++ "version": 0, ++ "options": { ++ "group_by_subject": False, ++ "sort_by_sent_date": False, ++ }, ++ "records": [], ++ } ++ ) ++ with pytest.raises(IncrementalThreadError, match="plain JSON containers"): ++ IncrementalThreadIndex.restore(root) ++ ++ nested = { ++ "schema_version": 1, ++ "version": 0, ++ "options": { ++ "group_by_subject": False, ++ "sort_by_sent_date": False, ++ }, ++ "records": _HostileList(), ++ } ++ with pytest.raises(IncrementalThreadError, match="plain JSON containers"): ++ IncrementalThreadIndex.restore(nested) From 5e7e8c468ff42727a7b2e46c231a2c613bd808e3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:54:58 +0900 Subject: [PATCH 54/97] ci: verify and apply plain-container snapshot hardening --- .../apply-pr20-plain-json-hardening.yml | 101 ++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 .github/workflows/apply-pr20-plain-json-hardening.yml diff --git a/.github/workflows/apply-pr20-plain-json-hardening.yml b/.github/workflows/apply-pr20-plain-json-hardening.yml new file mode 100644 index 0000000..51b0a0b --- /dev/null +++ b/.github/workflows/apply-pr20-plain-json-hardening.yml @@ -0,0 +1,101 @@ +name: Apply PR 20 plain JSON hardening + +on: + pull_request: + types: [synchronize] + paths: + - .github/workflows/apply-pr20-plain-json-hardening.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-plain-json-hardening + cancel-in-progress: false + +jobs: + apply-and-verify: + if: >- + github.event.pull_request.number == 20 && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.head.ref == 'feature/incremental-thread-index' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Verify and apply the reviewed patch + shell: bash + run: | + set -euo pipefail + printf '%s %s\n' \ + '7b3a7a72b804d8b4c71d0129ca0096da64b37d97da1a29989fd8f355199112f1' \ + 'tools/pr20-plain-json-hardening.patch' | sha256sum -c - + git apply --check tools/pr20-plain-json-hardening.patch + git apply tools/pr20-plain-json-hardening.patch + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Verify the patched repository state + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit the verified product update + shell: bash + run: | + set -euo pipefail + rm tools/pr20-plain-json-hardening.patch + rmdir tools 2>/dev/null || true + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add -A + if git diff --cached --name-only | grep -q '^\.github/workflows/'; then + echo 'Refusing to push workflow changes with the repository token.' >&2 + exit 1 + fi + git diff --cached --check + git commit -m 'fix: reject executable snapshot container subclasses' + git push origin HEAD:feature/incremental-thread-index From 4115e89139f327f667ef7894a2a2ef36c70ef34a Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 5 Aug 2026 12:55:37 +0000 Subject: [PATCH 55/97] fix: reject executable snapshot container subclasses --- CHANGELOG.md | 2 + docs/incremental-threading.md | 4 +- src/threadweave/incremental.py | 23 +++++ tests/test_incremental_snapshot.py | 45 ++++++++++ tools/pr20-plain-json-hardening.patch | 122 -------------------------- 5 files changed, 73 insertions(+), 123 deletions(-) delete mode 100644 tools/pr20-plain-json-hardening.patch diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c55587..a93e06d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Reject dictionary and list subclasses at the incremental snapshot restore + boundary before JSON encoding can invoke untrusted iterator overrides. - Serialize every read and write on one `IncrementalThreadIndex` with a process-local reentrant lock so same-version concurrent writers yield one commit and one explicit conflict, while readers observe only committed state. diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index 0aba4b3..d72b48d 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -200,7 +200,9 @@ or ISO-8601 datetime representation. Schema versions must be exact non-boolean integers. Restore rejects unknown versions, extra or missing fields, duplicate keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, and configured record or byte limits through `IncrementalThreadError` before -publishing state. +publishing state. Only built-in JSON dictionaries and lists are accepted; container +subclasses are rejected before serialization so hostile ``items`` or iterator +overrides cannot execute inside the restore boundary. ## Correctness and operational boundaries diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index 83e3773..d5a851b 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -798,6 +798,28 @@ def _decoded_date(value: object, name: str) -> str | datetime | None: raise IncrementalThreadError(f"{name} date kind is unsupported") +def _require_plain_json_containers(value: object) -> None: + """Reject executable container subclasses before JSON serialization. + + JSON-decoded state consists of built-in dictionaries, lists, and scalar + values. Requiring exact container types prevents untrusted ``items`` or + iterator overrides from executing inside the restore boundary. + """ + pending = [value] + while pending: + current = pending.pop() + if type(current) is dict: + pending.extend(dict.values(current)) + elif type(current) is list: + pending.extend(current) + elif current is None or isinstance(current, (str, int, float, bool)): + continue + else: + raise IncrementalThreadError( + "snapshot must contain only plain JSON containers and scalar values" + ) + + def _snapshot_json_bytes(value: object) -> bytes: """Serialize a snapshot canonically or raise a bounded domain error.""" try: @@ -1215,6 +1237,7 @@ def restore( ) if not isinstance(snapshot, Mapping): raise IncrementalThreadError("snapshot must be a mapping") + _require_plain_json_containers(snapshot) if len(_snapshot_json_bytes(snapshot)) > max_bytes: raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") _required_fields( diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py index e7d73d0..9020bd2 100644 --- a/tests/test_incremental_snapshot.py +++ b/tests/test_incremental_snapshot.py @@ -282,3 +282,48 @@ def test_restore_rejects_non_mapping_and_non_json_safe_values(): invalid["records"][0]["email_id"] = object() with pytest.raises(IncrementalThreadError, match="JSON"): IncrementalThreadIndex.restore(invalid) + + +class _HostileDictionary(dict): + """Dictionary subclass whose iteration must never cross the snapshot boundary.""" + + def items(self): + """Raise if generic JSON encoding invokes subclass behavior.""" + raise RuntimeError("hostile dictionary iteration") + + +class _HostileList(list): + """List subclass whose iteration must never cross the snapshot boundary.""" + + def __iter__(self): + """Raise if generic JSON encoding invokes subclass behavior.""" + raise RuntimeError("hostile list iteration") + + +def test_restore_rejects_container_subclasses_before_they_execute(): + """Only plain JSON containers may enter the untrusted snapshot decoder.""" + root = _HostileDictionary( + { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [], + } + ) + with pytest.raises(IncrementalThreadError, match="plain JSON containers"): + IncrementalThreadIndex.restore(root) + + nested = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": _HostileList(), + } + with pytest.raises(IncrementalThreadError, match="plain JSON containers"): + IncrementalThreadIndex.restore(nested) diff --git a/tools/pr20-plain-json-hardening.patch b/tools/pr20-plain-json-hardening.patch deleted file mode 100644 index 1b56505..0000000 --- a/tools/pr20-plain-json-hardening.patch +++ /dev/null @@ -1,122 +0,0 @@ -diff --git a/CHANGELOG.md b/CHANGELOG.md -index 2c55587..a93e06d 100644 ---- a/CHANGELOG.md -+++ b/CHANGELOG.md -@@ -6,6 +6,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - - ## Unreleased - -+- Reject dictionary and list subclasses at the incremental snapshot restore -+ boundary before JSON encoding can invoke untrusted iterator overrides. - - Serialize every read and write on one `IncrementalThreadIndex` with a - process-local reentrant lock so same-version concurrent writers yield one - commit and one explicit conflict, while readers observe only committed state. -diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md -index 0aba4b3..d72b48d 100644 ---- a/docs/incremental-threading.md -+++ b/docs/incremental-threading.md -@@ -200,7 +200,9 @@ or ISO-8601 datetime representation. Schema versions must be exact non-boolean - integers. Restore rejects unknown versions, extra or missing fields, duplicate - keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, - and configured record or byte limits through `IncrementalThreadError` before --publishing state. -+publishing state. Only built-in JSON dictionaries and lists are accepted; container -+subclasses are rejected before serialization so hostile ``items`` or iterator -+overrides cannot execute inside the restore boundary. - - ## Correctness and operational boundaries - -diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py -index 83e3773..d5a851b 100644 ---- a/src/threadweave/incremental.py -+++ b/src/threadweave/incremental.py -@@ -798,6 +798,28 @@ def _decoded_date(value: object, name: str) -> str | datetime | None: - raise IncrementalThreadError(f"{name} date kind is unsupported") - - -+def _require_plain_json_containers(value: object) -> None: -+ """Reject executable container subclasses before JSON serialization. -+ -+ JSON-decoded state consists of built-in dictionaries, lists, and scalar -+ values. Requiring exact container types prevents untrusted ``items`` or -+ iterator overrides from executing inside the restore boundary. -+ """ -+ pending = [value] -+ while pending: -+ current = pending.pop() -+ if type(current) is dict: -+ pending.extend(dict.values(current)) -+ elif type(current) is list: -+ pending.extend(current) -+ elif current is None or isinstance(current, (str, int, float, bool)): -+ continue -+ else: -+ raise IncrementalThreadError( -+ "snapshot must contain only plain JSON containers and scalar values" -+ ) -+ -+ - def _snapshot_json_bytes(value: object) -> bytes: - """Serialize a snapshot canonically or raise a bounded domain error.""" - try: -@@ -1215,6 +1237,7 @@ class IncrementalThreadIndex: - ) - if not isinstance(snapshot, Mapping): - raise IncrementalThreadError("snapshot must be a mapping") -+ _require_plain_json_containers(snapshot) - if len(_snapshot_json_bytes(snapshot)) > max_bytes: - raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") - _required_fields( -diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py -index e7d73d0..9020bd2 100644 ---- a/tests/test_incremental_snapshot.py -+++ b/tests/test_incremental_snapshot.py -@@ -282,3 +282,48 @@ def test_restore_rejects_non_mapping_and_non_json_safe_values(): - invalid["records"][0]["email_id"] = object() - with pytest.raises(IncrementalThreadError, match="JSON"): - IncrementalThreadIndex.restore(invalid) -+ -+ -+class _HostileDictionary(dict): -+ """Dictionary subclass whose iteration must never cross the snapshot boundary.""" -+ -+ def items(self): -+ """Raise if generic JSON encoding invokes subclass behavior.""" -+ raise RuntimeError("hostile dictionary iteration") -+ -+ -+class _HostileList(list): -+ """List subclass whose iteration must never cross the snapshot boundary.""" -+ -+ def __iter__(self): -+ """Raise if generic JSON encoding invokes subclass behavior.""" -+ raise RuntimeError("hostile list iteration") -+ -+ -+def test_restore_rejects_container_subclasses_before_they_execute(): -+ """Only plain JSON containers may enter the untrusted snapshot decoder.""" -+ root = _HostileDictionary( -+ { -+ "schema_version": 1, -+ "version": 0, -+ "options": { -+ "group_by_subject": False, -+ "sort_by_sent_date": False, -+ }, -+ "records": [], -+ } -+ ) -+ with pytest.raises(IncrementalThreadError, match="plain JSON containers"): -+ IncrementalThreadIndex.restore(root) -+ -+ nested = { -+ "schema_version": 1, -+ "version": 0, -+ "options": { -+ "group_by_subject": False, -+ "sort_by_sent_date": False, -+ }, -+ "records": _HostileList(), -+ } -+ with pytest.raises(IncrementalThreadError, match="plain JSON containers"): -+ IncrementalThreadIndex.restore(nested) From 635e14c1b4fa4d2fa371c9621319c4a42be858b1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 21:56:24 +0900 Subject: [PATCH 56/97] ci: remove completed plain-JSON hardening verifier --- .../apply-pr20-plain-json-hardening.yml | 101 ------------------ 1 file changed, 101 deletions(-) delete mode 100644 .github/workflows/apply-pr20-plain-json-hardening.yml diff --git a/.github/workflows/apply-pr20-plain-json-hardening.yml b/.github/workflows/apply-pr20-plain-json-hardening.yml deleted file mode 100644 index 51b0a0b..0000000 --- a/.github/workflows/apply-pr20-plain-json-hardening.yml +++ /dev/null @@ -1,101 +0,0 @@ -name: Apply PR 20 plain JSON hardening - -on: - pull_request: - types: [synchronize] - paths: - - .github/workflows/apply-pr20-plain-json-hardening.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-plain-json-hardening - cancel-in-progress: false - -jobs: - apply-and-verify: - if: >- - github.event.pull_request.number == 20 && - github.event.pull_request.head.repo.full_name == github.repository && - github.event.pull_request.head.ref == 'feature/incremental-thread-index' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Verify and apply the reviewed patch - shell: bash - run: | - set -euo pipefail - printf '%s %s\n' \ - '7b3a7a72b804d8b4c71d0129ca0096da64b37d97da1a29989fd8f355199112f1' \ - 'tools/pr20-plain-json-hardening.patch' | sha256sum -c - - git apply --check tools/pr20-plain-json-hardening.patch - git apply tools/pr20-plain-json-hardening.patch - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Verify the patched repository state - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit the verified product update - shell: bash - run: | - set -euo pipefail - rm tools/pr20-plain-json-hardening.patch - rmdir tools 2>/dev/null || true - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add -A - if git diff --cached --name-only | grep -q '^\.github/workflows/'; then - echo 'Refusing to push workflow changes with the repository token.' >&2 - exit 1 - fi - git diff --cached --check - git commit -m 'fix: reject executable snapshot container subclasses' - git push origin HEAD:feature/incremental-thread-index From 344d21ff9b88e2f011b803809ca1e571133d6012 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:03:54 +0900 Subject: [PATCH 57/97] test: stage cyclic snapshot regression --- tools/pr20-cycle-test.patch | 62 +++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 tools/pr20-cycle-test.patch diff --git a/tools/pr20-cycle-test.patch b/tools/pr20-cycle-test.patch new file mode 100644 index 0000000..9d24563 --- /dev/null +++ b/tools/pr20-cycle-test.patch @@ -0,0 +1,62 @@ +diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py +index 9020bd2..0ef4ea1 100644 +--- a/tests/test_incremental_snapshot.py ++++ b/tests/test_incremental_snapshot.py +@@ -3,6 +3,7 @@ + from __future__ import annotations + + import json ++import subprocess + import sys + from copy import deepcopy + from datetime import datetime, timezone +@@ -334,3 +335,40 @@ def test_restore_rejects_container_subclasses_before_they_execute(): + } + with pytest.raises(IncrementalThreadError, match="plain JSON containers"): + IncrementalThreadIndex.restore(nested) ++ ++ ++def test_restore_rejects_cyclic_plain_containers_without_hanging(): ++ """Cyclic built-in containers fail quickly instead of looping forever.""" ++ script = r""" ++from threadweave import IncrementalThreadError, IncrementalThreadIndex ++ ++mapping_cycle = { ++ "schema_version": 1, ++ "version": 0, ++ "options": { ++ "group_by_subject": False, ++ "sort_by_sent_date": False, ++ }, ++ "records": [], ++} ++mapping_cycle["self"] = mapping_cycle ++ ++list_cycle = [] ++list_cycle.append(list_cycle) ++nested_cycle = { ++ "schema_version": 1, ++ "version": 0, ++ "options": { ++ "group_by_subject": False, ++ "sort_by_sent_date": False, ++ }, ++ "records": list_cycle, ++} ++ ++for snapshot in (mapping_cycle, nested_cycle): ++ try: ++ IncrementalThreadIndex.restore(snapshot) ++ except IncrementalThreadError as error: ++ assert "cyclic" in str(error), str(error) ++ else: ++ raise AssertionError("cyclic snapshot was accepted") ++""" ++ result = subprocess.run( ++ [sys.executable, "-c", script], ++ capture_output=True, ++ text=True, ++ timeout=2, ++ check=False, ++ ) ++ assert result.returncode == 0, result.stderr From a1898712d28a0ddb90ec1d2242cbcdc87d9c410d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:04:48 +0900 Subject: [PATCH 58/97] fix: stage iterative cyclic snapshot detection --- tools/pr20-cycle-fix.patch | 105 +++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 tools/pr20-cycle-fix.patch diff --git a/tools/pr20-cycle-fix.patch b/tools/pr20-cycle-fix.patch new file mode 100644 index 0000000..119ecc9 --- /dev/null +++ b/tools/pr20-cycle-fix.patch @@ -0,0 +1,105 @@ +diff --git a/CHANGELOG.md b/CHANGELOG.md +index a93e06d..52a8987 100644 +--- a/CHANGELOG.md ++++ b/CHANGELOG.md +@@ -6,6 +6,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + + ## Unreleased + ++- Reject cyclic built-in dictionaries and lists at the incremental snapshot ++ restore boundary without recursion or unbounded traversal. + - Reject dictionary and list subclasses at the incremental snapshot restore + boundary before JSON encoding can invoke untrusted iterator overrides. + - Serialize every read and write on one `IncrementalThreadIndex` with a +diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md +index d72b48d..e19d7f2 100644 +--- a/docs/incremental-threading.md ++++ b/docs/incremental-threading.md +@@ -202,7 +202,9 @@ integers. Restore rejects unknown versions, extra or missing fields, duplicate + keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, + and configured record or byte limits through `IncrementalThreadError` before + publishing state. Only built-in JSON dictionaries and lists are accepted; container +-subclasses are rejected before serialization so hostile ``items`` or iterator +-overrides cannot execute inside the restore boundary. ++subclasses are rejected before serialization so hostile ``items`` or iterator ++overrides cannot execute inside the restore boundary. Cyclic built-in containers ++are detected with an iterative active-path guard and fail without recursion or ++unbounded traversal; repeated acyclic references remain valid JSON input. + + ## Correctness and operational boundaries + +diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py +index d5a851b..c91fcb6 100644 +--- a/src/threadweave/incremental.py ++++ b/src/threadweave/incremental.py +@@ -804,15 +804,29 @@ def _require_plain_json_containers(value: object) -> None: + values. Requiring exact container types prevents untrusted ``items`` or + iterator overrides from executing inside the restore boundary. + """ +- pending = [value] ++ pending = [(value, False)] ++ active_containers: set[int] = set() + while pending: +- current = pending.pop() +- if type(current) is dict: +- pending.extend(dict.values(current)) +- elif type(current) is list: +- pending.extend(current) ++ current, exiting = pending.pop() ++ if type(current) in {dict, list}: ++ identity = id(current) ++ if exiting: ++ active_containers.remove(identity) ++ continue ++ if identity in active_containers: ++ raise IncrementalThreadError( ++ "snapshot must not contain cyclic JSON containers" ++ ) ++ active_containers.add(identity) ++ pending.append((current, True)) ++ children = ( ++ dict.values(current) ++ if type(current) is dict ++ else current ++ ) ++ pending.extend((child, False) for child in children) + elif current is None or isinstance(current, (str, int, float, bool)): + continue + else: +diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py +index 0ef4ea1..b5e0c47 100644 +--- a/tests/test_incremental_snapshot.py ++++ b/tests/test_incremental_snapshot.py +@@ -8,6 +8,7 @@ from copy import deepcopy + from datetime import datetime, timezone + + import pytest ++import threadweave.incremental as incremental_module + + from threadweave import ( + IncrementalThreadError, +@@ -372,3 +373,24 @@ def test_restore_rejects_cyclic_plain_containers_without_hanging(): + check=False, + ) + assert result.returncode == 0, result.stderr ++ ++ ++def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): ++ """The iterative guard distinguishes cycles from repeated acyclic values.""" ++ mapping_cycle: dict[str, object] = {} ++ mapping_cycle["self"] = mapping_cycle ++ with pytest.raises(IncrementalThreadError, match="cyclic"): ++ incremental_module._require_plain_json_containers(mapping_cycle) ++ ++ list_cycle: list[object] = [] ++ list_cycle.append(list_cycle) ++ with pytest.raises(IncrementalThreadError, match="cyclic"): ++ incremental_module._require_plain_json_containers(list_cycle) ++ ++ shared: list[object] = [{"value": 1}] ++ incremental_module._require_plain_json_containers( ++ { ++ "first": shared, ++ "second": shared, ++ } ++ ) From 83141bbad7e26cf7840a9644be2b0203b3393a2b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:06:08 +0900 Subject: [PATCH 59/97] ci: prove and apply cyclic snapshot hardening --- .../workflows/apply-pr20-cycle-hardening.yml | 125 ++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 .github/workflows/apply-pr20-cycle-hardening.yml diff --git a/.github/workflows/apply-pr20-cycle-hardening.yml b/.github/workflows/apply-pr20-cycle-hardening.yml new file mode 100644 index 0000000..826e569 --- /dev/null +++ b/.github/workflows/apply-pr20-cycle-hardening.yml @@ -0,0 +1,125 @@ +name: Apply PR 20 cyclic snapshot hardening + +on: + pull_request: + types: [synchronize] + paths: + - .github/workflows/apply-pr20-cycle-hardening.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-cycle-hardening + cancel-in-progress: false + +jobs: + red-green-verify: + if: >- + github.event.pull_request.number == 20 && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.head.ref == 'feature/incremental-thread-index' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Prove the cyclic-container regression fails before the fix + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + printf '%s %s\n' \ + 'a1190b493a362ac3c9b0d26b8138e67c6f211be3a2fe16b3fde3e8e0afbf7ca7' \ + 'tools/pr20-cycle-test.patch' | sha256sum -c - + git apply --check tools/pr20-cycle-test.patch + git apply tools/pr20-cycle-test.patch + set +e + pytest -q tests/test_incremental_snapshot.py \ + -k restore_rejects_cyclic_plain_containers_without_hanging + red_status="$?" + set -e + if [ "$red_status" -eq 0 ]; then + echo 'The regression test unexpectedly passed before the fix.' >&2 + exit 1 + fi + echo 'Observed the expected pre-fix timeout/failure.' >>"$GITHUB_STEP_SUMMARY" + + - name: Apply the minimal iterative cycle detector + shell: bash + run: | + set -euo pipefail + printf '%s %s\n' \ + '59631c36acee10ca16359acb51041659f00b9a586515881ecd2129b440dd4b51' \ + 'tools/pr20-cycle-fix.patch' | sha256sum -c - + git apply --check tools/pr20-cycle-fix.patch + git apply tools/pr20-cycle-fix.patch + + - name: Verify the fixed repository state + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + pytest -q tests/test_incremental_snapshot.py \ + -k 'cyclic_plain_containers or plain_container_guard' + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit the verified product update + shell: bash + run: | + set -euo pipefail + rm tools/pr20-cycle-test.patch tools/pr20-cycle-fix.patch + rmdir tools 2>/dev/null || true + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add -A + if git diff --cached --name-only | grep -q '^\.github/workflows/'; then + echo 'Refusing to push workflow changes with the repository token.' >&2 + exit 1 + fi + git diff --cached --check + git commit -m 'fix: reject cyclic incremental snapshots iteratively' + git push origin HEAD:feature/incremental-thread-index From 812cacfadc37ee5665c701eb98a4c0ba27d3d978 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:08:37 +0900 Subject: [PATCH 60/97] ci: verify cyclic hardening patches by Git blob identity --- .github/workflows/apply-pr20-cycle-hardening.yml | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/.github/workflows/apply-pr20-cycle-hardening.yml b/.github/workflows/apply-pr20-cycle-hardening.yml index 826e569..21c0f99 100644 --- a/.github/workflows/apply-pr20-cycle-hardening.yml +++ b/.github/workflows/apply-pr20-cycle-hardening.yml @@ -44,9 +44,8 @@ jobs: PYTHONPATH: src run: | set -euo pipefail - printf '%s %s\n' \ - 'a1190b493a362ac3c9b0d26b8138e67c6f211be3a2fe16b3fde3e8e0afbf7ca7' \ - 'tools/pr20-cycle-test.patch' | sha256sum -c - + test "$(git hash-object tools/pr20-cycle-test.patch)" = \ + '9d2456384c78d1c2ecefaec210ab167aa0b1ce63' git apply --check tools/pr20-cycle-test.patch git apply tools/pr20-cycle-test.patch set +e @@ -64,9 +63,8 @@ jobs: shell: bash run: | set -euo pipefail - printf '%s %s\n' \ - '59631c36acee10ca16359acb51041659f00b9a586515881ecd2129b440dd4b51' \ - 'tools/pr20-cycle-fix.patch' | sha256sum -c - + test "$(git hash-object tools/pr20-cycle-fix.patch)" = \ + '119ecc9a08a98f633a2eed10611ddd40e92a91a5' git apply --check tools/pr20-cycle-fix.patch git apply tools/pr20-cycle-fix.patch From e0aced3bbec1c1c8b138f4ba73ea1587b4f1b07b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:12:39 +0900 Subject: [PATCH 61/97] ci: run the cyclic snapshot fix through an exact-state TDD verifier --- .../workflows/apply-pr20-cycle-hardening.yml | 162 ++++++++++++++++-- 1 file changed, 147 insertions(+), 15 deletions(-) diff --git a/.github/workflows/apply-pr20-cycle-hardening.yml b/.github/workflows/apply-pr20-cycle-hardening.yml index 21c0f99..d8b76fd 100644 --- a/.github/workflows/apply-pr20-cycle-hardening.yml +++ b/.github/workflows/apply-pr20-cycle-hardening.yml @@ -44,29 +44,161 @@ jobs: PYTHONPATH: src run: | set -euo pipefail - test "$(git hash-object tools/pr20-cycle-test.patch)" = \ - '9d2456384c78d1c2ecefaec210ab167aa0b1ce63' - git apply --check tools/pr20-cycle-test.patch - git apply tools/pr20-cycle-test.patch + test "$(git hash-object src/threadweave/incremental.py)" = \ + 'd5a851bd835bfa1b6acc00b7ac20608e6abeb4f9' set +e - pytest -q tests/test_incremental_snapshot.py \ - -k restore_rejects_cyclic_plain_containers_without_hanging + timeout 2s python - <<'PY' + from threadweave import IncrementalThreadIndex + + cycle = {} + cycle["self"] = cycle + IncrementalThreadIndex.restore(cycle) + PY red_status="$?" set -e - if [ "$red_status" -eq 0 ]; then - echo 'The regression test unexpectedly passed before the fix.' >&2 + if [ "$red_status" -ne 124 ]; then + echo "Expected the pre-fix restore call to time out; status=$red_status" >&2 exit 1 fi - echo 'Observed the expected pre-fix timeout/failure.' >>"$GITHUB_STEP_SUMMARY" + echo 'Observed the expected pre-fix non-terminating traversal.' \ + >>"$GITHUB_STEP_SUMMARY" - - name: Apply the minimal iterative cycle detector + - name: Apply the minimal iterative cycle detector and regression tests shell: bash run: | set -euo pipefail - test "$(git hash-object tools/pr20-cycle-fix.patch)" = \ - '119ecc9a08a98f633a2eed10611ddd40e92a91a5' - git apply --check tools/pr20-cycle-fix.patch - git apply tools/pr20-cycle-fix.patch + test "$(git hash-object tests/test_incremental_snapshot.py)" = \ + '9020bd2953811ebcb814cc0aeaf7f1035e4992e2' + test "$(git hash-object CHANGELOG.md)" = \ + 'a93e06da23a04c9a4d484c8478283be075771aa8' + test "$(git hash-object docs/incremental-threading.md)" = \ + 'd72b48dbeeb4f64a4c78b58151775a010b1b4d1a' + python - <<'PY' + from pathlib import Path + + source_path = Path('src/threadweave/incremental.py') + source = source_path.read_text(encoding='utf-8') + old_source = ''' pending = [value] + while pending: + current = pending.pop() + if type(current) is dict: + pending.extend(dict.values(current)) + elif type(current) is list: + pending.extend(current) + elif current is None or isinstance(current, (str, int, float, bool)): + continue + else: + raise IncrementalThreadError( + "snapshot must contain only plain JSON containers and scalar values" + ) + ''' + new_source = ''' pending = [(value, False)] + active_containers: set[int] = set() + while pending: + current, exiting = pending.pop() + if type(current) in {dict, list}: + identity = id(current) + if exiting: + active_containers.remove(identity) + continue + if identity in active_containers: + raise IncrementalThreadError( + "snapshot must not contain cyclic JSON containers" + ) + active_containers.add(identity) + pending.append((current, True)) + children = dict.values(current) if type(current) is dict else current + pending.extend((child, False) for child in children) + elif current is None or isinstance(current, (str, int, float, bool)): + continue + else: + raise IncrementalThreadError( + "snapshot must contain only plain JSON containers and scalar values" + ) + ''' + if source.count(old_source) != 1: + raise SystemExit('unexpected plain-container guard source state') + source_path.write_text(source.replace(old_source, new_source), encoding='utf-8') + + test_path = Path('tests/test_incremental_snapshot.py') + tests = test_path.read_text(encoding='utf-8') + import_anchor = 'import pytest\n\n' + if 'import threadweave.incremental as incremental_module\n' not in tests: + if tests.count(import_anchor) != 1: + raise SystemExit('unexpected snapshot test import state') + tests = tests.replace( + import_anchor, + 'import pytest\nimport threadweave.incremental as incremental_module\n\n', + ) + test_name = 'def test_plain_container_guard_rejects_cycles_and_allows_shared_values():' + if test_name in tests: + raise SystemExit('cycle regression test already exists') + tests += ''' + + def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): + """The iterative guard distinguishes cycles from shared acyclic values.""" + mapping_cycle: dict[str, object] = {} + mapping_cycle["self"] = mapping_cycle + with pytest.raises(IncrementalThreadError, match="cyclic"): + IncrementalThreadIndex.restore(mapping_cycle) + + list_cycle: list[object] = [] + list_cycle.append(list_cycle) + nested_cycle = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": list_cycle, + } + with pytest.raises(IncrementalThreadError, match="cyclic"): + IncrementalThreadIndex.restore(nested_cycle) + + shared: list[object] = [{"value": 1}] + incremental_module._require_plain_json_containers( + {"first": shared, "second": shared} + ) + ''' + test_path.write_text(tests, encoding='utf-8') + + changelog_path = Path('CHANGELOG.md') + changelog = changelog_path.read_text(encoding='utf-8') + changelog_anchor = '''## Unreleased + + - Reject dictionary and list subclasses at the incremental snapshot restore + ''' + changelog_replacement = '''## Unreleased + + - Reject cyclic built-in dictionaries and lists at the incremental snapshot + restore boundary without recursion or unbounded traversal. + - Reject dictionary and list subclasses at the incremental snapshot restore + ''' + if changelog.count(changelog_anchor) != 1: + raise SystemExit('unexpected CHANGELOG state') + changelog_path.write_text( + changelog.replace(changelog_anchor, changelog_replacement), + encoding='utf-8', + ) + + docs_path = Path('docs/incremental-threading.md') + docs = docs_path.read_text(encoding='utf-8') + docs_anchor = '''subclasses are rejected before serialization so hostile ``items`` or iterator + overrides cannot execute inside the restore boundary. + ''' + docs_replacement = '''subclasses are rejected before serialization so hostile ``items`` or iterator + overrides cannot execute inside the restore boundary. Cyclic built-in containers + are detected with an iterative active-path guard and fail without recursion or + unbounded traversal; repeated acyclic references remain valid JSON input. + ''' + if docs.count(docs_anchor) != 1: + raise SystemExit('unexpected incremental documentation state') + docs_path.write_text( + docs.replace(docs_anchor, docs_replacement), + encoding='utf-8', + ) + PY - name: Verify the fixed repository state shell: bash @@ -75,7 +207,7 @@ jobs: run: | set -euo pipefail pytest -q tests/test_incremental_snapshot.py \ - -k 'cyclic_plain_containers or plain_container_guard' + -k plain_container_guard_rejects_cycles ruff check . python -m compileall -q src tests scripts benchmarks python -m doctest \ From 57219ee96be965821b7083e0aee5528e07ce519f Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 5 Aug 2026 13:13:38 +0000 Subject: [PATCH 62/97] fix: reject cyclic incremental snapshots iteratively --- CHANGELOG.md | 2 + docs/incremental-threading.md | 4 +- src/threadweave/incremental.py | 22 ++++-- tests/test_incremental_snapshot.py | 28 ++++++++ tools/pr20-cycle-fix.patch | 105 ----------------------------- tools/pr20-cycle-test.patch | 62 ----------------- 6 files changed, 49 insertions(+), 174 deletions(-) delete mode 100644 tools/pr20-cycle-fix.patch delete mode 100644 tools/pr20-cycle-test.patch diff --git a/CHANGELOG.md b/CHANGELOG.md index a93e06d..8c8290d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Reject cyclic built-in dictionaries and lists at the incremental snapshot + restore boundary without recursion or unbounded traversal. - Reject dictionary and list subclasses at the incremental snapshot restore boundary before JSON encoding can invoke untrusted iterator overrides. - Serialize every read and write on one `IncrementalThreadIndex` with a diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index d72b48d..16f129a 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -202,7 +202,9 @@ keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicod and configured record or byte limits through `IncrementalThreadError` before publishing state. Only built-in JSON dictionaries and lists are accepted; container subclasses are rejected before serialization so hostile ``items`` or iterator -overrides cannot execute inside the restore boundary. +overrides cannot execute inside the restore boundary. Cyclic built-in containers +are detected with an iterative active-path guard and fail without recursion or +unbounded traversal; repeated acyclic references remain valid JSON input. ## Correctness and operational boundaries diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index d5a851b..aed1af8 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -805,13 +805,23 @@ def _require_plain_json_containers(value: object) -> None: values. Requiring exact container types prevents untrusted ``items`` or iterator overrides from executing inside the restore boundary. """ - pending = [value] + pending = [(value, False)] + active_containers: set[int] = set() while pending: - current = pending.pop() - if type(current) is dict: - pending.extend(dict.values(current)) - elif type(current) is list: - pending.extend(current) + current, exiting = pending.pop() + if type(current) in {dict, list}: + identity = id(current) + if exiting: + active_containers.remove(identity) + continue + if identity in active_containers: + raise IncrementalThreadError( + "snapshot must not contain cyclic JSON containers" + ) + active_containers.add(identity) + pending.append((current, True)) + children = dict.values(current) if type(current) is dict else current + pending.extend((child, False) for child in children) elif current is None or isinstance(current, (str, int, float, bool)): continue else: diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py index 9020bd2..4eafaec 100644 --- a/tests/test_incremental_snapshot.py +++ b/tests/test_incremental_snapshot.py @@ -8,6 +8,7 @@ from datetime import datetime, timezone import pytest +import threadweave.incremental as incremental_module from threadweave import ( IncrementalThreadError, @@ -327,3 +328,30 @@ def test_restore_rejects_container_subclasses_before_they_execute(): } with pytest.raises(IncrementalThreadError, match="plain JSON containers"): IncrementalThreadIndex.restore(nested) + + +def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): + """The iterative guard distinguishes cycles from shared acyclic values.""" + mapping_cycle: dict[str, object] = {} + mapping_cycle["self"] = mapping_cycle + with pytest.raises(IncrementalThreadError, match="cyclic"): + IncrementalThreadIndex.restore(mapping_cycle) + + list_cycle: list[object] = [] + list_cycle.append(list_cycle) + nested_cycle = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": list_cycle, + } + with pytest.raises(IncrementalThreadError, match="cyclic"): + IncrementalThreadIndex.restore(nested_cycle) + + shared: list[object] = [{"value": 1}] + incremental_module._require_plain_json_containers( + {"first": shared, "second": shared} + ) diff --git a/tools/pr20-cycle-fix.patch b/tools/pr20-cycle-fix.patch deleted file mode 100644 index 119ecc9..0000000 --- a/tools/pr20-cycle-fix.patch +++ /dev/null @@ -1,105 +0,0 @@ -diff --git a/CHANGELOG.md b/CHANGELOG.md -index a93e06d..52a8987 100644 ---- a/CHANGELOG.md -+++ b/CHANGELOG.md -@@ -6,6 +6,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - - ## Unreleased - -+- Reject cyclic built-in dictionaries and lists at the incremental snapshot -+ restore boundary without recursion or unbounded traversal. - - Reject dictionary and list subclasses at the incremental snapshot restore - boundary before JSON encoding can invoke untrusted iterator overrides. - - Serialize every read and write on one `IncrementalThreadIndex` with a -diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md -index d72b48d..e19d7f2 100644 ---- a/docs/incremental-threading.md -+++ b/docs/incremental-threading.md -@@ -202,7 +202,9 @@ integers. Restore rejects unknown versions, extra or missing fields, duplicate - keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, - and configured record or byte limits through `IncrementalThreadError` before - publishing state. Only built-in JSON dictionaries and lists are accepted; container --subclasses are rejected before serialization so hostile ``items`` or iterator --overrides cannot execute inside the restore boundary. -+subclasses are rejected before serialization so hostile ``items`` or iterator -+overrides cannot execute inside the restore boundary. Cyclic built-in containers -+are detected with an iterative active-path guard and fail without recursion or -+unbounded traversal; repeated acyclic references remain valid JSON input. - - ## Correctness and operational boundaries - -diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py -index d5a851b..c91fcb6 100644 ---- a/src/threadweave/incremental.py -+++ b/src/threadweave/incremental.py -@@ -804,15 +804,29 @@ def _require_plain_json_containers(value: object) -> None: - values. Requiring exact container types prevents untrusted ``items`` or - iterator overrides from executing inside the restore boundary. - """ -- pending = [value] -+ pending = [(value, False)] -+ active_containers: set[int] = set() - while pending: -- current = pending.pop() -- if type(current) is dict: -- pending.extend(dict.values(current)) -- elif type(current) is list: -- pending.extend(current) -+ current, exiting = pending.pop() -+ if type(current) in {dict, list}: -+ identity = id(current) -+ if exiting: -+ active_containers.remove(identity) -+ continue -+ if identity in active_containers: -+ raise IncrementalThreadError( -+ "snapshot must not contain cyclic JSON containers" -+ ) -+ active_containers.add(identity) -+ pending.append((current, True)) -+ children = ( -+ dict.values(current) -+ if type(current) is dict -+ else current -+ ) -+ pending.extend((child, False) for child in children) - elif current is None or isinstance(current, (str, int, float, bool)): - continue - else: -diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py -index 0ef4ea1..b5e0c47 100644 ---- a/tests/test_incremental_snapshot.py -+++ b/tests/test_incremental_snapshot.py -@@ -8,6 +8,7 @@ from copy import deepcopy - from datetime import datetime, timezone - - import pytest -+import threadweave.incremental as incremental_module - - from threadweave import ( - IncrementalThreadError, -@@ -372,3 +373,24 @@ def test_restore_rejects_cyclic_plain_containers_without_hanging(): - check=False, - ) - assert result.returncode == 0, result.stderr -+ -+ -+def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): -+ """The iterative guard distinguishes cycles from repeated acyclic values.""" -+ mapping_cycle: dict[str, object] = {} -+ mapping_cycle["self"] = mapping_cycle -+ with pytest.raises(IncrementalThreadError, match="cyclic"): -+ incremental_module._require_plain_json_containers(mapping_cycle) -+ -+ list_cycle: list[object] = [] -+ list_cycle.append(list_cycle) -+ with pytest.raises(IncrementalThreadError, match="cyclic"): -+ incremental_module._require_plain_json_containers(list_cycle) -+ -+ shared: list[object] = [{"value": 1}] -+ incremental_module._require_plain_json_containers( -+ { -+ "first": shared, -+ "second": shared, -+ } -+ ) diff --git a/tools/pr20-cycle-test.patch b/tools/pr20-cycle-test.patch deleted file mode 100644 index 9d24563..0000000 --- a/tools/pr20-cycle-test.patch +++ /dev/null @@ -1,62 +0,0 @@ -diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py -index 9020bd2..0ef4ea1 100644 ---- a/tests/test_incremental_snapshot.py -+++ b/tests/test_incremental_snapshot.py -@@ -3,6 +3,7 @@ - from __future__ import annotations - - import json -+import subprocess - import sys - from copy import deepcopy - from datetime import datetime, timezone -@@ -334,3 +335,40 @@ def test_restore_rejects_container_subclasses_before_they_execute(): - } - with pytest.raises(IncrementalThreadError, match="plain JSON containers"): - IncrementalThreadIndex.restore(nested) -+ -+ -+def test_restore_rejects_cyclic_plain_containers_without_hanging(): -+ """Cyclic built-in containers fail quickly instead of looping forever.""" -+ script = r""" -+from threadweave import IncrementalThreadError, IncrementalThreadIndex -+ -+mapping_cycle = { -+ "schema_version": 1, -+ "version": 0, -+ "options": { -+ "group_by_subject": False, -+ "sort_by_sent_date": False, -+ }, -+ "records": [], -+} -+mapping_cycle["self"] = mapping_cycle -+ -+list_cycle = [] -+list_cycle.append(list_cycle) -+nested_cycle = { -+ "schema_version": 1, -+ "version": 0, -+ "options": { -+ "group_by_subject": False, -+ "sort_by_sent_date": False, -+ }, -+ "records": list_cycle, -+} -+ -+for snapshot in (mapping_cycle, nested_cycle): -+ try: -+ IncrementalThreadIndex.restore(snapshot) -+ except IncrementalThreadError as error: -+ assert "cyclic" in str(error), str(error) -+ else: -+ raise AssertionError("cyclic snapshot was accepted") -+""" -+ result = subprocess.run( -+ [sys.executable, "-c", script], -+ capture_output=True, -+ text=True, -+ timeout=2, -+ check=False, -+ ) -+ assert result.returncode == 0, result.stderr From cf1cecbf1ca827eadd3fa4ff0b3dfb2a5cab71da Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 5 Aug 2026 22:14:22 +0900 Subject: [PATCH 63/97] ci: remove completed cyclic snapshot verifier --- .../workflows/apply-pr20-cycle-hardening.yml | 255 ------------------ 1 file changed, 255 deletions(-) delete mode 100644 .github/workflows/apply-pr20-cycle-hardening.yml diff --git a/.github/workflows/apply-pr20-cycle-hardening.yml b/.github/workflows/apply-pr20-cycle-hardening.yml deleted file mode 100644 index d8b76fd..0000000 --- a/.github/workflows/apply-pr20-cycle-hardening.yml +++ /dev/null @@ -1,255 +0,0 @@ -name: Apply PR 20 cyclic snapshot hardening - -on: - pull_request: - types: [synchronize] - paths: - - .github/workflows/apply-pr20-cycle-hardening.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-cycle-hardening - cancel-in-progress: false - -jobs: - red-green-verify: - if: >- - github.event.pull_request.number == 20 && - github.event.pull_request.head.repo.full_name == github.repository && - github.event.pull_request.head.ref == 'feature/incremental-thread-index' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Prove the cyclic-container regression fails before the fix - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - test "$(git hash-object src/threadweave/incremental.py)" = \ - 'd5a851bd835bfa1b6acc00b7ac20608e6abeb4f9' - set +e - timeout 2s python - <<'PY' - from threadweave import IncrementalThreadIndex - - cycle = {} - cycle["self"] = cycle - IncrementalThreadIndex.restore(cycle) - PY - red_status="$?" - set -e - if [ "$red_status" -ne 124 ]; then - echo "Expected the pre-fix restore call to time out; status=$red_status" >&2 - exit 1 - fi - echo 'Observed the expected pre-fix non-terminating traversal.' \ - >>"$GITHUB_STEP_SUMMARY" - - - name: Apply the minimal iterative cycle detector and regression tests - shell: bash - run: | - set -euo pipefail - test "$(git hash-object tests/test_incremental_snapshot.py)" = \ - '9020bd2953811ebcb814cc0aeaf7f1035e4992e2' - test "$(git hash-object CHANGELOG.md)" = \ - 'a93e06da23a04c9a4d484c8478283be075771aa8' - test "$(git hash-object docs/incremental-threading.md)" = \ - 'd72b48dbeeb4f64a4c78b58151775a010b1b4d1a' - python - <<'PY' - from pathlib import Path - - source_path = Path('src/threadweave/incremental.py') - source = source_path.read_text(encoding='utf-8') - old_source = ''' pending = [value] - while pending: - current = pending.pop() - if type(current) is dict: - pending.extend(dict.values(current)) - elif type(current) is list: - pending.extend(current) - elif current is None or isinstance(current, (str, int, float, bool)): - continue - else: - raise IncrementalThreadError( - "snapshot must contain only plain JSON containers and scalar values" - ) - ''' - new_source = ''' pending = [(value, False)] - active_containers: set[int] = set() - while pending: - current, exiting = pending.pop() - if type(current) in {dict, list}: - identity = id(current) - if exiting: - active_containers.remove(identity) - continue - if identity in active_containers: - raise IncrementalThreadError( - "snapshot must not contain cyclic JSON containers" - ) - active_containers.add(identity) - pending.append((current, True)) - children = dict.values(current) if type(current) is dict else current - pending.extend((child, False) for child in children) - elif current is None or isinstance(current, (str, int, float, bool)): - continue - else: - raise IncrementalThreadError( - "snapshot must contain only plain JSON containers and scalar values" - ) - ''' - if source.count(old_source) != 1: - raise SystemExit('unexpected plain-container guard source state') - source_path.write_text(source.replace(old_source, new_source), encoding='utf-8') - - test_path = Path('tests/test_incremental_snapshot.py') - tests = test_path.read_text(encoding='utf-8') - import_anchor = 'import pytest\n\n' - if 'import threadweave.incremental as incremental_module\n' not in tests: - if tests.count(import_anchor) != 1: - raise SystemExit('unexpected snapshot test import state') - tests = tests.replace( - import_anchor, - 'import pytest\nimport threadweave.incremental as incremental_module\n\n', - ) - test_name = 'def test_plain_container_guard_rejects_cycles_and_allows_shared_values():' - if test_name in tests: - raise SystemExit('cycle regression test already exists') - tests += ''' - - def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): - """The iterative guard distinguishes cycles from shared acyclic values.""" - mapping_cycle: dict[str, object] = {} - mapping_cycle["self"] = mapping_cycle - with pytest.raises(IncrementalThreadError, match="cyclic"): - IncrementalThreadIndex.restore(mapping_cycle) - - list_cycle: list[object] = [] - list_cycle.append(list_cycle) - nested_cycle = { - "schema_version": 1, - "version": 0, - "options": { - "group_by_subject": False, - "sort_by_sent_date": False, - }, - "records": list_cycle, - } - with pytest.raises(IncrementalThreadError, match="cyclic"): - IncrementalThreadIndex.restore(nested_cycle) - - shared: list[object] = [{"value": 1}] - incremental_module._require_plain_json_containers( - {"first": shared, "second": shared} - ) - ''' - test_path.write_text(tests, encoding='utf-8') - - changelog_path = Path('CHANGELOG.md') - changelog = changelog_path.read_text(encoding='utf-8') - changelog_anchor = '''## Unreleased - - - Reject dictionary and list subclasses at the incremental snapshot restore - ''' - changelog_replacement = '''## Unreleased - - - Reject cyclic built-in dictionaries and lists at the incremental snapshot - restore boundary without recursion or unbounded traversal. - - Reject dictionary and list subclasses at the incremental snapshot restore - ''' - if changelog.count(changelog_anchor) != 1: - raise SystemExit('unexpected CHANGELOG state') - changelog_path.write_text( - changelog.replace(changelog_anchor, changelog_replacement), - encoding='utf-8', - ) - - docs_path = Path('docs/incremental-threading.md') - docs = docs_path.read_text(encoding='utf-8') - docs_anchor = '''subclasses are rejected before serialization so hostile ``items`` or iterator - overrides cannot execute inside the restore boundary. - ''' - docs_replacement = '''subclasses are rejected before serialization so hostile ``items`` or iterator - overrides cannot execute inside the restore boundary. Cyclic built-in containers - are detected with an iterative active-path guard and fail without recursion or - unbounded traversal; repeated acyclic references remain valid JSON input. - ''' - if docs.count(docs_anchor) != 1: - raise SystemExit('unexpected incremental documentation state') - docs_path.write_text( - docs.replace(docs_anchor, docs_replacement), - encoding='utf-8', - ) - PY - - - name: Verify the fixed repository state - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - pytest -q tests/test_incremental_snapshot.py \ - -k plain_container_guard_rejects_cycles - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit the verified product update - shell: bash - run: | - set -euo pipefail - rm tools/pr20-cycle-test.patch tools/pr20-cycle-fix.patch - rmdir tools 2>/dev/null || true - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add -A - if git diff --cached --name-only | grep -q '^\.github/workflows/'; then - echo 'Refusing to push workflow changes with the repository token.' >&2 - exit 1 - fi - git diff --cached --check - git commit -m 'fix: reject cyclic incremental snapshots iteratively' - git push origin HEAD:feature/incremental-thread-index From e77e146494b09e2d16a1a330a6a4f3b2c3d6a1ca Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 11:57:42 +0900 Subject: [PATCH 64/97] test: stage bounded snapshot expansion hardening --- tools/pr20-snapshot-expansion-hardening.patch | 266 ++++++++++++++++++ 1 file changed, 266 insertions(+) create mode 100644 tools/pr20-snapshot-expansion-hardening.patch diff --git a/tools/pr20-snapshot-expansion-hardening.patch b/tools/pr20-snapshot-expansion-hardening.patch new file mode 100644 index 0000000..2e9fa6d --- /dev/null +++ b/tools/pr20-snapshot-expansion-hardening.patch @@ -0,0 +1,266 @@ +--- a/src/threadweave/incremental.py ++++ b/src/threadweave/incremental.py +@@ -799,14 +799,17 @@ + + + def _require_plain_json_containers(value: object) -> None: +- """Reject executable container subclasses before JSON serialization. +- +- JSON-decoded state consists of built-in dictionaries, lists, and scalar +- values. Requiring exact container types prevents untrusted ``items`` or +- iterator overrides from executing inside the restore boundary. ++ """Reject executable, cyclic, or aliased containers before serialization. ++ ++ JSON decoding produces a tree of built-in dictionaries, lists, string ++ keys, and scalar values. Requiring exact runtime types prevents attacker- ++ controlled iteration, comparison, or scalar subclasses from executing, ++ while rejecting repeated container identities prevents a compact Python ++ object graph from expanding exponentially during JSON encoding. + """ + pending = [(value, False)] + active_containers: set[int] = set() ++ seen_containers: set[int] = set() + while pending: + current, exiting = pending.pop() + if type(current) in {dict, list}: +@@ -818,11 +821,22 @@ + raise IncrementalThreadError( + "snapshot must not contain cyclic JSON containers" + ) ++ if identity in seen_containers: ++ raise IncrementalThreadError( ++ "snapshot must not contain reused JSON container objects" ++ ) ++ if type(current) is dict and any( ++ type(key) is not str for key in dict.keys(current) ++ ): ++ raise IncrementalThreadError( ++ "snapshot object keys must be plain strings" ++ ) ++ seen_containers.add(identity) + active_containers.add(identity) + pending.append((current, True)) + children = dict.values(current) if type(current) is dict else current + pending.extend((child, False) for child in children) +- elif current is None or isinstance(current, (str, int, float, bool)): ++ elif current is None or type(current) in {str, int, float, bool}: + continue + else: + raise IncrementalThreadError( +@@ -830,21 +844,27 @@ + ) + + +-def _snapshot_json_bytes(value: object) -> bytes: +- """Serialize a snapshot canonically or raise a bounded domain error.""" ++def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: ++ """Return canonical UTF-8 size while stopping at the configured byte limit.""" ++ encoder = json.JSONEncoder( ++ ensure_ascii=False, ++ allow_nan=False, ++ sort_keys=True, ++ separators=(",", ":"), ++ ) ++ encoded_bytes = 0 + try: +- encoded = json.dumps( +- value, +- ensure_ascii=False, +- allow_nan=False, +- sort_keys=True, +- separators=(",", ":"), +- ) +- return encoded.encode("utf-8") ++ for chunk in encoder.iterencode(value): ++ encoded_bytes += len(chunk.encode("utf-8")) ++ if encoded_bytes > maximum_bytes: ++ raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") ++ except IncrementalThreadError: ++ raise + except (RecursionError, TypeError, UnicodeError, ValueError) as error: + raise IncrementalThreadError( + "snapshot must contain only JSON-safe values" + ) from error ++ return encoded_bytes + + + def _required_fields(value: Mapping[str, object], expected: set[str], name: str) -> None: +@@ -1224,8 +1244,7 @@ + }, + "records": records, + } +- if len(_snapshot_json_bytes(snapshot)) > self._max_snapshot_bytes: +- raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") ++ _bounded_snapshot_json_size(snapshot, self._max_snapshot_bytes) + return snapshot + + @classmethod +@@ -1248,8 +1267,7 @@ + if not isinstance(snapshot, Mapping): + raise IncrementalThreadError("snapshot must be a mapping") + _require_plain_json_containers(snapshot) +- if len(_snapshot_json_bytes(snapshot)) > max_bytes: +- raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") ++ _bounded_snapshot_json_size(snapshot, max_bytes) + _required_fields( + snapshot, + {"schema_version", "version", "options", "records"}, +--- a/tests/test_incremental_snapshot.py ++++ b/tests/test_incremental_snapshot.py +@@ -301,6 +301,14 @@ + raise RuntimeError("hostile list iteration") + + ++class _HostileString(str): ++ """String subclass whose comparison must not run during sorted encoding.""" ++ ++ def __lt__(self, _other: object) -> bool: ++ """Raise if JSON key sorting reaches attacker-controlled comparison.""" ++ raise RuntimeError("hostile string comparison") ++ ++ + def test_restore_rejects_container_subclasses_before_they_execute(): + """Only plain JSON containers may enter the untrusted snapshot decoder.""" + root = _HostileDictionary( +@@ -330,8 +338,22 @@ + IncrementalThreadIndex.restore(nested) + + +-def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): +- """The iterative guard distinguishes cycles from shared acyclic values.""" ++def test_plain_json_guard_rejects_executable_key_and_scalar_subclasses(): ++ """Sorted JSON encoding cannot invoke attacker-controlled scalar methods.""" ++ hostile_key = _HostileString("unexpected") ++ with pytest.raises(IncrementalThreadError, match="plain strings"): ++ incremental_module._require_plain_json_containers( ++ {hostile_key: "value", "schema_version": 1} ++ ) ++ ++ with pytest.raises(IncrementalThreadError, match="scalar values"): ++ incremental_module._require_plain_json_containers( ++ {"value": _HostileString("hostile")} ++ ) ++ ++ ++def test_plain_container_guard_rejects_cycles_and_reused_containers(): ++ """JSON snapshots are trees: cycles and shared container identities fail closed.""" + mapping_cycle: dict[str, object] = {} + mapping_cycle["self"] = mapping_cycle + with pytest.raises(IncrementalThreadError, match="cyclic"): +@@ -352,6 +374,45 @@ + IncrementalThreadIndex.restore(nested_cycle) + + shared: list[object] = [{"value": 1}] +- incremental_module._require_plain_json_containers( +- {"first": shared, "second": shared} +- ) ++ with pytest.raises(IncrementalThreadError, match="reused"): ++ incremental_module._require_plain_json_containers( ++ {"first": shared, "second": shared} ++ ) ++ ++ compact_graph: object = [] ++ for _ in range(28): ++ compact_graph = [compact_graph, compact_graph] ++ hostile_snapshot = { ++ "schema_version": 1, ++ "version": 0, ++ "options": { ++ "group_by_subject": False, ++ "sort_by_sent_date": False, ++ }, ++ "records": compact_graph, ++ } ++ with pytest.raises(IncrementalThreadError, match="reused"): ++ IncrementalThreadIndex.restore(hostile_snapshot) ++ ++ ++def test_snapshot_size_validation_stops_at_the_utf8_limit(monkeypatch): ++ """Byte-limit enforcement stops the incremental encoder before later chunks.""" ++ later_chunks_requested: list[bool] = [] ++ ++ class _ChunkedEncoder: ++ """Yield one oversized UTF-8 chunk, then fail if iteration continues.""" ++ ++ def __init__(self, **_options: object) -> None: ++ """Accept the production encoder options used by the helper.""" ++ ++ def iterencode(self, _value: object): ++ """Yield one four-byte chunk and record any forbidden second request.""" ++ yield '"é"' ++ later_chunks_requested.append(True) ++ raise AssertionError("encoder continued after the configured byte limit") ++ ++ monkeypatch.setattr(incremental_module.json, "JSONEncoder", _ChunkedEncoder) ++ ++ with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): ++ incremental_module._bounded_snapshot_json_size({}, 3) ++ assert later_chunks_requested == [] +--- a/docs/incremental-threading.md ++++ b/docs/incremental-threading.md +@@ -200,11 +200,16 @@ + integers. Restore rejects unknown versions, extra or missing fields, duplicate + keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, + and configured record or byte limits through `IncrementalThreadError` before +-publishing state. Only built-in JSON dictionaries and lists are accepted; container +-subclasses are rejected before serialization so hostile ``items`` or iterator +-overrides cannot execute inside the restore boundary. Cyclic built-in containers +-are detected with an iterative active-path guard and fail without recursion or +-unbounded traversal; repeated acyclic references remain valid JSON input. ++publishing state. Only built-in JSON dictionaries, lists, scalar values, and plain ++string object keys are accepted. Container, key, and scalar subclasses are rejected ++before serialization so hostile iteration, comparison, or conversion methods cannot ++execute inside the restore boundary. RFC 8259 represents JSON as nested arrays, ++objects, and scalar values rather than an identity-bearing object graph. ThreadWeave ++therefore rejects cyclic or reused built-in container identities with an iterative ++active-path and seen-object guard. This prevents a compact Python DAG from expanding ++exponentially during encoding. The configured UTF-8 byte limit is counted from ++incremental encoder chunks and aborts without materializing a second complete JSON ++string or byte array. + + ## Correctness and operational boundaries + +@@ -224,6 +229,9 @@ + + ## References + ++Bray, T. (Ed.). (2017). *The JavaScript Object Notation (JSON) data interchange ++format* (RFC 8259). RFC Editor. https://doi.org/10.17487/RFC8259 ++ + Gondwana, B. (2018). *IMAP extension for object identifiers* (RFC 8474). RFC + Editor. https://doi.org/10.17487/RFC8474 + +--- a/README.md ++++ b/README.md +@@ -220,8 +220,10 @@ + reported explicitly. `roots` returns a defensive structural copy, so callers may + traverse or edit that graph without corrupting reusable index state; payload objects + remain caller-owned references. Internal sent-date tie-break positions never become +-public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads and reject +-unknown, malformed, or oversized input. See ++public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject ++unknown, malformed, cyclic, aliased, or oversized input, and stop UTF-8 size ++validation at the configured byte limit without building a second full serialized ++copy. See + [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, + identity, snapshot, complexity, and RFC boundaries. + +--- a/CHANGELOG.md ++++ b/CHANGELOG.md +@@ -6,10 +6,14 @@ + + ## Unreleased + ++- Bound incremental snapshot size checks to streaming UTF-8 encoding and reject ++ reused container identities so compact Python object graphs cannot trigger ++ exponential JSON expansion or a second full serialized copy in memory. + - Reject cyclic built-in dictionaries and lists at the incremental snapshot + restore boundary without recursion or unbounded traversal. +-- Reject dictionary and list subclasses at the incremental snapshot restore +- boundary before JSON encoding can invoke untrusted iterator overrides. ++- Reject container and scalar subclasses plus non-plain-string object keys at ++ the incremental snapshot restore boundary before sorted JSON encoding can ++ invoke attacker-controlled iteration or comparison methods. + - Serialize every read and write on one `IncrementalThreadIndex` with a + process-local reentrant lock so same-version concurrent writers yield one + commit and one explicit conflict, while readers observe only committed state. From 504d6f21c88a3407dd7953dc4ce9aefd5405c14f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 12:00:13 +0900 Subject: [PATCH 65/97] ci: verify bounded incremental snapshot expansion --- ...pply-pr20-snapshot-expansion-hardening.yml | 178 ++++++++++++++++++ 1 file changed, 178 insertions(+) create mode 100644 .github/workflows/apply-pr20-snapshot-expansion-hardening.yml diff --git a/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml b/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml new file mode 100644 index 0000000..30dffad --- /dev/null +++ b/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml @@ -0,0 +1,178 @@ +name: Apply PR 20 snapshot expansion hardening + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/apply-pr20-snapshot-expansion-hardening.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-snapshot-expansion-hardening + cancel-in-progress: false + +jobs: + red-green-verify: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Verify the reviewed baseline and patch + shell: bash + run: | + set -euo pipefail + test "$(git hash-object src/threadweave/incremental.py)" = \ + 'aed1af878c829704cdc3a9f6e539b53f5cb50aa5' + test "$(git hash-object tests/test_incremental_snapshot.py)" = \ + '4eafaec5ed305886bc44805ac8791b052d8ee8ce' + test "$(git hash-object docs/incremental-threading.md)" = \ + '16f129afb1f53571004fdb76aa1241d88c0e1f4c' + test "$(git hash-object README.md)" = \ + '8f839a24cb608dea83ac9197a669c3cff4f57c74' + test "$(git hash-object CHANGELOG.md)" = \ + '8c8290df7f076e872f1597c28134fc45bcc85242' + test "$(git hash-object tools/pr20-snapshot-expansion-hardening.patch)" = \ + '2e9fa6de4b5da5c344efaaecaf4ea5d45b80564b' + git apply --check tools/pr20-snapshot-expansion-hardening.patch + + - name: Prove the regressions fail before implementation + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + + set +e + timeout 2s python - <<'PY' + from threadweave import IncrementalThreadIndex + + compact_graph = [] + for _ in range(28): + compact_graph = [compact_graph, compact_graph] + IncrementalThreadIndex.restore( + { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": compact_graph, + }, + max_snapshot_bytes=128, + ) + PY + expansion_status="$?" + set -e + if [ "$expansion_status" -ne 124 ]; then + echo "Expected compact graph expansion to time out; status=$expansion_status" >&2 + exit 1 + fi + + set +e + python - <<'PY' >"$RUNNER_TEMP/hostile-key-red.log" 2>&1 + from threadweave import IncrementalThreadIndex + + class HostileString(str): + def __lt__(self, _other): + raise RuntimeError("hostile string comparison executed") + + IncrementalThreadIndex.restore( + { + HostileString("unexpected"): 1, + "schema_version": 1, + } + ) + PY + hostile_status="$?" + set -e + cat "$RUNNER_TEMP/hostile-key-red.log" + if [ "$hostile_status" -eq 0 ] || \ + ! grep -q 'hostile string comparison executed' \ + "$RUNNER_TEMP/hostile-key-red.log"; then + echo 'Expected sorted encoding to execute the hostile comparison.' >&2 + exit 1 + fi + echo 'Observed both expected pre-fix failures.' >>"$GITHUB_STEP_SUMMARY" + + - name: Apply the bounded snapshot fix + shell: bash + run: | + set -euo pipefail + git apply tools/pr20-snapshot-expansion-hardening.patch + rm tools/pr20-snapshot-expansion-hardening.patch + rmdir tools 2>/dev/null || true + git diff --check + + - name: Verify the fixed repository state + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + pytest -q tests/test_incremental_snapshot.py \ + -k 'plain_json_guard or reused_containers or size_validation' + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit the verified product update + shell: bash + run: | + set -euo pipefail + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add \ + src/threadweave/incremental.py \ + tests/test_incremental_snapshot.py \ + docs/incremental-threading.md \ + README.md \ + CHANGELOG.md + git add -u tools/pr20-snapshot-expansion-hardening.patch + git diff --cached --check + git commit -m 'fix: bound incremental snapshot expansion' + git push origin HEAD:feature/incremental-thread-index From f3b3fb8073d7d2c09ef739e6c4a54d50a1dee2ce Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 12:06:37 +0900 Subject: [PATCH 66/97] ci: correct snapshot hardening red proof --- .github/workflows/apply-pr20-snapshot-expansion-hardening.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml b/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml index 30dffad..159739c 100644 --- a/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml +++ b/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml @@ -98,8 +98,8 @@ jobs: IncrementalThreadIndex.restore( { - HostileString("unexpected"): 1, "schema_version": 1, + HostileString("unexpected"): 1, } ) PY From 6b69d0a859d47396cb6faeb3366c34ff6b482aa9 Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 6 Aug 2026 03:07:26 +0000 Subject: [PATCH 67/97] fix: bound incremental snapshot expansion --- CHANGELOG.md | 8 +- README.md | 6 +- docs/incremental-threading.md | 18 +- src/threadweave/incremental.py | 56 ++-- tests/test_incremental_snapshot.py | 71 ++++- tools/pr20-snapshot-expansion-hardening.patch | 266 ------------------ 6 files changed, 126 insertions(+), 299 deletions(-) delete mode 100644 tools/pr20-snapshot-expansion-hardening.patch diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c8290d..6ae633f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,10 +6,14 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Bound incremental snapshot size checks to streaming UTF-8 encoding and reject + reused container identities so compact Python object graphs cannot trigger + exponential JSON expansion or a second full serialized copy in memory. - Reject cyclic built-in dictionaries and lists at the incremental snapshot restore boundary without recursion or unbounded traversal. -- Reject dictionary and list subclasses at the incremental snapshot restore - boundary before JSON encoding can invoke untrusted iterator overrides. +- Reject container and scalar subclasses plus non-plain-string object keys at + the incremental snapshot restore boundary before sorted JSON encoding can + invoke attacker-controlled iteration or comparison methods. - Serialize every read and write on one `IncrementalThreadIndex` with a process-local reentrant lock so same-version concurrent writers yield one commit and one explicit conflict, while readers observe only committed state. diff --git a/README.md b/README.md index 8f839a2..c42e0a1 100644 --- a/README.md +++ b/README.md @@ -220,8 +220,10 @@ full-rebuild parity is the correctness oracle. Structural merges and splits are reported explicitly. `roots` returns a defensive structural copy, so callers may traverse or edit that graph without corrupting reusable index state; payload objects remain caller-owned references. Internal sent-date tie-break positions never become -public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads and reject -unknown, malformed, or oversized input. See +public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject +unknown, malformed, cyclic, aliased, or oversized input, and stop UTF-8 size +validation at the configured byte limit without building a second full serialized +copy. See [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, identity, snapshot, complexity, and RFC boundaries. diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index 16f129a..cc6d7e5 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -200,11 +200,16 @@ or ISO-8601 datetime representation. Schema versions must be exact non-boolean integers. Restore rejects unknown versions, extra or missing fields, duplicate keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, and configured record or byte limits through `IncrementalThreadError` before -publishing state. Only built-in JSON dictionaries and lists are accepted; container -subclasses are rejected before serialization so hostile ``items`` or iterator -overrides cannot execute inside the restore boundary. Cyclic built-in containers -are detected with an iterative active-path guard and fail without recursion or -unbounded traversal; repeated acyclic references remain valid JSON input. +publishing state. Only built-in JSON dictionaries, lists, scalar values, and plain +string object keys are accepted. Container, key, and scalar subclasses are rejected +before serialization so hostile iteration, comparison, or conversion methods cannot +execute inside the restore boundary. RFC 8259 represents JSON as nested arrays, +objects, and scalar values rather than an identity-bearing object graph. ThreadWeave +therefore rejects cyclic or reused built-in container identities with an iterative +active-path and seen-object guard. This prevents a compact Python DAG from expanding +exponentially during encoding. The configured UTF-8 byte limit is counted from +incremental encoder chunks and aborts without materializing a second complete JSON +string or byte array. ## Correctness and operational boundaries @@ -224,6 +229,9 @@ therefore reported separately from delta application. ## References +Bray, T. (Ed.). (2017). *The JavaScript Object Notation (JSON) data interchange +format* (RFC 8259). RFC Editor. https://doi.org/10.17487/RFC8259 + Gondwana, B. (2018). *IMAP extension for object identifiers* (RFC 8474). RFC Editor. https://doi.org/10.17487/RFC8474 diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index aed1af8..c388fc3 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -799,14 +799,17 @@ def _decoded_date(value: object, name: str) -> str | datetime | None: def _require_plain_json_containers(value: object) -> None: - """Reject executable container subclasses before JSON serialization. + """Reject executable, cyclic, or aliased containers before serialization. - JSON-decoded state consists of built-in dictionaries, lists, and scalar - values. Requiring exact container types prevents untrusted ``items`` or - iterator overrides from executing inside the restore boundary. + JSON decoding produces a tree of built-in dictionaries, lists, string + keys, and scalar values. Requiring exact runtime types prevents attacker- + controlled iteration, comparison, or scalar subclasses from executing, + while rejecting repeated container identities prevents a compact Python + object graph from expanding exponentially during JSON encoding. """ pending = [(value, False)] active_containers: set[int] = set() + seen_containers: set[int] = set() while pending: current, exiting = pending.pop() if type(current) in {dict, list}: @@ -818,11 +821,22 @@ def _require_plain_json_containers(value: object) -> None: raise IncrementalThreadError( "snapshot must not contain cyclic JSON containers" ) + if identity in seen_containers: + raise IncrementalThreadError( + "snapshot must not contain reused JSON container objects" + ) + if type(current) is dict and any( + type(key) is not str for key in dict.keys(current) + ): + raise IncrementalThreadError( + "snapshot object keys must be plain strings" + ) + seen_containers.add(identity) active_containers.add(identity) pending.append((current, True)) children = dict.values(current) if type(current) is dict else current pending.extend((child, False) for child in children) - elif current is None or isinstance(current, (str, int, float, bool)): + elif current is None or type(current) in {str, int, float, bool}: continue else: raise IncrementalThreadError( @@ -830,21 +844,27 @@ def _require_plain_json_containers(value: object) -> None: ) -def _snapshot_json_bytes(value: object) -> bytes: - """Serialize a snapshot canonically or raise a bounded domain error.""" +def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: + """Return canonical UTF-8 size while stopping at the configured byte limit.""" + encoder = json.JSONEncoder( + ensure_ascii=False, + allow_nan=False, + sort_keys=True, + separators=(",", ":"), + ) + encoded_bytes = 0 try: - encoded = json.dumps( - value, - ensure_ascii=False, - allow_nan=False, - sort_keys=True, - separators=(",", ":"), - ) - return encoded.encode("utf-8") + for chunk in encoder.iterencode(value): + encoded_bytes += len(chunk.encode("utf-8")) + if encoded_bytes > maximum_bytes: + raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") + except IncrementalThreadError: + raise except (RecursionError, TypeError, UnicodeError, ValueError) as error: raise IncrementalThreadError( "snapshot must contain only JSON-safe values" ) from error + return encoded_bytes def _required_fields(value: Mapping[str, object], expected: set[str], name: str) -> None: @@ -1224,8 +1244,7 @@ def _snapshot_locked(self) -> dict[str, object]: }, "records": records, } - if len(_snapshot_json_bytes(snapshot)) > self._max_snapshot_bytes: - raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") + _bounded_snapshot_json_size(snapshot, self._max_snapshot_bytes) return snapshot @classmethod @@ -1248,8 +1267,7 @@ def restore( if not isinstance(snapshot, Mapping): raise IncrementalThreadError("snapshot must be a mapping") _require_plain_json_containers(snapshot) - if len(_snapshot_json_bytes(snapshot)) > max_bytes: - raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") + _bounded_snapshot_json_size(snapshot, max_bytes) _required_fields( snapshot, {"schema_version", "version", "options", "records"}, diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py index 4eafaec..1303ed9 100644 --- a/tests/test_incremental_snapshot.py +++ b/tests/test_incremental_snapshot.py @@ -301,6 +301,14 @@ def __iter__(self): raise RuntimeError("hostile list iteration") +class _HostileString(str): + """String subclass whose comparison must not run during sorted encoding.""" + + def __lt__(self, _other: object) -> bool: + """Raise if JSON key sorting reaches attacker-controlled comparison.""" + raise RuntimeError("hostile string comparison") + + def test_restore_rejects_container_subclasses_before_they_execute(): """Only plain JSON containers may enter the untrusted snapshot decoder.""" root = _HostileDictionary( @@ -330,8 +338,22 @@ def test_restore_rejects_container_subclasses_before_they_execute(): IncrementalThreadIndex.restore(nested) -def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): - """The iterative guard distinguishes cycles from shared acyclic values.""" +def test_plain_json_guard_rejects_executable_key_and_scalar_subclasses(): + """Sorted JSON encoding cannot invoke attacker-controlled scalar methods.""" + hostile_key = _HostileString("unexpected") + with pytest.raises(IncrementalThreadError, match="plain strings"): + incremental_module._require_plain_json_containers( + {hostile_key: "value", "schema_version": 1} + ) + + with pytest.raises(IncrementalThreadError, match="scalar values"): + incremental_module._require_plain_json_containers( + {"value": _HostileString("hostile")} + ) + + +def test_plain_container_guard_rejects_cycles_and_reused_containers(): + """JSON snapshots are trees: cycles and shared container identities fail closed.""" mapping_cycle: dict[str, object] = {} mapping_cycle["self"] = mapping_cycle with pytest.raises(IncrementalThreadError, match="cyclic"): @@ -352,6 +374,45 @@ def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): IncrementalThreadIndex.restore(nested_cycle) shared: list[object] = [{"value": 1}] - incremental_module._require_plain_json_containers( - {"first": shared, "second": shared} - ) + with pytest.raises(IncrementalThreadError, match="reused"): + incremental_module._require_plain_json_containers( + {"first": shared, "second": shared} + ) + + compact_graph: object = [] + for _ in range(28): + compact_graph = [compact_graph, compact_graph] + hostile_snapshot = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": compact_graph, + } + with pytest.raises(IncrementalThreadError, match="reused"): + IncrementalThreadIndex.restore(hostile_snapshot) + + +def test_snapshot_size_validation_stops_at_the_utf8_limit(monkeypatch): + """Byte-limit enforcement stops the incremental encoder before later chunks.""" + later_chunks_requested: list[bool] = [] + + class _ChunkedEncoder: + """Yield one oversized UTF-8 chunk, then fail if iteration continues.""" + + def __init__(self, **_options: object) -> None: + """Accept the production encoder options used by the helper.""" + + def iterencode(self, _value: object): + """Yield one four-byte chunk and record any forbidden second request.""" + yield '"é"' + later_chunks_requested.append(True) + raise AssertionError("encoder continued after the configured byte limit") + + monkeypatch.setattr(incremental_module.json, "JSONEncoder", _ChunkedEncoder) + + with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): + incremental_module._bounded_snapshot_json_size({}, 3) + assert later_chunks_requested == [] diff --git a/tools/pr20-snapshot-expansion-hardening.patch b/tools/pr20-snapshot-expansion-hardening.patch deleted file mode 100644 index 2e9fa6d..0000000 --- a/tools/pr20-snapshot-expansion-hardening.patch +++ /dev/null @@ -1,266 +0,0 @@ ---- a/src/threadweave/incremental.py -+++ b/src/threadweave/incremental.py -@@ -799,14 +799,17 @@ - - - def _require_plain_json_containers(value: object) -> None: -- """Reject executable container subclasses before JSON serialization. -- -- JSON-decoded state consists of built-in dictionaries, lists, and scalar -- values. Requiring exact container types prevents untrusted ``items`` or -- iterator overrides from executing inside the restore boundary. -+ """Reject executable, cyclic, or aliased containers before serialization. -+ -+ JSON decoding produces a tree of built-in dictionaries, lists, string -+ keys, and scalar values. Requiring exact runtime types prevents attacker- -+ controlled iteration, comparison, or scalar subclasses from executing, -+ while rejecting repeated container identities prevents a compact Python -+ object graph from expanding exponentially during JSON encoding. - """ - pending = [(value, False)] - active_containers: set[int] = set() -+ seen_containers: set[int] = set() - while pending: - current, exiting = pending.pop() - if type(current) in {dict, list}: -@@ -818,11 +821,22 @@ - raise IncrementalThreadError( - "snapshot must not contain cyclic JSON containers" - ) -+ if identity in seen_containers: -+ raise IncrementalThreadError( -+ "snapshot must not contain reused JSON container objects" -+ ) -+ if type(current) is dict and any( -+ type(key) is not str for key in dict.keys(current) -+ ): -+ raise IncrementalThreadError( -+ "snapshot object keys must be plain strings" -+ ) -+ seen_containers.add(identity) - active_containers.add(identity) - pending.append((current, True)) - children = dict.values(current) if type(current) is dict else current - pending.extend((child, False) for child in children) -- elif current is None or isinstance(current, (str, int, float, bool)): -+ elif current is None or type(current) in {str, int, float, bool}: - continue - else: - raise IncrementalThreadError( -@@ -830,21 +844,27 @@ - ) - - --def _snapshot_json_bytes(value: object) -> bytes: -- """Serialize a snapshot canonically or raise a bounded domain error.""" -+def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: -+ """Return canonical UTF-8 size while stopping at the configured byte limit.""" -+ encoder = json.JSONEncoder( -+ ensure_ascii=False, -+ allow_nan=False, -+ sort_keys=True, -+ separators=(",", ":"), -+ ) -+ encoded_bytes = 0 - try: -- encoded = json.dumps( -- value, -- ensure_ascii=False, -- allow_nan=False, -- sort_keys=True, -- separators=(",", ":"), -- ) -- return encoded.encode("utf-8") -+ for chunk in encoder.iterencode(value): -+ encoded_bytes += len(chunk.encode("utf-8")) -+ if encoded_bytes > maximum_bytes: -+ raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") -+ except IncrementalThreadError: -+ raise - except (RecursionError, TypeError, UnicodeError, ValueError) as error: - raise IncrementalThreadError( - "snapshot must contain only JSON-safe values" - ) from error -+ return encoded_bytes - - - def _required_fields(value: Mapping[str, object], expected: set[str], name: str) -> None: -@@ -1224,8 +1244,7 @@ - }, - "records": records, - } -- if len(_snapshot_json_bytes(snapshot)) > self._max_snapshot_bytes: -- raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") -+ _bounded_snapshot_json_size(snapshot, self._max_snapshot_bytes) - return snapshot - - @classmethod -@@ -1248,8 +1267,7 @@ - if not isinstance(snapshot, Mapping): - raise IncrementalThreadError("snapshot must be a mapping") - _require_plain_json_containers(snapshot) -- if len(_snapshot_json_bytes(snapshot)) > max_bytes: -- raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") -+ _bounded_snapshot_json_size(snapshot, max_bytes) - _required_fields( - snapshot, - {"schema_version", "version", "options", "records"}, ---- a/tests/test_incremental_snapshot.py -+++ b/tests/test_incremental_snapshot.py -@@ -301,6 +301,14 @@ - raise RuntimeError("hostile list iteration") - - -+class _HostileString(str): -+ """String subclass whose comparison must not run during sorted encoding.""" -+ -+ def __lt__(self, _other: object) -> bool: -+ """Raise if JSON key sorting reaches attacker-controlled comparison.""" -+ raise RuntimeError("hostile string comparison") -+ -+ - def test_restore_rejects_container_subclasses_before_they_execute(): - """Only plain JSON containers may enter the untrusted snapshot decoder.""" - root = _HostileDictionary( -@@ -330,8 +338,22 @@ - IncrementalThreadIndex.restore(nested) - - --def test_plain_container_guard_rejects_cycles_and_allows_shared_values(): -- """The iterative guard distinguishes cycles from shared acyclic values.""" -+def test_plain_json_guard_rejects_executable_key_and_scalar_subclasses(): -+ """Sorted JSON encoding cannot invoke attacker-controlled scalar methods.""" -+ hostile_key = _HostileString("unexpected") -+ with pytest.raises(IncrementalThreadError, match="plain strings"): -+ incremental_module._require_plain_json_containers( -+ {hostile_key: "value", "schema_version": 1} -+ ) -+ -+ with pytest.raises(IncrementalThreadError, match="scalar values"): -+ incremental_module._require_plain_json_containers( -+ {"value": _HostileString("hostile")} -+ ) -+ -+ -+def test_plain_container_guard_rejects_cycles_and_reused_containers(): -+ """JSON snapshots are trees: cycles and shared container identities fail closed.""" - mapping_cycle: dict[str, object] = {} - mapping_cycle["self"] = mapping_cycle - with pytest.raises(IncrementalThreadError, match="cyclic"): -@@ -352,6 +374,45 @@ - IncrementalThreadIndex.restore(nested_cycle) - - shared: list[object] = [{"value": 1}] -- incremental_module._require_plain_json_containers( -- {"first": shared, "second": shared} -- ) -+ with pytest.raises(IncrementalThreadError, match="reused"): -+ incremental_module._require_plain_json_containers( -+ {"first": shared, "second": shared} -+ ) -+ -+ compact_graph: object = [] -+ for _ in range(28): -+ compact_graph = [compact_graph, compact_graph] -+ hostile_snapshot = { -+ "schema_version": 1, -+ "version": 0, -+ "options": { -+ "group_by_subject": False, -+ "sort_by_sent_date": False, -+ }, -+ "records": compact_graph, -+ } -+ with pytest.raises(IncrementalThreadError, match="reused"): -+ IncrementalThreadIndex.restore(hostile_snapshot) -+ -+ -+def test_snapshot_size_validation_stops_at_the_utf8_limit(monkeypatch): -+ """Byte-limit enforcement stops the incremental encoder before later chunks.""" -+ later_chunks_requested: list[bool] = [] -+ -+ class _ChunkedEncoder: -+ """Yield one oversized UTF-8 chunk, then fail if iteration continues.""" -+ -+ def __init__(self, **_options: object) -> None: -+ """Accept the production encoder options used by the helper.""" -+ -+ def iterencode(self, _value: object): -+ """Yield one four-byte chunk and record any forbidden second request.""" -+ yield '"é"' -+ later_chunks_requested.append(True) -+ raise AssertionError("encoder continued after the configured byte limit") -+ -+ monkeypatch.setattr(incremental_module.json, "JSONEncoder", _ChunkedEncoder) -+ -+ with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): -+ incremental_module._bounded_snapshot_json_size({}, 3) -+ assert later_chunks_requested == [] ---- a/docs/incremental-threading.md -+++ b/docs/incremental-threading.md -@@ -200,11 +200,16 @@ - integers. Restore rejects unknown versions, extra or missing fields, duplicate - keys, malformed types, invalid external IDs, hostile nesting, unencodable Unicode, - and configured record or byte limits through `IncrementalThreadError` before --publishing state. Only built-in JSON dictionaries and lists are accepted; container --subclasses are rejected before serialization so hostile ``items`` or iterator --overrides cannot execute inside the restore boundary. Cyclic built-in containers --are detected with an iterative active-path guard and fail without recursion or --unbounded traversal; repeated acyclic references remain valid JSON input. -+publishing state. Only built-in JSON dictionaries, lists, scalar values, and plain -+string object keys are accepted. Container, key, and scalar subclasses are rejected -+before serialization so hostile iteration, comparison, or conversion methods cannot -+execute inside the restore boundary. RFC 8259 represents JSON as nested arrays, -+objects, and scalar values rather than an identity-bearing object graph. ThreadWeave -+therefore rejects cyclic or reused built-in container identities with an iterative -+active-path and seen-object guard. This prevents a compact Python DAG from expanding -+exponentially during encoding. The configured UTF-8 byte limit is counted from -+incremental encoder chunks and aborts without materializing a second complete JSON -+string or byte array. - - ## Correctness and operational boundaries - -@@ -224,6 +229,9 @@ - - ## References - -+Bray, T. (Ed.). (2017). *The JavaScript Object Notation (JSON) data interchange -+format* (RFC 8259). RFC Editor. https://doi.org/10.17487/RFC8259 -+ - Gondwana, B. (2018). *IMAP extension for object identifiers* (RFC 8474). RFC - Editor. https://doi.org/10.17487/RFC8474 - ---- a/README.md -+++ b/README.md -@@ -220,8 +220,10 @@ - reported explicitly. `roots` returns a defensive structural copy, so callers may - traverse or edit that graph without corrupting reusable index state; payload objects - remain caller-owned references. Internal sent-date tie-break positions never become --public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads and reject --unknown, malformed, or oversized input. See -+public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject -+unknown, malformed, cyclic, aliased, or oversized input, and stop UTF-8 size -+validation at the configured byte limit without building a second full serialized -+copy. See - [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, - identity, snapshot, complexity, and RFC boundaries. - ---- a/CHANGELOG.md -+++ b/CHANGELOG.md -@@ -6,10 +6,14 @@ - - ## Unreleased - -+- Bound incremental snapshot size checks to streaming UTF-8 encoding and reject -+ reused container identities so compact Python object graphs cannot trigger -+ exponential JSON expansion or a second full serialized copy in memory. - - Reject cyclic built-in dictionaries and lists at the incremental snapshot - restore boundary without recursion or unbounded traversal. --- Reject dictionary and list subclasses at the incremental snapshot restore -- boundary before JSON encoding can invoke untrusted iterator overrides. -+- Reject container and scalar subclasses plus non-plain-string object keys at -+ the incremental snapshot restore boundary before sorted JSON encoding can -+ invoke attacker-controlled iteration or comparison methods. - - Serialize every read and write on one `IncrementalThreadIndex` with a - process-local reentrant lock so same-version concurrent writers yield one - commit and one explicit conflict, while readers observe only committed state. From aa5c199b706c6023f64eeab396ea94d6ad69dfe3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 12:12:34 +0900 Subject: [PATCH 68/97] ci: remove completed snapshot expansion verifier --- ...pply-pr20-snapshot-expansion-hardening.yml | 178 ------------------ 1 file changed, 178 deletions(-) delete mode 100644 .github/workflows/apply-pr20-snapshot-expansion-hardening.yml diff --git a/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml b/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml deleted file mode 100644 index 159739c..0000000 --- a/.github/workflows/apply-pr20-snapshot-expansion-hardening.yml +++ /dev/null @@ -1,178 +0,0 @@ -name: Apply PR 20 snapshot expansion hardening - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - .github/workflows/apply-pr20-snapshot-expansion-hardening.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-snapshot-expansion-hardening - cancel-in-progress: false - -jobs: - red-green-verify: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Verify the reviewed baseline and patch - shell: bash - run: | - set -euo pipefail - test "$(git hash-object src/threadweave/incremental.py)" = \ - 'aed1af878c829704cdc3a9f6e539b53f5cb50aa5' - test "$(git hash-object tests/test_incremental_snapshot.py)" = \ - '4eafaec5ed305886bc44805ac8791b052d8ee8ce' - test "$(git hash-object docs/incremental-threading.md)" = \ - '16f129afb1f53571004fdb76aa1241d88c0e1f4c' - test "$(git hash-object README.md)" = \ - '8f839a24cb608dea83ac9197a669c3cff4f57c74' - test "$(git hash-object CHANGELOG.md)" = \ - '8c8290df7f076e872f1597c28134fc45bcc85242' - test "$(git hash-object tools/pr20-snapshot-expansion-hardening.patch)" = \ - '2e9fa6de4b5da5c344efaaecaf4ea5d45b80564b' - git apply --check tools/pr20-snapshot-expansion-hardening.patch - - - name: Prove the regressions fail before implementation - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - - set +e - timeout 2s python - <<'PY' - from threadweave import IncrementalThreadIndex - - compact_graph = [] - for _ in range(28): - compact_graph = [compact_graph, compact_graph] - IncrementalThreadIndex.restore( - { - "schema_version": 1, - "version": 0, - "options": { - "group_by_subject": False, - "sort_by_sent_date": False, - }, - "records": compact_graph, - }, - max_snapshot_bytes=128, - ) - PY - expansion_status="$?" - set -e - if [ "$expansion_status" -ne 124 ]; then - echo "Expected compact graph expansion to time out; status=$expansion_status" >&2 - exit 1 - fi - - set +e - python - <<'PY' >"$RUNNER_TEMP/hostile-key-red.log" 2>&1 - from threadweave import IncrementalThreadIndex - - class HostileString(str): - def __lt__(self, _other): - raise RuntimeError("hostile string comparison executed") - - IncrementalThreadIndex.restore( - { - "schema_version": 1, - HostileString("unexpected"): 1, - } - ) - PY - hostile_status="$?" - set -e - cat "$RUNNER_TEMP/hostile-key-red.log" - if [ "$hostile_status" -eq 0 ] || \ - ! grep -q 'hostile string comparison executed' \ - "$RUNNER_TEMP/hostile-key-red.log"; then - echo 'Expected sorted encoding to execute the hostile comparison.' >&2 - exit 1 - fi - echo 'Observed both expected pre-fix failures.' >>"$GITHUB_STEP_SUMMARY" - - - name: Apply the bounded snapshot fix - shell: bash - run: | - set -euo pipefail - git apply tools/pr20-snapshot-expansion-hardening.patch - rm tools/pr20-snapshot-expansion-hardening.patch - rmdir tools 2>/dev/null || true - git diff --check - - - name: Verify the fixed repository state - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - pytest -q tests/test_incremental_snapshot.py \ - -k 'plain_json_guard or reused_containers or size_validation' - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit the verified product update - shell: bash - run: | - set -euo pipefail - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add \ - src/threadweave/incremental.py \ - tests/test_incremental_snapshot.py \ - docs/incremental-threading.md \ - README.md \ - CHANGELOG.md - git add -u tools/pr20-snapshot-expansion-hardening.patch - git diff --cached --check - git commit -m 'fix: bound incremental snapshot expansion' - git push origin HEAD:feature/incremental-thread-index From eb301a8a9e3a486b2350182e220ea6f736b04d95 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 12:24:14 +0900 Subject: [PATCH 69/97] ci: apply PR 20 ordering review fix --- .../apply-pr20-ordering-review-fix.yml | 127 ++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 .github/workflows/apply-pr20-ordering-review-fix.yml diff --git a/.github/workflows/apply-pr20-ordering-review-fix.yml b/.github/workflows/apply-pr20-ordering-review-fix.yml new file mode 100644 index 0000000..eefc643 --- /dev/null +++ b/.github/workflows/apply-pr20-ordering-review-fix.yml @@ -0,0 +1,127 @@ +name: Apply PR 20 ordering review fix + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/apply-pr20-ordering-review-fix.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-ordering-review-fix + cancel-in-progress: false + +jobs: + verify-and-apply: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Apply the review fix against the exact blob + shell: bash + run: | + set -euo pipefail + test "$(git hash-object tests/test_incremental_snapshot.py)" = \ + '1303ed9d760a022b093e52ca34b22363e655ed5d' + python - <<'PY' + from pathlib import Path + + path = Path("tests/test_incremental_snapshot.py") + text = path.read_text(encoding="utf-8") + old = '''class _HostileString(str): + """String subclass whose comparison must not run during sorted encoding.""" + + def __lt__(self, _other: object) -> bool: + """Raise if JSON key sorting reaches attacker-controlled comparison.""" + raise RuntimeError("hostile string comparison") + ''' + new = '''class _HostileString(str): + """String subclass whose comparisons must not run during sorted encoding.""" + + def __lt__(self, _other: object) -> bool: + """Reject less-than comparison if sorted encoding reaches this key.""" + raise TypeError("hostile string comparison") + + def __le__(self, _other: object) -> bool: + """Reject less-than-or-equal comparison if encoding reaches this key.""" + raise TypeError("hostile string comparison") + + def __gt__(self, _other: object) -> bool: + """Reject greater-than comparison if sorted encoding reaches this key.""" + raise TypeError("hostile string comparison") + + def __ge__(self, _other: object) -> bool: + """Reject greater-than-or-equal comparison if encoding reaches this key.""" + raise TypeError("hostile string comparison") + ''' + if text.count(old) != 1: + raise SystemExit("unexpected hostile-string test state") + path.write_text(text.replace(old, new), encoding="utf-8") + PY + git diff --check + + - name: Verify the complete repository + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + pytest -q tests/test_incremental_snapshot.py + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit the verified review fix + shell: bash + run: | + set -euo pipefail + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add tests/test_incremental_snapshot.py + git diff --cached --check + git commit -m 'test: complete hostile ordering protocol' + git push origin HEAD:feature/incremental-thread-index From 00626815f406e5c8f920d8deef0e90021ebdd315 Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 6 Aug 2026 03:24:47 +0000 Subject: [PATCH 70/97] test: complete hostile ordering protocol --- tests/test_incremental_snapshot.py | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py index 1303ed9..d8d0db6 100644 --- a/tests/test_incremental_snapshot.py +++ b/tests/test_incremental_snapshot.py @@ -302,11 +302,23 @@ def __iter__(self): class _HostileString(str): - """String subclass whose comparison must not run during sorted encoding.""" + """String subclass whose comparisons must not run during sorted encoding.""" def __lt__(self, _other: object) -> bool: - """Raise if JSON key sorting reaches attacker-controlled comparison.""" - raise RuntimeError("hostile string comparison") + """Reject less-than comparison if sorted encoding reaches this key.""" + raise TypeError("hostile string comparison") + + def __le__(self, _other: object) -> bool: + """Reject less-than-or-equal comparison if encoding reaches this key.""" + raise TypeError("hostile string comparison") + + def __gt__(self, _other: object) -> bool: + """Reject greater-than comparison if sorted encoding reaches this key.""" + raise TypeError("hostile string comparison") + + def __ge__(self, _other: object) -> bool: + """Reject greater-than-or-equal comparison if encoding reaches this key.""" + raise TypeError("hostile string comparison") def test_restore_rejects_container_subclasses_before_they_execute(): From 7a0d2f0fe4c9205db47236f28f3cdf86b29de4c0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 12:25:42 +0900 Subject: [PATCH 71/97] ci: remove completed ordering review verifier --- .../apply-pr20-ordering-review-fix.yml | 127 ------------------ 1 file changed, 127 deletions(-) delete mode 100644 .github/workflows/apply-pr20-ordering-review-fix.yml diff --git a/.github/workflows/apply-pr20-ordering-review-fix.yml b/.github/workflows/apply-pr20-ordering-review-fix.yml deleted file mode 100644 index eefc643..0000000 --- a/.github/workflows/apply-pr20-ordering-review-fix.yml +++ /dev/null @@ -1,127 +0,0 @@ -name: Apply PR 20 ordering review fix - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - .github/workflows/apply-pr20-ordering-review-fix.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-ordering-review-fix - cancel-in-progress: false - -jobs: - verify-and-apply: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Apply the review fix against the exact blob - shell: bash - run: | - set -euo pipefail - test "$(git hash-object tests/test_incremental_snapshot.py)" = \ - '1303ed9d760a022b093e52ca34b22363e655ed5d' - python - <<'PY' - from pathlib import Path - - path = Path("tests/test_incremental_snapshot.py") - text = path.read_text(encoding="utf-8") - old = '''class _HostileString(str): - """String subclass whose comparison must not run during sorted encoding.""" - - def __lt__(self, _other: object) -> bool: - """Raise if JSON key sorting reaches attacker-controlled comparison.""" - raise RuntimeError("hostile string comparison") - ''' - new = '''class _HostileString(str): - """String subclass whose comparisons must not run during sorted encoding.""" - - def __lt__(self, _other: object) -> bool: - """Reject less-than comparison if sorted encoding reaches this key.""" - raise TypeError("hostile string comparison") - - def __le__(self, _other: object) -> bool: - """Reject less-than-or-equal comparison if encoding reaches this key.""" - raise TypeError("hostile string comparison") - - def __gt__(self, _other: object) -> bool: - """Reject greater-than comparison if sorted encoding reaches this key.""" - raise TypeError("hostile string comparison") - - def __ge__(self, _other: object) -> bool: - """Reject greater-than-or-equal comparison if encoding reaches this key.""" - raise TypeError("hostile string comparison") - ''' - if text.count(old) != 1: - raise SystemExit("unexpected hostile-string test state") - path.write_text(text.replace(old, new), encoding="utf-8") - PY - git diff --check - - - name: Verify the complete repository - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - pytest -q tests/test_incremental_snapshot.py - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit the verified review fix - shell: bash - run: | - set -euo pipefail - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add tests/test_incremental_snapshot.py - git diff --cached --check - git commit -m 'test: complete hostile ordering protocol' - git push origin HEAD:feature/incremental-thread-index From 856498b3420e2bcef59d41d02a6ff86620f866a6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:20:20 +0900 Subject: [PATCH 72/97] ci: verify allocation-free snapshot UTF-8 sizing --- .../apply-pr20-utf8-size-hardening.yml | 285 ++++++++++++++++++ 1 file changed, 285 insertions(+) create mode 100644 .github/workflows/apply-pr20-utf8-size-hardening.yml diff --git a/.github/workflows/apply-pr20-utf8-size-hardening.yml b/.github/workflows/apply-pr20-utf8-size-hardening.yml new file mode 100644 index 0000000..cd150ee --- /dev/null +++ b/.github/workflows/apply-pr20-utf8-size-hardening.yml @@ -0,0 +1,285 @@ +name: Apply PR 20 allocation-free UTF-8 sizing + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/apply-pr20-utf8-size-hardening.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-utf8-size-hardening + cancel-in-progress: false + +jobs: + red-green-verify: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Verify the reviewed baseline + shell: bash + run: | + set -euo pipefail + test "$(git hash-object src/threadweave/incremental.py)" = \ + 'c388fc3ebaf153e2aa6961627f3852c9deab6c54' + test "$(git hash-object tests/test_incremental_snapshot.py)" = \ + 'd8d0db61e93c84e882e9aa55cf3ba5aadc2153ad' + test "$(git hash-object docs/incremental-threading.md)" = \ + 'cc6d7e5fc06ea5515b1808b246efd8a33d039999' + test "$(git hash-object README.md)" = \ + 'c42e0a1189caf1482363fadba1eb2b5293754451' + test "$(git hash-object CHANGELOG.md)" = \ + '6ae633fb70f24f7886b2a813bff6a864613b42f8' + + - name: Prove the full-chunk bytes copy occurs before the fix + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + import threadweave.incremental as incremental + + class NoEncodeString(str): + def encode(self, encoding='utf-8', errors='strict'): + raise AssertionError((encoding, errors)) + + class SingleChunkEncoder: + def __init__(self, **_options): + pass + + def iterencode(self, _value): + yield NoEncodeString('"aé€😀"') + + incremental.json.JSONEncoder = SingleChunkEncoder + try: + incremental._bounded_snapshot_json_size({}, 100) + except AssertionError: + pass + else: + raise SystemExit('expected the pre-fix helper to call str.encode') + PYCODE + echo 'Observed the expected pre-fix full-chunk encode call.' \ + >>"$GITHUB_STEP_SUMMARY" + + - name: Apply the allocation-free UTF-8 sizing patch + shell: bash + run: | + set -euo pipefail + git apply - <<'PATCH' + --- a/src/threadweave/incremental.py + +++ b/src/threadweave/incremental.py + @@ -844,6 +844,36 @@ + ) + + + +def _bounded_utf8_size(value: str, maximum_bytes: int) -> int: + + """Count UTF-8 bytes up to a limit without allocating an encoded copy.""" + + if str.isascii(value): + + return str.__len__(value) + + + + encoded_bytes = 0 + + for index in range(str.__len__(value)): + + code_point = ord(str.__getitem__(value, index)) + + if code_point <= 0x7F: + + width = 1 + + elif code_point <= 0x7FF: + + width = 2 + + elif 0xD800 <= code_point <= 0xDFFF: + + raise UnicodeEncodeError( + + "utf-8", + + value, + + index, + + index + 1, + + "surrogates not allowed", + + ) + + elif code_point <= 0xFFFF: + + width = 3 + + else: + + width = 4 + + encoded_bytes += width + + if encoded_bytes > maximum_bytes: + + break + + return encoded_bytes + + + + + def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: + """Return canonical UTF-8 size while stopping at the configured byte limit.""" + encoder = json.JSONEncoder( + @@ -855,7 +885,10 @@ + encoded_bytes = 0 + try: + for chunk in encoder.iterencode(value): + - encoded_bytes += len(chunk.encode("utf-8")) + + encoded_bytes += _bounded_utf8_size( + + chunk, + + maximum_bytes - encoded_bytes, + + ) + if encoded_bytes > maximum_bytes: + raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") + except IncrementalThreadError: + --- a/tests/test_incremental_snapshot.py + +++ b/tests/test_incremental_snapshot.py + @@ -407,6 +407,35 @@ + IncrementalThreadIndex.restore(hostile_snapshot) + + + +def test_utf8_size_counting_avoids_chunk_encoding(monkeypatch): + + """Large JSON chunks are counted without allocating an encoded bytes copy.""" + + + + class _NoEncodeString(str): + + """Raise if the size checker calls ``str.encode`` on an encoder chunk.""" + + + + def encode( + + self, + + encoding: str = "utf-8", + + errors: str = "strict", + + ) -> bytes: + + """Expose accidental full-chunk byte allocation as a test failure.""" + + raise AssertionError((encoding, errors)) + + + + class _SingleChunkEncoder: + + """Yield one non-ASCII JSON chunk whose ``encode`` method is forbidden.""" + + + + def __init__(self, **_options: object) -> None: + + """Accept the production encoder configuration.""" + + + + def iterencode(self, _value: object): + + """Yield one representative built JSON string chunk.""" + + yield _NoEncodeString('"aé€😀"') + + + + monkeypatch.setattr(incremental_module.json, "JSONEncoder", _SingleChunkEncoder) + + + + assert incremental_module._bounded_snapshot_json_size({}, 100) == 12 + + + + + def test_snapshot_size_validation_stops_at_the_utf8_limit(monkeypatch): + """Byte-limit enforcement stops the incremental encoder before later chunks.""" + later_chunks_requested: list[bool] = [] + --- a/docs/incremental-threading.md + +++ b/docs/incremental-threading.md + @@ -208,8 +208,9 @@ + therefore rejects cyclic or reused built-in container identities with an iterative + active-path and seen-object guard. This prevents a compact Python DAG from expanding + exponentially during encoding. The configured UTF-8 byte limit is counted from + -incremental encoder chunks and aborts without materializing a second complete JSON + -string or byte array. + +incremental encoder chunks with allocation-free code-point width accounting. The + +check aborts without calling ``str.encode`` on a complete chunk or materializing a + +second complete JSON string or byte array. + + ## Correctness and operational boundaries + + --- a/README.md + +++ b/README.md + @@ -221,9 +221,9 @@ + traverse or edit that graph without corrupting reusable index state; payload objects + remain caller-owned references. Internal sent-date tie-break positions never become + public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject + -unknown, malformed, cyclic, aliased, or oversized input, and stop UTF-8 size + -validation at the configured byte limit without building a second full serialized + -copy. See + +unknown, malformed, cyclic, aliased, or oversized input, and count UTF-8 + +code-point width to stop at the configured byte limit without encoding a second + +full bytes copy. See + [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, + identity, snapshot, complexity, and RFC boundaries. + + --- a/CHANGELOG.md + +++ b/CHANGELOG.md + @@ -6,9 +6,10 @@ + + ## Unreleased + + -- Bound incremental snapshot size checks to streaming UTF-8 encoding and reject + - reused container identities so compact Python object graphs cannot trigger + - exponential JSON expansion or a second full serialized copy in memory. + +- Bound incremental snapshot size checks to streaming UTF-8 code-point + + accounting without encoding each JSON chunk into a second bytes object, and + + reject reused container identities so compact Python object graphs cannot + + trigger exponential JSON expansion. + - Reject cyclic built-in dictionaries and lists at the incremental snapshot + restore boundary without recursion or unbounded traversal. + - Reject container and scalar subclasses plus non-plain-string object keys at + PATCH + git diff --check + + - name: Verify focused snapshot behavior + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + pytest -q tests/test_incremental_snapshot.py \ + -k 'utf8_size_counting or size_validation or unencodable_unicode' + + - name: Verify the complete repository + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit the verified product update + shell: bash + run: | + set -euo pipefail + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add \ + src/threadweave/incremental.py \ + tests/test_incremental_snapshot.py \ + docs/incremental-threading.md \ + README.md \ + CHANGELOG.md + git diff --cached --check + git commit -m 'perf: avoid snapshot UTF-8 bytes duplication' + git push origin HEAD:feature/incremental-thread-index From b38cd8443c74c809496da4d6b347f4365e25f8b3 Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 6 Aug 2026 06:20:53 +0000 Subject: [PATCH 73/97] perf: avoid snapshot UTF-8 bytes duplication --- CHANGELOG.md | 7 +++--- README.md | 6 ++--- docs/incremental-threading.md | 5 +++-- src/threadweave/incremental.py | 35 +++++++++++++++++++++++++++++- tests/test_incremental_snapshot.py | 29 +++++++++++++++++++++++++ 5 files changed, 73 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ae633f..3d43ba9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,9 +6,10 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased -- Bound incremental snapshot size checks to streaming UTF-8 encoding and reject - reused container identities so compact Python object graphs cannot trigger - exponential JSON expansion or a second full serialized copy in memory. +- Bound incremental snapshot size checks to streaming UTF-8 code-point + accounting without encoding each JSON chunk into a second bytes object, and + reject reused container identities so compact Python object graphs cannot + trigger exponential JSON expansion. - Reject cyclic built-in dictionaries and lists at the incremental snapshot restore boundary without recursion or unbounded traversal. - Reject container and scalar subclasses plus non-plain-string object keys at diff --git a/README.md b/README.md index c42e0a1..b6d0c42 100644 --- a/README.md +++ b/README.md @@ -221,9 +221,9 @@ reported explicitly. `roots` returns a defensive structural copy, so callers may traverse or edit that graph without corrupting reusable index state; payload objects remain caller-owned references. Internal sent-date tie-break positions never become public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject -unknown, malformed, cyclic, aliased, or oversized input, and stop UTF-8 size -validation at the configured byte limit without building a second full serialized -copy. See +unknown, malformed, cyclic, aliased, or oversized input, and count UTF-8 +code-point width to stop at the configured byte limit without encoding a second +full bytes copy. See [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, identity, snapshot, complexity, and RFC boundaries. diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index cc6d7e5..35529ec 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -208,8 +208,9 @@ objects, and scalar values rather than an identity-bearing object graph. ThreadW therefore rejects cyclic or reused built-in container identities with an iterative active-path and seen-object guard. This prevents a compact Python DAG from expanding exponentially during encoding. The configured UTF-8 byte limit is counted from -incremental encoder chunks and aborts without materializing a second complete JSON -string or byte array. +incremental encoder chunks with allocation-free code-point width accounting. The +check aborts without calling ``str.encode`` on a complete chunk or materializing a +second complete JSON string or byte array. ## Correctness and operational boundaries diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index c388fc3..53c5aec 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -844,6 +844,36 @@ def _require_plain_json_containers(value: object) -> None: ) +def _bounded_utf8_size(value: str, maximum_bytes: int) -> int: + """Count UTF-8 bytes up to a limit without allocating an encoded copy.""" + if str.isascii(value): + return str.__len__(value) + + encoded_bytes = 0 + for index in range(str.__len__(value)): + code_point = ord(str.__getitem__(value, index)) + if code_point <= 0x7F: + width = 1 + elif code_point <= 0x7FF: + width = 2 + elif 0xD800 <= code_point <= 0xDFFF: + raise UnicodeEncodeError( + "utf-8", + value, + index, + index + 1, + "surrogates not allowed", + ) + elif code_point <= 0xFFFF: + width = 3 + else: + width = 4 + encoded_bytes += width + if encoded_bytes > maximum_bytes: + break + return encoded_bytes + + def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: """Return canonical UTF-8 size while stopping at the configured byte limit.""" encoder = json.JSONEncoder( @@ -855,7 +885,10 @@ def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: encoded_bytes = 0 try: for chunk in encoder.iterencode(value): - encoded_bytes += len(chunk.encode("utf-8")) + encoded_bytes += _bounded_utf8_size( + chunk, + maximum_bytes - encoded_bytes, + ) if encoded_bytes > maximum_bytes: raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") except IncrementalThreadError: diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py index d8d0db6..3179e04 100644 --- a/tests/test_incremental_snapshot.py +++ b/tests/test_incremental_snapshot.py @@ -407,6 +407,35 @@ def test_plain_container_guard_rejects_cycles_and_reused_containers(): IncrementalThreadIndex.restore(hostile_snapshot) +def test_utf8_size_counting_avoids_chunk_encoding(monkeypatch): + """Large JSON chunks are counted without allocating an encoded bytes copy.""" + + class _NoEncodeString(str): + """Raise if the size checker calls ``str.encode`` on an encoder chunk.""" + + def encode( + self, + encoding: str = "utf-8", + errors: str = "strict", + ) -> bytes: + """Expose accidental full-chunk byte allocation as a test failure.""" + raise AssertionError((encoding, errors)) + + class _SingleChunkEncoder: + """Yield one non-ASCII JSON chunk whose ``encode`` method is forbidden.""" + + def __init__(self, **_options: object) -> None: + """Accept the production encoder configuration.""" + + def iterencode(self, _value: object): + """Yield one representative built JSON string chunk.""" + yield _NoEncodeString('"aé€😀"') + + monkeypatch.setattr(incremental_module.json, "JSONEncoder", _SingleChunkEncoder) + + assert incremental_module._bounded_snapshot_json_size({}, 100) == 12 + + def test_snapshot_size_validation_stops_at_the_utf8_limit(monkeypatch): """Byte-limit enforcement stops the incremental encoder before later chunks.""" later_chunks_requested: list[bool] = [] From 1f159c402484636dbaf87af707ba9560f8eba4cd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:22:36 +0900 Subject: [PATCH 74/97] ci: remove completed UTF-8 sizing verifier --- .../apply-pr20-utf8-size-hardening.yml | 285 ------------------ 1 file changed, 285 deletions(-) delete mode 100644 .github/workflows/apply-pr20-utf8-size-hardening.yml diff --git a/.github/workflows/apply-pr20-utf8-size-hardening.yml b/.github/workflows/apply-pr20-utf8-size-hardening.yml deleted file mode 100644 index cd150ee..0000000 --- a/.github/workflows/apply-pr20-utf8-size-hardening.yml +++ /dev/null @@ -1,285 +0,0 @@ -name: Apply PR 20 allocation-free UTF-8 sizing - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - .github/workflows/apply-pr20-utf8-size-hardening.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-utf8-size-hardening - cancel-in-progress: false - -jobs: - red-green-verify: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Verify the reviewed baseline - shell: bash - run: | - set -euo pipefail - test "$(git hash-object src/threadweave/incremental.py)" = \ - 'c388fc3ebaf153e2aa6961627f3852c9deab6c54' - test "$(git hash-object tests/test_incremental_snapshot.py)" = \ - 'd8d0db61e93c84e882e9aa55cf3ba5aadc2153ad' - test "$(git hash-object docs/incremental-threading.md)" = \ - 'cc6d7e5fc06ea5515b1808b246efd8a33d039999' - test "$(git hash-object README.md)" = \ - 'c42e0a1189caf1482363fadba1eb2b5293754451' - test "$(git hash-object CHANGELOG.md)" = \ - '6ae633fb70f24f7886b2a813bff6a864613b42f8' - - - name: Prove the full-chunk bytes copy occurs before the fix - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - import threadweave.incremental as incremental - - class NoEncodeString(str): - def encode(self, encoding='utf-8', errors='strict'): - raise AssertionError((encoding, errors)) - - class SingleChunkEncoder: - def __init__(self, **_options): - pass - - def iterencode(self, _value): - yield NoEncodeString('"aé€😀"') - - incremental.json.JSONEncoder = SingleChunkEncoder - try: - incremental._bounded_snapshot_json_size({}, 100) - except AssertionError: - pass - else: - raise SystemExit('expected the pre-fix helper to call str.encode') - PYCODE - echo 'Observed the expected pre-fix full-chunk encode call.' \ - >>"$GITHUB_STEP_SUMMARY" - - - name: Apply the allocation-free UTF-8 sizing patch - shell: bash - run: | - set -euo pipefail - git apply - <<'PATCH' - --- a/src/threadweave/incremental.py - +++ b/src/threadweave/incremental.py - @@ -844,6 +844,36 @@ - ) - - - +def _bounded_utf8_size(value: str, maximum_bytes: int) -> int: - + """Count UTF-8 bytes up to a limit without allocating an encoded copy.""" - + if str.isascii(value): - + return str.__len__(value) - + - + encoded_bytes = 0 - + for index in range(str.__len__(value)): - + code_point = ord(str.__getitem__(value, index)) - + if code_point <= 0x7F: - + width = 1 - + elif code_point <= 0x7FF: - + width = 2 - + elif 0xD800 <= code_point <= 0xDFFF: - + raise UnicodeEncodeError( - + "utf-8", - + value, - + index, - + index + 1, - + "surrogates not allowed", - + ) - + elif code_point <= 0xFFFF: - + width = 3 - + else: - + width = 4 - + encoded_bytes += width - + if encoded_bytes > maximum_bytes: - + break - + return encoded_bytes - + - + - def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: - """Return canonical UTF-8 size while stopping at the configured byte limit.""" - encoder = json.JSONEncoder( - @@ -855,7 +885,10 @@ - encoded_bytes = 0 - try: - for chunk in encoder.iterencode(value): - - encoded_bytes += len(chunk.encode("utf-8")) - + encoded_bytes += _bounded_utf8_size( - + chunk, - + maximum_bytes - encoded_bytes, - + ) - if encoded_bytes > maximum_bytes: - raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") - except IncrementalThreadError: - --- a/tests/test_incremental_snapshot.py - +++ b/tests/test_incremental_snapshot.py - @@ -407,6 +407,35 @@ - IncrementalThreadIndex.restore(hostile_snapshot) - - - +def test_utf8_size_counting_avoids_chunk_encoding(monkeypatch): - + """Large JSON chunks are counted without allocating an encoded bytes copy.""" - + - + class _NoEncodeString(str): - + """Raise if the size checker calls ``str.encode`` on an encoder chunk.""" - + - + def encode( - + self, - + encoding: str = "utf-8", - + errors: str = "strict", - + ) -> bytes: - + """Expose accidental full-chunk byte allocation as a test failure.""" - + raise AssertionError((encoding, errors)) - + - + class _SingleChunkEncoder: - + """Yield one non-ASCII JSON chunk whose ``encode`` method is forbidden.""" - + - + def __init__(self, **_options: object) -> None: - + """Accept the production encoder configuration.""" - + - + def iterencode(self, _value: object): - + """Yield one representative built JSON string chunk.""" - + yield _NoEncodeString('"aé€😀"') - + - + monkeypatch.setattr(incremental_module.json, "JSONEncoder", _SingleChunkEncoder) - + - + assert incremental_module._bounded_snapshot_json_size({}, 100) == 12 - + - + - def test_snapshot_size_validation_stops_at_the_utf8_limit(monkeypatch): - """Byte-limit enforcement stops the incremental encoder before later chunks.""" - later_chunks_requested: list[bool] = [] - --- a/docs/incremental-threading.md - +++ b/docs/incremental-threading.md - @@ -208,8 +208,9 @@ - therefore rejects cyclic or reused built-in container identities with an iterative - active-path and seen-object guard. This prevents a compact Python DAG from expanding - exponentially during encoding. The configured UTF-8 byte limit is counted from - -incremental encoder chunks and aborts without materializing a second complete JSON - -string or byte array. - +incremental encoder chunks with allocation-free code-point width accounting. The - +check aborts without calling ``str.encode`` on a complete chunk or materializing a - +second complete JSON string or byte array. - - ## Correctness and operational boundaries - - --- a/README.md - +++ b/README.md - @@ -221,9 +221,9 @@ - traverse or edit that graph without corrupting reusable index state; payload objects - remain caller-owned references. Internal sent-date tie-break positions never become - public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject - -unknown, malformed, cyclic, aliased, or oversized input, and stop UTF-8 size - -validation at the configured byte limit without building a second full serialized - -copy. See - +unknown, malformed, cyclic, aliased, or oversized input, and count UTF-8 - +code-point width to stop at the configured byte limit without encoding a second - +full bytes copy. See - [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, - identity, snapshot, complexity, and RFC boundaries. - - --- a/CHANGELOG.md - +++ b/CHANGELOG.md - @@ -6,9 +6,10 @@ - - ## Unreleased - - -- Bound incremental snapshot size checks to streaming UTF-8 encoding and reject - - reused container identities so compact Python object graphs cannot trigger - - exponential JSON expansion or a second full serialized copy in memory. - +- Bound incremental snapshot size checks to streaming UTF-8 code-point - + accounting without encoding each JSON chunk into a second bytes object, and - + reject reused container identities so compact Python object graphs cannot - + trigger exponential JSON expansion. - - Reject cyclic built-in dictionaries and lists at the incremental snapshot - restore boundary without recursion or unbounded traversal. - - Reject container and scalar subclasses plus non-plain-string object keys at - PATCH - git diff --check - - - name: Verify focused snapshot behavior - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - pytest -q tests/test_incremental_snapshot.py \ - -k 'utf8_size_counting or size_validation or unencodable_unicode' - - - name: Verify the complete repository - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit the verified product update - shell: bash - run: | - set -euo pipefail - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add \ - src/threadweave/incremental.py \ - tests/test_incremental_snapshot.py \ - docs/incremental-threading.md \ - README.md \ - CHANGELOG.md - git diff --cached --check - git commit -m 'perf: avoid snapshot UTF-8 bytes duplication' - git push origin HEAD:feature/incremental-thread-index From 00b0422737ca883ae5e71072fdccdf0f0c79771e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 15:59:15 +0900 Subject: [PATCH 75/97] ci: verify snapshot restore preflight --- .../apply-pr20-restore-preflight.yml | 543 ++++++++++++++++++ 1 file changed, 543 insertions(+) create mode 100644 .github/workflows/apply-pr20-restore-preflight.yml diff --git a/.github/workflows/apply-pr20-restore-preflight.yml b/.github/workflows/apply-pr20-restore-preflight.yml new file mode 100644 index 0000000..92e7288 --- /dev/null +++ b/.github/workflows/apply-pr20-restore-preflight.yml @@ -0,0 +1,543 @@ +name: Apply PR 20 snapshot restore preflight + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/apply-pr20-restore-preflight.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-restore-preflight + cancel-in-progress: false + +jobs: + red-green-verify: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Verify the reviewed baseline + shell: bash + run: | + set -euo pipefail + test "$(git hash-object src/threadweave/incremental.py)" = \ + '53c5aec50bf3411da7f1d8597de4a8f11239d1e4' + test "$(git hash-object tests/test_incremental_snapshot.py)" = \ + '3179e044533e423c5fb4d55dfae66c2627df08f9' + test "$(git hash-object docs/incremental-threading.md)" = \ + '35529ecc24de15386fe792175d4c17951c448b93' + test "$(git hash-object README.md)" = \ + 'b6d0c42f1a985a590993e46a4178224b2de71dc7' + test "$(git hash-object CHANGELOG.md)" = \ + '3d43ba99b556161f124a9ea624e81ae76b8dc74b' + + - name: Prove bounds are checked too late before the fix + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + import threadweave.incremental as incremental + from threadweave import IncrementalThreadError, IncrementalThreadIndex + + oversized_records = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [object(), object()], + } + try: + IncrementalThreadIndex.restore( + oversized_records, + max_snapshot_records=1, + max_snapshot_bytes=10_000, + ) + except IncrementalThreadError as error: + if "max_snapshot_records" in str(error): + raise SystemExit("record bound already runs before nested traversal") + else: + raise SystemExit("expected invalid nested records to fail") + + unknown_root = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [], + "unexpected": [object()], + } + try: + IncrementalThreadIndex.restore(unknown_root) + except IncrementalThreadError as error: + if "snapshot fields" in str(error): + raise SystemExit("root schema already runs before nested traversal") + else: + raise SystemExit("expected the unknown root field to fail") + + class ForbiddenEncoder: + def __init__(self, **_options): + raise AssertionError("JSON encoder constructed before structural bound") + + incremental.json.JSONEncoder = ForbiddenEncoder + structurally_oversized = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [None] * 100, + } + try: + IncrementalThreadIndex.restore( + structurally_oversized, + max_snapshot_records=200, + max_snapshot_bytes=10, + ) + except AssertionError: + pass + else: + raise SystemExit("expected pre-fix JSON encoder construction") + PYCODE + echo 'Observed all three expected pre-fix boundary failures.' \ + >>"$GITHUB_STEP_SUMMARY" + + - name: Apply the snapshot preflight patch + shell: bash + run: | + set -euo pipefail + git apply - <<'PATCH' + diff --git a/CHANGELOG.md b/CHANGELOG.md + index 3d43ba9..78cec06 100644 + --- a/CHANGELOG.md + +++ b/CHANGELOG.md + @@ -6,6 +6,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + + ## Unreleased + + +- Preflight incremental snapshot root fields and record counts before nested + + traversal, and cap plain-container validation by the configured byte limit so + + oversized hostile trees fail before JSON encoder construction. + - Bound incremental snapshot size checks to streaming UTF-8 code-point + accounting without encoding each JSON chunk into a second bytes object, and + reject reused container identities so compact Python object graphs cannot + diff --git a/README.md b/README.md + index b6d0c42..648b017 100644 + --- a/README.md + +++ b/README.md + @@ -223,7 +223,8 @@ remain caller-owned references. Internal sent-date tie-break positions never bec + public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject + unknown, malformed, cyclic, aliased, or oversized input, and count UTF-8 + code-point width to stop at the configured byte limit without encoding a second + -full bytes copy. See + +full bytes copy. Restore checks the fixed root schema and record-count bound before + +nested traversal, then caps the structural scan with the same byte limit. See + [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, + identity, snapshot, complexity, and RFC boundaries. + + diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md + index 35529ec..e0851f1 100644 + --- a/docs/incremental-threading.md + +++ b/docs/incremental-threading.md + @@ -210,7 +210,11 @@ active-path and seen-object guard. This prevents a compact Python DAG from expan + exponentially during encoding. The configured UTF-8 byte limit is counted from + incremental encoder chunks with allocation-free code-point width accounting. The + check aborts without calling ``str.encode`` on a complete chunk or materializing a + -second complete JSON string or byte array. + +second complete JSON string or byte array. Restore also preflights the exact + +root/options schema and the record-count limit before inspecting nested values. The + +plain-container walk stops when its visited-value count exceeds the configured byte + +limit; every JSON value requires at least one encoded byte, so that condition proves + +the snapshot is oversized before the JSON encoder is constructed. + + ## Correctness and operational boundaries + + diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py + index 53c5aec..674d097 100644 + --- a/src/threadweave/incremental.py + +++ b/src/threadweave/incremental.py + @@ -798,20 +798,30 @@ def _decoded_date(value: object, name: str) -> str | datetime | None: + raise IncrementalThreadError(f"{name} date kind is unsupported") + + + -def _require_plain_json_containers(value: object) -> None: + - """Reject executable, cyclic, or aliased containers before serialization. + +def _require_plain_json_containers( + + value: object, + + *, + + maximum_nodes: int | None = None, + +) -> None: + + """Reject executable, cyclic, aliased, or structurally oversized JSON trees. + + JSON decoding produces a tree of built-in dictionaries, lists, string + keys, and scalar values. Requiring exact runtime types prevents attacker- + controlled iteration, comparison, or scalar subclasses from executing, + while rejecting repeated container identities prevents a compact Python + - object graph from expanding exponentially during JSON encoding. + + object graph from expanding exponentially during JSON encoding. A node + + ceiling derived from the byte limit bounds validation before serialization. + """ + pending = [(value, False)] + active_containers: set[int] = set() + seen_containers: set[int] = set() + + visited_nodes = 0 + while pending: + current, exiting = pending.pop() + + if not exiting: + + visited_nodes += 1 + + if maximum_nodes is not None and visited_nodes > maximum_nodes: + + raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") + if type(current) in {dict, list}: + identity = id(current) + if exiting: + @@ -900,10 +910,26 @@ def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: + return encoded_bytes + + + -def _required_fields(value: Mapping[str, object], expected: set[str], name: str) -> None: + - """Require an exact untrusted mapping field set.""" + - if set(value) != expected: + +def _required_plain_object( + + value: object, + + expected: set[str], + + name: str, + +) -> dict[str, object]: + + """Return one exact built-in JSON object with the required plain-string keys.""" + + if not isinstance(value, Mapping): + + raise IncrementalThreadError(f"{name} must be a mapping") + + if type(value) is not dict: + + raise IncrementalThreadError( + + "snapshot must contain only plain JSON containers and scalar values" + + ) + + if dict.__len__(value) != len(expected): + raise IncrementalThreadError(f"{name} fields do not match the schema") + + keys = dict.keys(value) + + if any(type(key) is not str for key in keys): + + raise IncrementalThreadError("snapshot object keys must be plain strings") + + if set(keys) != expected: + + raise IncrementalThreadError(f"{name} fields do not match the schema") + + return value + + + class IncrementalThreadIndex: + @@ -1297,30 +1323,43 @@ class IncrementalThreadIndex: + max_snapshot_bytes, + "max_snapshot_bytes", + ) + - if not isinstance(snapshot, Mapping): + - raise IncrementalThreadError("snapshot must be a mapping") + - _require_plain_json_containers(snapshot) + - _bounded_snapshot_json_size(snapshot, max_bytes) + - _required_fields( + + snapshot_object = _required_plain_object( + snapshot, + {"schema_version", "version", "options", "records"}, + "snapshot", + ) + - schema_version = snapshot["schema_version"] + + raw_options = snapshot_object["options"] + + if not isinstance(raw_options, Mapping): + + raise IncrementalThreadError("snapshot options must be a mapping") + + options = _required_plain_object( + + raw_options, + + {"group_by_subject", "sort_by_sent_date"}, + + "option", + + ) + + encoded_records = snapshot_object["records"] + + if not isinstance(encoded_records, list): + + raise IncrementalThreadError("snapshot records must be a list") + + if type(encoded_records) is not list: + + raise IncrementalThreadError( + + "snapshot must contain only plain JSON containers and scalar values" + + ) + + if len(encoded_records) > max_records: + + raise IncrementalThreadError("snapshot exceeds max_snapshot_records") + + _require_plain_json_containers( + + snapshot_object, + + maximum_nodes=max_bytes, + + ) + + _bounded_snapshot_json_size(snapshot_object, max_bytes) + + schema_version = snapshot_object["schema_version"] + if ( + isinstance(schema_version, bool) + or not isinstance(schema_version, int) + or schema_version != _SNAPSHOT_SCHEMA_VERSION + ): + raise IncrementalThreadError("unsupported snapshot schema_version") + - version = _validated_nonnegative_integer(snapshot["version"], "version") + - options = snapshot["options"] + - if not isinstance(options, Mapping): + - raise IncrementalThreadError("snapshot options must be a mapping") + - _required_fields( + - options, + - {"group_by_subject", "sort_by_sent_date"}, + - "option", + + version = _validated_nonnegative_integer( + + snapshot_object["version"], + + "version", + ) + group_by_subject = options["group_by_subject"] + sort_by_sent_date = options["sort_by_sent_date"] + @@ -1328,12 +1367,6 @@ class IncrementalThreadIndex: + raise IncrementalThreadError("group_by_subject must be a boolean") + if not isinstance(sort_by_sent_date, bool): + raise IncrementalThreadError("sort_by_sent_date must be a boolean") + - encoded_records = snapshot["records"] + - if not isinstance(encoded_records, list): + - raise IncrementalThreadError("snapshot records must be a list") + - if len(encoded_records) > max_records: + - raise IncrementalThreadError("snapshot exceeds max_snapshot_records") + - + records: list[IndexedMessage] = [] + seen_keys: set[str] = set() + record_fields = {"message_key", "email_id", "thread_id", "message"} + @@ -1348,17 +1381,21 @@ class IncrementalThreadIndex: + "uid", + } + for encoded_record in encoded_records: + - if not isinstance(encoded_record, Mapping): + - raise IncrementalThreadError("snapshot record must be a mapping") + - _required_fields(encoded_record, record_fields, "record") + + encoded_record = _required_plain_object( + + encoded_record, + + record_fields, + + "record", + + ) + key = _validated_message_key(encoded_record["message_key"]) + if key in seen_keys: + raise IncrementalThreadError(f"duplicate message_key in snapshot: {key}") + seen_keys.add(key) + encoded_message = encoded_record["message"] + - if not isinstance(encoded_message, Mapping): + - raise IncrementalThreadError("snapshot message must be a mapping") + - _required_fields(encoded_message, message_fields, "message") + + encoded_message = _required_plain_object( + + encoded_message, + + message_fields, + + "message", + + ) + in_reply_to = encoded_message["in_reply_to"] + references = encoded_message["references"] + if not isinstance(in_reply_to, list) or not all( + diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py + index 3179e04..8bb237a 100644 + --- a/tests/test_incremental_snapshot.py + +++ b/tests/test_incremental_snapshot.py + @@ -194,8 +194,7 @@ def test_restore_reports_excessive_json_nesting_as_a_domain_error(): + "group_by_subject": False, + "sort_by_sent_date": False, + }, + - "records": [], + - "unexpected": nested, + + "records": nested, + } + + with pytest.raises(IncrementalThreadError, match="JSON-safe"): + @@ -358,6 +357,18 @@ def test_plain_json_guard_rejects_executable_key_and_scalar_subclasses(): + {hostile_key: "value", "schema_version": 1} + ) + + + hostile_snapshot = { + + "schema_version": 1, + + "version": 0, + + "options": { + + "group_by_subject": False, + + "sort_by_sent_date": False, + + }, + + _HostileString("records"): [], + + } + + with pytest.raises(IncrementalThreadError, match="plain strings"): + + IncrementalThreadIndex.restore(hostile_snapshot) + + + with pytest.raises(IncrementalThreadError, match="scalar values"): + incremental_module._require_plain_json_containers( + {"value": _HostileString("hostile")} + @@ -368,8 +379,17 @@ def test_plain_container_guard_rejects_cycles_and_reused_containers(): + """JSON snapshots are trees: cycles and shared container identities fail closed.""" + mapping_cycle: dict[str, object] = {} + mapping_cycle["self"] = mapping_cycle + + mapping_snapshot = { + + "schema_version": 1, + + "version": 0, + + "options": { + + "group_by_subject": False, + + "sort_by_sent_date": False, + + }, + + "records": [mapping_cycle], + + } + with pytest.raises(IncrementalThreadError, match="cyclic"): + - IncrementalThreadIndex.restore(mapping_cycle) + + IncrementalThreadIndex.restore(mapping_snapshot) + + list_cycle: list[object] = [] + list_cycle.append(list_cycle) + @@ -457,3 +477,81 @@ def test_snapshot_size_validation_stops_at_the_utf8_limit(monkeypatch): + with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): + incremental_module._bounded_snapshot_json_size({}, 3) + assert later_chunks_requested == [] + + + + + +def test_restore_checks_record_limit_before_nested_record_values(): + + """The record-count limit rejects an oversized list before nested traversal.""" + + snapshot = { + + "schema_version": 1, + + "version": 0, + + "options": { + + "group_by_subject": False, + + "sort_by_sent_date": False, + + }, + + "records": [object(), object()], + + } + + + + with pytest.raises(IncrementalThreadError, match="max_snapshot_records"): + + IncrementalThreadIndex.restore( + + snapshot, + + max_snapshot_records=1, + + max_snapshot_bytes=10_000, + + ) + + + + + +def test_restore_checks_root_fields_before_untrusted_extra_values(): + + """Unknown root fields fail before their nested values are traversed.""" + + snapshot = { + + "schema_version": 1, + + "version": 0, + + "options": { + + "group_by_subject": False, + + "sort_by_sent_date": False, + + }, + + "records": [], + + "unexpected": [object()], + + } + + + + with pytest.raises(IncrementalThreadError, match="snapshot fields"): + + IncrementalThreadIndex.restore(snapshot) + + + + same_size_wrong_fields = { + + "schema_version": 1, + + "version": 0, + + "options": { + + "group_by_subject": False, + + "sort_by_sent_date": False, + + }, + + "unexpected": None, + + } + + with pytest.raises(IncrementalThreadError, match="snapshot fields"): + + IncrementalThreadIndex.restore(same_size_wrong_fields) + + + + + +def test_restore_bounds_plain_container_scan_by_snapshot_byte_limit(monkeypatch): + + """A small byte limit stops structural traversal before JSON encoding.""" + + + + class _ForbiddenEncoder: + + """Expose an attempt to encode after the structural limit is exceeded.""" + + + + def __init__(self, **_options: object) -> None: + + """Fail because bounded validation must reject before encoding.""" + + raise AssertionError("JSON encoder should not be constructed") + + + + monkeypatch.setattr(incremental_module.json, "JSONEncoder", _ForbiddenEncoder) + + snapshot = { + + "schema_version": 1, + + "version": 0, + + "options": { + + "group_by_subject": False, + + "sort_by_sent_date": False, + + }, + + "records": [None] * 100, + + } + + + + with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): + + IncrementalThreadIndex.restore( + + snapshot, + + max_snapshot_records=200, + + max_snapshot_bytes=10, + + ) + PATCH + git diff --check + + - name: Verify focused snapshot behavior + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + pytest -q tests/test_incremental_snapshot.py \ + -k 'record_limit_before or root_fields_before or bounds_plain_container or excessive_json_nesting or cycles_and_reused' + + - name: Verify the complete repository + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit the verified product update + shell: bash + run: | + set -euo pipefail + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add \ + src/threadweave/incremental.py \ + tests/test_incremental_snapshot.py \ + docs/incremental-threading.md \ + README.md \ + CHANGELOG.md + git diff --cached --check + git commit -m 'fix: preflight incremental snapshot bounds' + git push origin HEAD:feature/incremental-thread-index From 68d3c34a574f478aa3bc71295d658f69269d2a79 Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 6 Aug 2026 06:59:50 +0000 Subject: [PATCH 76/97] fix: preflight incremental snapshot bounds --- CHANGELOG.md | 3 + README.md | 3 +- docs/incremental-threading.md | 6 +- src/threadweave/incremental.py | 101 +++++++++++++++++++--------- tests/test_incremental_snapshot.py | 104 ++++++++++++++++++++++++++++- 5 files changed, 180 insertions(+), 37 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d43ba9..78cec06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Preflight incremental snapshot root fields and record counts before nested + traversal, and cap plain-container validation by the configured byte limit so + oversized hostile trees fail before JSON encoder construction. - Bound incremental snapshot size checks to streaming UTF-8 code-point accounting without encoding each JSON chunk into a second bytes object, and reject reused container identities so compact Python object graphs cannot diff --git a/README.md b/README.md index b6d0c42..648b017 100644 --- a/README.md +++ b/README.md @@ -223,7 +223,8 @@ remain caller-owned references. Internal sent-date tie-break positions never bec public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject unknown, malformed, cyclic, aliased, or oversized input, and count UTF-8 code-point width to stop at the configured byte limit without encoding a second -full bytes copy. See +full bytes copy. Restore checks the fixed root schema and record-count bound before +nested traversal, then caps the structural scan with the same byte limit. See [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, identity, snapshot, complexity, and RFC boundaries. diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index 35529ec..e0851f1 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -210,7 +210,11 @@ active-path and seen-object guard. This prevents a compact Python DAG from expan exponentially during encoding. The configured UTF-8 byte limit is counted from incremental encoder chunks with allocation-free code-point width accounting. The check aborts without calling ``str.encode`` on a complete chunk or materializing a -second complete JSON string or byte array. +second complete JSON string or byte array. Restore also preflights the exact +root/options schema and the record-count limit before inspecting nested values. The +plain-container walk stops when its visited-value count exceeds the configured byte +limit; every JSON value requires at least one encoded byte, so that condition proves +the snapshot is oversized before the JSON encoder is constructed. ## Correctness and operational boundaries diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index 53c5aec..674d097 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -798,20 +798,30 @@ def _decoded_date(value: object, name: str) -> str | datetime | None: raise IncrementalThreadError(f"{name} date kind is unsupported") -def _require_plain_json_containers(value: object) -> None: - """Reject executable, cyclic, or aliased containers before serialization. +def _require_plain_json_containers( + value: object, + *, + maximum_nodes: int | None = None, +) -> None: + """Reject executable, cyclic, aliased, or structurally oversized JSON trees. JSON decoding produces a tree of built-in dictionaries, lists, string keys, and scalar values. Requiring exact runtime types prevents attacker- controlled iteration, comparison, or scalar subclasses from executing, while rejecting repeated container identities prevents a compact Python - object graph from expanding exponentially during JSON encoding. + object graph from expanding exponentially during JSON encoding. A node + ceiling derived from the byte limit bounds validation before serialization. """ pending = [(value, False)] active_containers: set[int] = set() seen_containers: set[int] = set() + visited_nodes = 0 while pending: current, exiting = pending.pop() + if not exiting: + visited_nodes += 1 + if maximum_nodes is not None and visited_nodes > maximum_nodes: + raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") if type(current) in {dict, list}: identity = id(current) if exiting: @@ -900,10 +910,26 @@ def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: return encoded_bytes -def _required_fields(value: Mapping[str, object], expected: set[str], name: str) -> None: - """Require an exact untrusted mapping field set.""" - if set(value) != expected: +def _required_plain_object( + value: object, + expected: set[str], + name: str, +) -> dict[str, object]: + """Return one exact built-in JSON object with the required plain-string keys.""" + if not isinstance(value, Mapping): + raise IncrementalThreadError(f"{name} must be a mapping") + if type(value) is not dict: + raise IncrementalThreadError( + "snapshot must contain only plain JSON containers and scalar values" + ) + if dict.__len__(value) != len(expected): raise IncrementalThreadError(f"{name} fields do not match the schema") + keys = dict.keys(value) + if any(type(key) is not str for key in keys): + raise IncrementalThreadError("snapshot object keys must be plain strings") + if set(keys) != expected: + raise IncrementalThreadError(f"{name} fields do not match the schema") + return value class IncrementalThreadIndex: @@ -1297,30 +1323,43 @@ def restore( max_snapshot_bytes, "max_snapshot_bytes", ) - if not isinstance(snapshot, Mapping): - raise IncrementalThreadError("snapshot must be a mapping") - _require_plain_json_containers(snapshot) - _bounded_snapshot_json_size(snapshot, max_bytes) - _required_fields( + snapshot_object = _required_plain_object( snapshot, {"schema_version", "version", "options", "records"}, "snapshot", ) - schema_version = snapshot["schema_version"] + raw_options = snapshot_object["options"] + if not isinstance(raw_options, Mapping): + raise IncrementalThreadError("snapshot options must be a mapping") + options = _required_plain_object( + raw_options, + {"group_by_subject", "sort_by_sent_date"}, + "option", + ) + encoded_records = snapshot_object["records"] + if not isinstance(encoded_records, list): + raise IncrementalThreadError("snapshot records must be a list") + if type(encoded_records) is not list: + raise IncrementalThreadError( + "snapshot must contain only plain JSON containers and scalar values" + ) + if len(encoded_records) > max_records: + raise IncrementalThreadError("snapshot exceeds max_snapshot_records") + _require_plain_json_containers( + snapshot_object, + maximum_nodes=max_bytes, + ) + _bounded_snapshot_json_size(snapshot_object, max_bytes) + schema_version = snapshot_object["schema_version"] if ( isinstance(schema_version, bool) or not isinstance(schema_version, int) or schema_version != _SNAPSHOT_SCHEMA_VERSION ): raise IncrementalThreadError("unsupported snapshot schema_version") - version = _validated_nonnegative_integer(snapshot["version"], "version") - options = snapshot["options"] - if not isinstance(options, Mapping): - raise IncrementalThreadError("snapshot options must be a mapping") - _required_fields( - options, - {"group_by_subject", "sort_by_sent_date"}, - "option", + version = _validated_nonnegative_integer( + snapshot_object["version"], + "version", ) group_by_subject = options["group_by_subject"] sort_by_sent_date = options["sort_by_sent_date"] @@ -1328,12 +1367,6 @@ def restore( raise IncrementalThreadError("group_by_subject must be a boolean") if not isinstance(sort_by_sent_date, bool): raise IncrementalThreadError("sort_by_sent_date must be a boolean") - encoded_records = snapshot["records"] - if not isinstance(encoded_records, list): - raise IncrementalThreadError("snapshot records must be a list") - if len(encoded_records) > max_records: - raise IncrementalThreadError("snapshot exceeds max_snapshot_records") - records: list[IndexedMessage] = [] seen_keys: set[str] = set() record_fields = {"message_key", "email_id", "thread_id", "message"} @@ -1348,17 +1381,21 @@ def restore( "uid", } for encoded_record in encoded_records: - if not isinstance(encoded_record, Mapping): - raise IncrementalThreadError("snapshot record must be a mapping") - _required_fields(encoded_record, record_fields, "record") + encoded_record = _required_plain_object( + encoded_record, + record_fields, + "record", + ) key = _validated_message_key(encoded_record["message_key"]) if key in seen_keys: raise IncrementalThreadError(f"duplicate message_key in snapshot: {key}") seen_keys.add(key) encoded_message = encoded_record["message"] - if not isinstance(encoded_message, Mapping): - raise IncrementalThreadError("snapshot message must be a mapping") - _required_fields(encoded_message, message_fields, "message") + encoded_message = _required_plain_object( + encoded_message, + message_fields, + "message", + ) in_reply_to = encoded_message["in_reply_to"] references = encoded_message["references"] if not isinstance(in_reply_to, list) or not all( diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py index 3179e04..8bb237a 100644 --- a/tests/test_incremental_snapshot.py +++ b/tests/test_incremental_snapshot.py @@ -194,8 +194,7 @@ def test_restore_reports_excessive_json_nesting_as_a_domain_error(): "group_by_subject": False, "sort_by_sent_date": False, }, - "records": [], - "unexpected": nested, + "records": nested, } with pytest.raises(IncrementalThreadError, match="JSON-safe"): @@ -358,6 +357,18 @@ def test_plain_json_guard_rejects_executable_key_and_scalar_subclasses(): {hostile_key: "value", "schema_version": 1} ) + hostile_snapshot = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + _HostileString("records"): [], + } + with pytest.raises(IncrementalThreadError, match="plain strings"): + IncrementalThreadIndex.restore(hostile_snapshot) + with pytest.raises(IncrementalThreadError, match="scalar values"): incremental_module._require_plain_json_containers( {"value": _HostileString("hostile")} @@ -368,8 +379,17 @@ def test_plain_container_guard_rejects_cycles_and_reused_containers(): """JSON snapshots are trees: cycles and shared container identities fail closed.""" mapping_cycle: dict[str, object] = {} mapping_cycle["self"] = mapping_cycle + mapping_snapshot = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [mapping_cycle], + } with pytest.raises(IncrementalThreadError, match="cyclic"): - IncrementalThreadIndex.restore(mapping_cycle) + IncrementalThreadIndex.restore(mapping_snapshot) list_cycle: list[object] = [] list_cycle.append(list_cycle) @@ -457,3 +477,81 @@ def iterencode(self, _value: object): with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): incremental_module._bounded_snapshot_json_size({}, 3) assert later_chunks_requested == [] + + +def test_restore_checks_record_limit_before_nested_record_values(): + """The record-count limit rejects an oversized list before nested traversal.""" + snapshot = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [object(), object()], + } + + with pytest.raises(IncrementalThreadError, match="max_snapshot_records"): + IncrementalThreadIndex.restore( + snapshot, + max_snapshot_records=1, + max_snapshot_bytes=10_000, + ) + + +def test_restore_checks_root_fields_before_untrusted_extra_values(): + """Unknown root fields fail before their nested values are traversed.""" + snapshot = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [], + "unexpected": [object()], + } + + with pytest.raises(IncrementalThreadError, match="snapshot fields"): + IncrementalThreadIndex.restore(snapshot) + + same_size_wrong_fields = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "unexpected": None, + } + with pytest.raises(IncrementalThreadError, match="snapshot fields"): + IncrementalThreadIndex.restore(same_size_wrong_fields) + + +def test_restore_bounds_plain_container_scan_by_snapshot_byte_limit(monkeypatch): + """A small byte limit stops structural traversal before JSON encoding.""" + + class _ForbiddenEncoder: + """Expose an attempt to encode after the structural limit is exceeded.""" + + def __init__(self, **_options: object) -> None: + """Fail because bounded validation must reject before encoding.""" + raise AssertionError("JSON encoder should not be constructed") + + monkeypatch.setattr(incremental_module.json, "JSONEncoder", _ForbiddenEncoder) + snapshot = { + "schema_version": 1, + "version": 0, + "options": { + "group_by_subject": False, + "sort_by_sent_date": False, + }, + "records": [None] * 100, + } + + with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): + IncrementalThreadIndex.restore( + snapshot, + max_snapshot_records=200, + max_snapshot_bytes=10, + ) From a38532c2b62504a2600bf0d2c8f536057b97fe01 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 16:01:13 +0900 Subject: [PATCH 77/97] ci: remove completed snapshot preflight verifier --- .../apply-pr20-restore-preflight.yml | 543 ------------------ 1 file changed, 543 deletions(-) delete mode 100644 .github/workflows/apply-pr20-restore-preflight.yml diff --git a/.github/workflows/apply-pr20-restore-preflight.yml b/.github/workflows/apply-pr20-restore-preflight.yml deleted file mode 100644 index 92e7288..0000000 --- a/.github/workflows/apply-pr20-restore-preflight.yml +++ /dev/null @@ -1,543 +0,0 @@ -name: Apply PR 20 snapshot restore preflight - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - .github/workflows/apply-pr20-restore-preflight.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-restore-preflight - cancel-in-progress: false - -jobs: - red-green-verify: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' - runs-on: ubuntu-24.04 - timeout-minutes: 30 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Verify the reviewed baseline - shell: bash - run: | - set -euo pipefail - test "$(git hash-object src/threadweave/incremental.py)" = \ - '53c5aec50bf3411da7f1d8597de4a8f11239d1e4' - test "$(git hash-object tests/test_incremental_snapshot.py)" = \ - '3179e044533e423c5fb4d55dfae66c2627df08f9' - test "$(git hash-object docs/incremental-threading.md)" = \ - '35529ecc24de15386fe792175d4c17951c448b93' - test "$(git hash-object README.md)" = \ - 'b6d0c42f1a985a590993e46a4178224b2de71dc7' - test "$(git hash-object CHANGELOG.md)" = \ - '3d43ba99b556161f124a9ea624e81ae76b8dc74b' - - - name: Prove bounds are checked too late before the fix - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - import threadweave.incremental as incremental - from threadweave import IncrementalThreadError, IncrementalThreadIndex - - oversized_records = { - "schema_version": 1, - "version": 0, - "options": { - "group_by_subject": False, - "sort_by_sent_date": False, - }, - "records": [object(), object()], - } - try: - IncrementalThreadIndex.restore( - oversized_records, - max_snapshot_records=1, - max_snapshot_bytes=10_000, - ) - except IncrementalThreadError as error: - if "max_snapshot_records" in str(error): - raise SystemExit("record bound already runs before nested traversal") - else: - raise SystemExit("expected invalid nested records to fail") - - unknown_root = { - "schema_version": 1, - "version": 0, - "options": { - "group_by_subject": False, - "sort_by_sent_date": False, - }, - "records": [], - "unexpected": [object()], - } - try: - IncrementalThreadIndex.restore(unknown_root) - except IncrementalThreadError as error: - if "snapshot fields" in str(error): - raise SystemExit("root schema already runs before nested traversal") - else: - raise SystemExit("expected the unknown root field to fail") - - class ForbiddenEncoder: - def __init__(self, **_options): - raise AssertionError("JSON encoder constructed before structural bound") - - incremental.json.JSONEncoder = ForbiddenEncoder - structurally_oversized = { - "schema_version": 1, - "version": 0, - "options": { - "group_by_subject": False, - "sort_by_sent_date": False, - }, - "records": [None] * 100, - } - try: - IncrementalThreadIndex.restore( - structurally_oversized, - max_snapshot_records=200, - max_snapshot_bytes=10, - ) - except AssertionError: - pass - else: - raise SystemExit("expected pre-fix JSON encoder construction") - PYCODE - echo 'Observed all three expected pre-fix boundary failures.' \ - >>"$GITHUB_STEP_SUMMARY" - - - name: Apply the snapshot preflight patch - shell: bash - run: | - set -euo pipefail - git apply - <<'PATCH' - diff --git a/CHANGELOG.md b/CHANGELOG.md - index 3d43ba9..78cec06 100644 - --- a/CHANGELOG.md - +++ b/CHANGELOG.md - @@ -6,6 +6,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - - ## Unreleased - - +- Preflight incremental snapshot root fields and record counts before nested - + traversal, and cap plain-container validation by the configured byte limit so - + oversized hostile trees fail before JSON encoder construction. - - Bound incremental snapshot size checks to streaming UTF-8 code-point - accounting without encoding each JSON chunk into a second bytes object, and - reject reused container identities so compact Python object graphs cannot - diff --git a/README.md b/README.md - index b6d0c42..648b017 100644 - --- a/README.md - +++ b/README.md - @@ -223,7 +223,8 @@ remain caller-owned references. Internal sent-date tie-break positions never bec - public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject - unknown, malformed, cyclic, aliased, or oversized input, and count UTF-8 - code-point width to stop at the configured byte limit without encoding a second - -full bytes copy. See - +full bytes copy. Restore checks the fixed root schema and record-count bound before - +nested traversal, then caps the structural scan with the same byte limit. See - [`docs/incremental-threading.md`](docs/incremental-threading.md) for the atomicity, - identity, snapshot, complexity, and RFC boundaries. - - diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md - index 35529ec..e0851f1 100644 - --- a/docs/incremental-threading.md - +++ b/docs/incremental-threading.md - @@ -210,7 +210,11 @@ active-path and seen-object guard. This prevents a compact Python DAG from expan - exponentially during encoding. The configured UTF-8 byte limit is counted from - incremental encoder chunks with allocation-free code-point width accounting. The - check aborts without calling ``str.encode`` on a complete chunk or materializing a - -second complete JSON string or byte array. - +second complete JSON string or byte array. Restore also preflights the exact - +root/options schema and the record-count limit before inspecting nested values. The - +plain-container walk stops when its visited-value count exceeds the configured byte - +limit; every JSON value requires at least one encoded byte, so that condition proves - +the snapshot is oversized before the JSON encoder is constructed. - - ## Correctness and operational boundaries - - diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py - index 53c5aec..674d097 100644 - --- a/src/threadweave/incremental.py - +++ b/src/threadweave/incremental.py - @@ -798,20 +798,30 @@ def _decoded_date(value: object, name: str) -> str | datetime | None: - raise IncrementalThreadError(f"{name} date kind is unsupported") - - - -def _require_plain_json_containers(value: object) -> None: - - """Reject executable, cyclic, or aliased containers before serialization. - +def _require_plain_json_containers( - + value: object, - + *, - + maximum_nodes: int | None = None, - +) -> None: - + """Reject executable, cyclic, aliased, or structurally oversized JSON trees. - - JSON decoding produces a tree of built-in dictionaries, lists, string - keys, and scalar values. Requiring exact runtime types prevents attacker- - controlled iteration, comparison, or scalar subclasses from executing, - while rejecting repeated container identities prevents a compact Python - - object graph from expanding exponentially during JSON encoding. - + object graph from expanding exponentially during JSON encoding. A node - + ceiling derived from the byte limit bounds validation before serialization. - """ - pending = [(value, False)] - active_containers: set[int] = set() - seen_containers: set[int] = set() - + visited_nodes = 0 - while pending: - current, exiting = pending.pop() - + if not exiting: - + visited_nodes += 1 - + if maximum_nodes is not None and visited_nodes > maximum_nodes: - + raise IncrementalThreadError("snapshot exceeds max_snapshot_bytes") - if type(current) in {dict, list}: - identity = id(current) - if exiting: - @@ -900,10 +910,26 @@ def _bounded_snapshot_json_size(value: object, maximum_bytes: int) -> int: - return encoded_bytes - - - -def _required_fields(value: Mapping[str, object], expected: set[str], name: str) -> None: - - """Require an exact untrusted mapping field set.""" - - if set(value) != expected: - +def _required_plain_object( - + value: object, - + expected: set[str], - + name: str, - +) -> dict[str, object]: - + """Return one exact built-in JSON object with the required plain-string keys.""" - + if not isinstance(value, Mapping): - + raise IncrementalThreadError(f"{name} must be a mapping") - + if type(value) is not dict: - + raise IncrementalThreadError( - + "snapshot must contain only plain JSON containers and scalar values" - + ) - + if dict.__len__(value) != len(expected): - raise IncrementalThreadError(f"{name} fields do not match the schema") - + keys = dict.keys(value) - + if any(type(key) is not str for key in keys): - + raise IncrementalThreadError("snapshot object keys must be plain strings") - + if set(keys) != expected: - + raise IncrementalThreadError(f"{name} fields do not match the schema") - + return value - - - class IncrementalThreadIndex: - @@ -1297,30 +1323,43 @@ class IncrementalThreadIndex: - max_snapshot_bytes, - "max_snapshot_bytes", - ) - - if not isinstance(snapshot, Mapping): - - raise IncrementalThreadError("snapshot must be a mapping") - - _require_plain_json_containers(snapshot) - - _bounded_snapshot_json_size(snapshot, max_bytes) - - _required_fields( - + snapshot_object = _required_plain_object( - snapshot, - {"schema_version", "version", "options", "records"}, - "snapshot", - ) - - schema_version = snapshot["schema_version"] - + raw_options = snapshot_object["options"] - + if not isinstance(raw_options, Mapping): - + raise IncrementalThreadError("snapshot options must be a mapping") - + options = _required_plain_object( - + raw_options, - + {"group_by_subject", "sort_by_sent_date"}, - + "option", - + ) - + encoded_records = snapshot_object["records"] - + if not isinstance(encoded_records, list): - + raise IncrementalThreadError("snapshot records must be a list") - + if type(encoded_records) is not list: - + raise IncrementalThreadError( - + "snapshot must contain only plain JSON containers and scalar values" - + ) - + if len(encoded_records) > max_records: - + raise IncrementalThreadError("snapshot exceeds max_snapshot_records") - + _require_plain_json_containers( - + snapshot_object, - + maximum_nodes=max_bytes, - + ) - + _bounded_snapshot_json_size(snapshot_object, max_bytes) - + schema_version = snapshot_object["schema_version"] - if ( - isinstance(schema_version, bool) - or not isinstance(schema_version, int) - or schema_version != _SNAPSHOT_SCHEMA_VERSION - ): - raise IncrementalThreadError("unsupported snapshot schema_version") - - version = _validated_nonnegative_integer(snapshot["version"], "version") - - options = snapshot["options"] - - if not isinstance(options, Mapping): - - raise IncrementalThreadError("snapshot options must be a mapping") - - _required_fields( - - options, - - {"group_by_subject", "sort_by_sent_date"}, - - "option", - + version = _validated_nonnegative_integer( - + snapshot_object["version"], - + "version", - ) - group_by_subject = options["group_by_subject"] - sort_by_sent_date = options["sort_by_sent_date"] - @@ -1328,12 +1367,6 @@ class IncrementalThreadIndex: - raise IncrementalThreadError("group_by_subject must be a boolean") - if not isinstance(sort_by_sent_date, bool): - raise IncrementalThreadError("sort_by_sent_date must be a boolean") - - encoded_records = snapshot["records"] - - if not isinstance(encoded_records, list): - - raise IncrementalThreadError("snapshot records must be a list") - - if len(encoded_records) > max_records: - - raise IncrementalThreadError("snapshot exceeds max_snapshot_records") - - - records: list[IndexedMessage] = [] - seen_keys: set[str] = set() - record_fields = {"message_key", "email_id", "thread_id", "message"} - @@ -1348,17 +1381,21 @@ class IncrementalThreadIndex: - "uid", - } - for encoded_record in encoded_records: - - if not isinstance(encoded_record, Mapping): - - raise IncrementalThreadError("snapshot record must be a mapping") - - _required_fields(encoded_record, record_fields, "record") - + encoded_record = _required_plain_object( - + encoded_record, - + record_fields, - + "record", - + ) - key = _validated_message_key(encoded_record["message_key"]) - if key in seen_keys: - raise IncrementalThreadError(f"duplicate message_key in snapshot: {key}") - seen_keys.add(key) - encoded_message = encoded_record["message"] - - if not isinstance(encoded_message, Mapping): - - raise IncrementalThreadError("snapshot message must be a mapping") - - _required_fields(encoded_message, message_fields, "message") - + encoded_message = _required_plain_object( - + encoded_message, - + message_fields, - + "message", - + ) - in_reply_to = encoded_message["in_reply_to"] - references = encoded_message["references"] - if not isinstance(in_reply_to, list) or not all( - diff --git a/tests/test_incremental_snapshot.py b/tests/test_incremental_snapshot.py - index 3179e04..8bb237a 100644 - --- a/tests/test_incremental_snapshot.py - +++ b/tests/test_incremental_snapshot.py - @@ -194,8 +194,7 @@ def test_restore_reports_excessive_json_nesting_as_a_domain_error(): - "group_by_subject": False, - "sort_by_sent_date": False, - }, - - "records": [], - - "unexpected": nested, - + "records": nested, - } - - with pytest.raises(IncrementalThreadError, match="JSON-safe"): - @@ -358,6 +357,18 @@ def test_plain_json_guard_rejects_executable_key_and_scalar_subclasses(): - {hostile_key: "value", "schema_version": 1} - ) - - + hostile_snapshot = { - + "schema_version": 1, - + "version": 0, - + "options": { - + "group_by_subject": False, - + "sort_by_sent_date": False, - + }, - + _HostileString("records"): [], - + } - + with pytest.raises(IncrementalThreadError, match="plain strings"): - + IncrementalThreadIndex.restore(hostile_snapshot) - + - with pytest.raises(IncrementalThreadError, match="scalar values"): - incremental_module._require_plain_json_containers( - {"value": _HostileString("hostile")} - @@ -368,8 +379,17 @@ def test_plain_container_guard_rejects_cycles_and_reused_containers(): - """JSON snapshots are trees: cycles and shared container identities fail closed.""" - mapping_cycle: dict[str, object] = {} - mapping_cycle["self"] = mapping_cycle - + mapping_snapshot = { - + "schema_version": 1, - + "version": 0, - + "options": { - + "group_by_subject": False, - + "sort_by_sent_date": False, - + }, - + "records": [mapping_cycle], - + } - with pytest.raises(IncrementalThreadError, match="cyclic"): - - IncrementalThreadIndex.restore(mapping_cycle) - + IncrementalThreadIndex.restore(mapping_snapshot) - - list_cycle: list[object] = [] - list_cycle.append(list_cycle) - @@ -457,3 +477,81 @@ def test_snapshot_size_validation_stops_at_the_utf8_limit(monkeypatch): - with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): - incremental_module._bounded_snapshot_json_size({}, 3) - assert later_chunks_requested == [] - + - + - +def test_restore_checks_record_limit_before_nested_record_values(): - + """The record-count limit rejects an oversized list before nested traversal.""" - + snapshot = { - + "schema_version": 1, - + "version": 0, - + "options": { - + "group_by_subject": False, - + "sort_by_sent_date": False, - + }, - + "records": [object(), object()], - + } - + - + with pytest.raises(IncrementalThreadError, match="max_snapshot_records"): - + IncrementalThreadIndex.restore( - + snapshot, - + max_snapshot_records=1, - + max_snapshot_bytes=10_000, - + ) - + - + - +def test_restore_checks_root_fields_before_untrusted_extra_values(): - + """Unknown root fields fail before their nested values are traversed.""" - + snapshot = { - + "schema_version": 1, - + "version": 0, - + "options": { - + "group_by_subject": False, - + "sort_by_sent_date": False, - + }, - + "records": [], - + "unexpected": [object()], - + } - + - + with pytest.raises(IncrementalThreadError, match="snapshot fields"): - + IncrementalThreadIndex.restore(snapshot) - + - + same_size_wrong_fields = { - + "schema_version": 1, - + "version": 0, - + "options": { - + "group_by_subject": False, - + "sort_by_sent_date": False, - + }, - + "unexpected": None, - + } - + with pytest.raises(IncrementalThreadError, match="snapshot fields"): - + IncrementalThreadIndex.restore(same_size_wrong_fields) - + - + - +def test_restore_bounds_plain_container_scan_by_snapshot_byte_limit(monkeypatch): - + """A small byte limit stops structural traversal before JSON encoding.""" - + - + class _ForbiddenEncoder: - + """Expose an attempt to encode after the structural limit is exceeded.""" - + - + def __init__(self, **_options: object) -> None: - + """Fail because bounded validation must reject before encoding.""" - + raise AssertionError("JSON encoder should not be constructed") - + - + monkeypatch.setattr(incremental_module.json, "JSONEncoder", _ForbiddenEncoder) - + snapshot = { - + "schema_version": 1, - + "version": 0, - + "options": { - + "group_by_subject": False, - + "sort_by_sent_date": False, - + }, - + "records": [None] * 100, - + } - + - + with pytest.raises(IncrementalThreadError, match="max_snapshot_bytes"): - + IncrementalThreadIndex.restore( - + snapshot, - + max_snapshot_records=200, - + max_snapshot_bytes=10, - + ) - PATCH - git diff --check - - - name: Verify focused snapshot behavior - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - pytest -q tests/test_incremental_snapshot.py \ - -k 'record_limit_before or root_fields_before or bounds_plain_container or excessive_json_nesting or cycles_and_reused' - - - name: Verify the complete repository - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit the verified product update - shell: bash - run: | - set -euo pipefail - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add \ - src/threadweave/incremental.py \ - tests/test_incremental_snapshot.py \ - docs/incremental-threading.md \ - README.md \ - CHANGELOG.md - git diff --cached --check - git commit -m 'fix: preflight incremental snapshot bounds' - git push origin HEAD:feature/incremental-thread-index From 11448424965dc9f3cca10431bde21b211c87e67c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 16:55:41 +0900 Subject: [PATCH 78/97] perf: stage bounded incremental state overlays --- tools/pr20-bounded-state-overlays.patch | 821 ++++++++++++++++++++++++ 1 file changed, 821 insertions(+) create mode 100644 tools/pr20-bounded-state-overlays.patch diff --git a/tools/pr20-bounded-state-overlays.patch b/tools/pr20-bounded-state-overlays.patch new file mode 100644 index 0000000..58f0da7 --- /dev/null +++ b/tools/pr20-bounded-state-overlays.patch @@ -0,0 +1,821 @@ +diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md +index e7c1bad..26cbec1 100644 +--- a/ARCHITECTURE.md ++++ b/ARCHITECTURE.md +@@ -48,8 +48,13 @@ is maintained in incremental code. + commit; the later transaction observes the new version and fails explicitly. + - The process-local lock is not a distributed lock. Naruon or another host must + serialize durable writes across processes and persist the optimistic version. +-- `apply` validates and computes on isolated transaction state, then commits once. +-- Reverse connectivity buckets use copy-on-write mutation. ++- `apply` validates and computes on isolated transaction overlays, then commits once. ++- Default-mode updates retain the existing state-map objects and publish only touched ++ record, position, token, component, EMAILID, and THREADID entries. ++- Reverse connectivity buckets use copy-on-write mutation; RFC 8474 indexes retain ++ compact association/count state rather than one set object per message. ++- RFC sent-date ordering may still scan the mailbox to derive global ranks and reject ++ effective sequence-number collisions; the default first-appearance mode does not. + - Complete root/projection views are lazy caches invalidated by a successful change. + - `roots` returns a defensive structural copy; payload references stay caller-owned. + - Snapshot schema version 1 contains structural metadata only, never payload objects +@@ -70,7 +75,8 @@ The deterministic benchmark runs incremental and full-rebuild workers in separat + processes. It compares projection SHA-256 values and records initial-build time, + delta-application time, full-view materialization time, full-rebuild time, affected + message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 +-existing messages. ++existing messages. Focused performance contracts additionally reject default-mode ++small-delta implementations that iterate or replace unrelated state maps. + + ## Integration policy + +diff --git a/CHANGELOG.md b/CHANGELOG.md +index 78cec06..726e79f 100644 +--- a/CHANGELOG.md ++++ b/CHANGELOG.md +@@ -6,6 +6,10 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + + ## Unreleased + ++- Apply default-mode incremental deltas through bounded overlay mappings and ++ touched reverse indexes instead of cloning or scanning every mailbox state map; ++ unchanged records, positions, components, and RFC 8474 identity namespaces stay ++ in place until one validated commit. + - Preflight incremental snapshot root fields and record counts before nested + traversal, and cap plain-container validation by the configured byte limit so + oversized hostile trees fail before JSON encoder construction. +diff --git a/README.md b/README.md +index 648b017..b03c420 100644 +--- a/README.md ++++ b/README.md +@@ -216,8 +216,11 @@ assert IncrementalThreadIndex.restore(index.snapshot()).projections == ( + ``` + + Every affected component is recomputed through the canonical batch threader, and +-full-rebuild parity is the correctness oracle. Structural merges and splits are +-reported explicitly. `roots` returns a defensive structural copy, so callers may ++full-rebuild parity is the correctness oracle. Default-mode transactions stage ++records, positions, connectivity, component ownership, and RFC 8474 identity ++changes through bounded overlays; they do not copy or iterate unrelated mailbox ++state before one validated commit. Structural merges and splits are reported ++explicitly. `roots` returns a defensive structural copy, so callers may + traverse or edit that graph without corrupting reusable index state; payload objects + remain caller-owned references. Internal sent-date tie-break positions never become + public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject +diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md +index e0851f1..320191b 100644 +--- a/docs/incremental-threading.md ++++ b/docs/incremental-threading.md +@@ -143,9 +143,15 @@ old component; a new bridge message includes every old component touched by its + new tokens. The candidate region is repartitioned iteratively and each resulting + component is processed by `thread_messages`. + +-Unchanged components retain their existing internal `Container` roots. Applying a +-change does not eagerly rebuild the complete public forest: only the affected old +-and new component views are composed for `ThreadDelta`. The complete ordered forest ++Unchanged components retain their existing internal `Container` roots. In the ++default first-appearance ordering mode, a transaction uses overlay mappings and ++copy-on-write reverse buckets for only the changed records and affected components. ++It does not clone or iterate the unrelated record, position, token, component, ++EMAILID, or THREADID maps before the single commit point. Sent-date ordering still ++requires a global rank and effective-sequence validation because IMAP ordering is a ++mailbox-wide contract. Applying a change does not eagerly rebuild the complete ++public forest: only the affected old and new component views are composed for ++`ThreadDelta`. The complete ordered forest + is materialized once, on demand, when `roots` or `projections` is requested. Public + roots are defensive structural copies; editing their parent, child, or message + metadata cannot corrupt internal state, while caller payload objects remain shared by +diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py +index 674d097..44e5616 100644 +--- a/src/threadweave/incremental.py ++++ b/src/threadweave/incremental.py +@@ -10,11 +10,11 @@ payloads. + from __future__ import annotations + + import json +-from collections.abc import Iterable, Mapping, Sequence ++from collections.abc import Iterable, Iterator, Mapping, Sequence + from dataclasses import dataclass + from datetime import datetime + from _thread import RLock +-from typing import Literal ++from typing import Literal, TypeVar + + from threadweave.collation import unicode_casemap_key + from threadweave.container import Container +@@ -45,6 +45,57 @@ _DEFAULT_MAX_SNAPSHOT_RECORDS = 1_000_000 + _DEFAULT_MAX_SNAPSHOT_BYTES = 256 * 1024 * 1024 + _SNAPSHOT_SCHEMA_VERSION = 1 + ++_KeyT = TypeVar("_KeyT") ++_ValueT = TypeVar("_ValueT") ++ ++ ++class _OverlayMapping(Mapping[_KeyT, _ValueT]): ++ """Expose staged updates over a base mapping without copying unrelated entries.""" ++ ++ def __init__( ++ self, ++ base: Mapping[_KeyT, _ValueT], ++ updates: Mapping[_KeyT, _ValueT], ++ removals: frozenset[_KeyT] = frozenset(), ++ ) -> None: ++ """Retain immutable transaction views of one base mapping and its delta.""" ++ self._base = base ++ self._updates = updates ++ self._removals = removals ++ ++ def __getitem__(self, key: _KeyT) -> _ValueT: ++ """Return a staged value, excluding keys removed by the transaction.""" ++ if key in self._updates: ++ return self._updates[key] ++ if key in self._removals: ++ raise KeyError(key) ++ return self._base[key] ++ ++ def __iter__(self) -> Iterator[_KeyT]: ++ """Iterate the logical mapping only when a global operation requires it.""" ++ for key in self._base: ++ if key not in self._removals and key not in self._updates: ++ yield key ++ yield from self._updates ++ ++ def __len__(self) -> int: ++ """Return logical size from the bounded transaction delta.""" ++ removed = sum( ++ 1 ++ for key in self._removals ++ if key in self._base and key not in self._updates ++ ) ++ added = sum(1 for key in self._updates if key not in self._base) ++ return len(self._base) - removed + added ++ ++ def __contains__(self, key: object) -> bool: ++ """Test logical membership without iterating the base mapping.""" ++ if key in self._updates: ++ return True ++ if key in self._removals: ++ return False ++ return key in self._base ++ + + class IncrementalThreadError(ValueError): + """Raised when an incremental change or snapshot violates its contract.""" +@@ -369,47 +420,172 @@ def _connectivity_tokens( + return frozenset(tokens) + + +-def _writable_token_bucket( +- token: str, +- keys_by_token: dict[str, set[str]], +- copied_tokens: set[str], ++def _writable_bucket( ++ bucket_key: str, ++ base_buckets: Mapping[str, set[str]], ++ bucket_updates: dict[str, set[str]], + ) -> set[str]: +- """Return one copy-on-write reverse bucket owned by the transaction.""" +- if token not in copied_tokens: +- keys_by_token[token] = set(keys_by_token.get(token, set())) +- copied_tokens.add(token) +- elif token not in keys_by_token: +- keys_by_token[token] = set() +- return keys_by_token[token] ++ """Return one transaction-owned copy of a reverse-index bucket.""" ++ if bucket_key not in bucket_updates: ++ bucket_updates[bucket_key] = set(base_buckets.get(bucket_key, set())) ++ return bucket_updates[bucket_key] + + + def _remove_key_from_buckets( + key: str, +- tokens_by_key: dict[str, frozenset[str]], +- keys_by_token: dict[str, set[str]], +- copied_tokens: set[str], +-) -> frozenset[str]: +- """Remove one key from transaction-owned token buckets.""" +- old_tokens = tokens_by_key.pop(key, frozenset()) +- for token in old_tokens: +- bucket = _writable_token_bucket(token, keys_by_token, copied_tokens) +- bucket.discard(key) +- if not bucket: +- del keys_by_token[token] +- return old_tokens ++ bucket_keys: Iterable[str], ++ base_buckets: Mapping[str, set[str]], ++ bucket_updates: dict[str, set[str]], ++) -> None: ++ """Stage removal of one record key from selected reverse-index buckets.""" ++ for bucket_key in bucket_keys: ++ _writable_bucket(bucket_key, base_buckets, bucket_updates).discard(key) + + + def _add_key_to_buckets( + key: str, +- tokens: frozenset[str], +- tokens_by_key: dict[str, frozenset[str]], +- keys_by_token: dict[str, set[str]], +- copied_tokens: set[str], ++ bucket_keys: Iterable[str], ++ base_buckets: Mapping[str, set[str]], ++ bucket_updates: dict[str, set[str]], ++) -> None: ++ """Stage insertion of one record key into selected reverse-index buckets.""" ++ for bucket_key in bucket_keys: ++ _writable_bucket(bucket_key, base_buckets, bucket_updates).add(key) ++ ++ ++def _commit_bucket_updates( ++ target: dict[str, set[str]], ++ bucket_updates: Mapping[str, set[str]], + ) -> None: +- """Insert one key through transaction-owned copy-on-write buckets.""" +- tokens_by_key[key] = tokens +- for token in tokens: +- _writable_token_bucket(token, keys_by_token, copied_tokens).add(key) ++ """Publish touched reverse buckets and discard buckets that became empty.""" ++ for bucket_key, values in bucket_updates.items(): ++ if values: ++ target[bucket_key] = values ++ else: ++ target.pop(bucket_key, None) ++ ++ ++def _email_state_after( ++ email_id: str, ++ base_states: Mapping[str, tuple[str | None, int]], ++ state_updates: Mapping[str, tuple[str | None, int] | None], ++) -> tuple[str | None, int] | None: ++ """Return one staged EMAILID association and reference count.""" ++ if email_id in state_updates: ++ return state_updates[email_id] ++ return base_states.get(email_id) ++ ++ ++def _thread_count_after( ++ thread_id: str, ++ base_counts: Mapping[str, int], ++ count_updates: Mapping[str, int], ++) -> int: ++ """Return one staged THREADID reference count.""" ++ return count_updates.get(thread_id, base_counts.get(thread_id, 0)) ++ ++ ++def _stage_external_identity( ++ record: IndexedMessage, ++ adjustment: Literal[-1, 1], ++ base_email_states: Mapping[str, tuple[str | None, int]], ++ email_state_updates: dict[str, tuple[str | None, int] | None], ++ base_thread_counts: Mapping[str, int], ++ thread_count_updates: dict[str, int], ++ touched_values: set[str], ++) -> None: ++ """Stage one record's RFC 8474 identity contribution without a global scan.""" ++ email_id = record.email_id ++ if email_id is not None: ++ touched_values.add(email_id) ++ current_state = _email_state_after( ++ email_id, ++ base_email_states, ++ email_state_updates, ++ ) ++ if adjustment < 0: ++ if current_state is None or current_state[0] != record.thread_id: ++ raise IncrementalThreadError("internal EMAILID index is inconsistent") ++ next_count = current_state[1] - 1 ++ email_state_updates[email_id] = ( ++ None ++ if next_count == 0 ++ else (current_state[0], next_count) ++ ) ++ else: ++ if current_state is not None and current_state[0] != record.thread_id: ++ raise ExternalIdentityError( ++ f"messages with EMAILID {email_id!r} must expose the same THREADID" ++ ) ++ email_state_updates[email_id] = ( ++ record.thread_id, ++ 1 if current_state is None else current_state[1] + 1, ++ ) ++ ++ thread_id = record.thread_id ++ if thread_id is not None: ++ touched_values.add(thread_id) ++ next_count = ( ++ _thread_count_after( ++ thread_id, ++ base_thread_counts, ++ thread_count_updates, ++ ) ++ + adjustment ++ ) ++ if next_count < 0: ++ raise IncrementalThreadError("internal THREADID index is inconsistent") ++ thread_count_updates[thread_id] = next_count ++ ++ ++def _validate_touched_identity_namespaces( ++ touched_values: Iterable[str], ++ base_email_states: Mapping[str, tuple[str | None, int]], ++ email_state_updates: Mapping[str, tuple[str | None, int] | None], ++ base_thread_counts: Mapping[str, int], ++ thread_count_updates: Mapping[str, int], ++) -> None: ++ """Reject touched ObjectID values present in both RFC 8474 namespaces.""" ++ reused_values = sorted( ++ identity_value ++ for identity_value in touched_values ++ if _email_state_after( ++ identity_value, ++ base_email_states, ++ email_state_updates, ++ ) ++ is not None ++ and _thread_count_after( ++ identity_value, ++ base_thread_counts, ++ thread_count_updates, ++ ) ++ > 0 ++ ) ++ if reused_values: ++ raise ExternalIdentityError( ++ "EMAILID and THREADID must use disjoint ObjectID values: " ++ f"{reused_values!r}" ++ ) ++ ++ ++def _commit_external_identity_updates( ++ email_states: dict[str, tuple[str | None, int]], ++ email_state_updates: Mapping[str, tuple[str | None, int] | None], ++ thread_counts: dict[str, int], ++ thread_count_updates: Mapping[str, int], ++) -> None: ++ """Publish touched compact RFC 8474 indexes after transaction validation.""" ++ for email_id, state in email_state_updates.items(): ++ if state is None: ++ email_states.pop(email_id, None) ++ else: ++ email_states[email_id] = state ++ for thread_id, count in thread_count_updates.items(): ++ if count == 0: ++ thread_counts.pop(thread_id, None) ++ else: ++ thread_counts[thread_id] = count + + + def _ordered_keys(keys: Iterable[str], positions: Mapping[str, int]) -> tuple[str, ...]: +@@ -442,32 +568,6 @@ def _validate_effective_sequence_numbers( + used[sequence_number] = key + + +-def _validate_external_identities(records: Mapping[str, IndexedMessage]) -> None: +- """Enforce RFC 8474 EMAILID/THREADID consistency and namespace separation.""" +- thread_id_by_email_id: dict[str, str | None] = {} +- email_ids: set[str] = set() +- thread_ids: set[str] = set() +- for record in records.values(): +- email_id = record.email_id +- thread_id = record.thread_id +- if email_id is not None: +- email_ids.add(email_id) +- if email_id not in thread_id_by_email_id: +- thread_id_by_email_id[email_id] = thread_id +- elif thread_id_by_email_id[email_id] != thread_id: +- raise ExternalIdentityError( +- f"messages with EMAILID {email_id!r} must expose the same THREADID" +- ) +- if thread_id is not None: +- thread_ids.add(thread_id) +- reused_values = email_ids & thread_ids +- if reused_values: +- raise ExternalIdentityError( +- "EMAILID and THREADID must use disjoint ObjectID values: " +- f"{sorted(reused_values)!r}" +- ) +- +- + def _validate_replacement_identity( + old_record: IndexedMessage, + new_record: IndexedMessage, +@@ -965,6 +1065,8 @@ class IncrementalThreadIndex: + self._next_position = 1 + self._tokens_by_key: dict[str, frozenset[str]] = {} + self._keys_by_token: dict[str, set[str]] = {} ++ self._email_id_states: dict[str, tuple[str | None, int]] = {} ++ self._thread_id_counts: dict[str, int] = {} + self._component_by_key: dict[str, str] = {} + self._keys_by_component: dict[str, tuple[str, ...]] = {} + self._roots: tuple[Container, ...] | None = () +@@ -1065,10 +1167,11 @@ class IncrementalThreadIndex: + addition_keys = {record.message_key for record in change_set.additions} + replacement_keys = {record.message_key for record in change_set.replacements} + removal_keys = set(change_set.removals) +- existing_keys = set(self._records) +- already_present = addition_keys & existing_keys +- missing_replacements = replacement_keys - existing_keys +- missing_removals = removal_keys - existing_keys ++ already_present = {key for key in addition_keys if key in self._records} ++ missing_replacements = { ++ key for key in replacement_keys if key not in self._records ++ } ++ missing_removals = {key for key in removal_keys if key not in self._records} + if already_present: + raise IncrementalThreadError( + f"addition keys already exist: {sorted(already_present)!r}" +@@ -1094,64 +1197,103 @@ class IncrementalThreadIndex: + replacement, + ) + +- records = dict(self._records) +- positions = dict(self._positions) +- tokens_by_key = dict(self._tokens_by_key) +- keys_by_token = dict(self._keys_by_token) +- copied_tokens: set[str] = set() ++ record_updates = { ++ record.message_key: record ++ for record in (*copied_replacements, *copied_additions) ++ } ++ records = _OverlayMapping( ++ self._records, ++ record_updates, ++ frozenset(removal_keys), ++ ) ++ position_updates: dict[str, int] = {} + next_position = self._next_position ++ for addition in copied_additions: ++ position_updates[addition.message_key] = next_position ++ next_position += 1 ++ positions = _OverlayMapping( ++ self._positions, ++ position_updates, ++ frozenset(removal_keys), ++ ) ++ ++ token_updates: dict[str, frozenset[str]] = {} ++ token_bucket_updates: dict[str, set[str]] = {} ++ email_state_updates: dict[str, tuple[str | None, int] | None] = {} ++ thread_count_updates: dict[str, int] = {} + changed_existing_keys = replacement_keys | removal_keys + candidate_seeds: set[str] = set() + touched_tokens: set[str] = set() ++ touched_identity_values: set[str] = set() + + for key in changed_existing_keys: + component_id = self._component_by_key.get(key) + if component_id is not None: + candidate_seeds.update(self._keys_by_component[component_id]) +- touched_tokens.update( +- _remove_key_from_buckets( +- key, +- tokens_by_key, +- keys_by_token, +- copied_tokens, +- ) ++ old_tokens = self._tokens_by_key.get(key, frozenset()) ++ touched_tokens.update(old_tokens) ++ _remove_key_from_buckets( ++ key, ++ old_tokens, ++ self._keys_by_token, ++ token_bucket_updates, ++ ) ++ _stage_external_identity( ++ self._records[key], ++ -1, ++ self._email_id_states, ++ email_state_updates, ++ self._thread_id_counts, ++ thread_count_updates, ++ touched_identity_values, + ) + +- for key in removal_keys: +- records.pop(key) +- positions.pop(key) +- for replacement in copied_replacements: +- records[replacement.message_key] = replacement ++ for record in (*copied_replacements, *copied_additions): + tokens = _connectivity_tokens( +- replacement, ++ record, + group_by_subject=self._group_by_subject, + ) ++ token_updates[record.message_key] = tokens + touched_tokens.update(tokens) + _add_key_to_buckets( +- replacement.message_key, ++ record.message_key, + tokens, +- tokens_by_key, +- keys_by_token, +- copied_tokens, ++ self._keys_by_token, ++ token_bucket_updates, + ) +- candidate_seeds.add(replacement.message_key) +- for addition in copied_additions: +- records[addition.message_key] = addition +- positions[addition.message_key] = next_position +- next_position += 1 +- tokens = _connectivity_tokens( +- addition, +- group_by_subject=self._group_by_subject, ++ _stage_external_identity( ++ record, ++ 1, ++ self._email_id_states, ++ email_state_updates, ++ self._thread_id_counts, ++ thread_count_updates, ++ touched_identity_values, + ) +- touched_tokens.update(tokens) +- _add_key_to_buckets( +- addition.message_key, +- tokens, +- tokens_by_key, +- keys_by_token, +- copied_tokens, +- ) +- candidate_seeds.add(addition.message_key) ++ candidate_seeds.add(record.message_key) ++ ++ tokens_by_key = _OverlayMapping( ++ self._tokens_by_key, ++ token_updates, ++ frozenset(removal_keys), ++ ) ++ keys_by_token = _OverlayMapping(self._keys_by_token, token_bucket_updates) + + for token in touched_tokens: + candidate_seeds.update(keys_by_token.get(token, set())) +@@ -1186,14 +1328,19 @@ class IncrementalThreadIndex: + group_by_subject=self._group_by_subject, + sort_by_sent_date=self._sort_by_sent_date, + ) + +- _validate_external_identities(records) ++ _validate_touched_identity_namespaces( ++ touched_identity_values, ++ self._email_id_states, ++ email_state_updates, ++ self._thread_id_counts, ++ thread_count_updates, ++ ) + ranks = ( + _current_ranks(positions) + if self._sort_by_sent_date + else positions + ) + if self._sort_by_sent_date: + _validate_effective_sequence_numbers(records, ranks) +@@ -1200,30 +1347,14 @@ class IncrementalThreadIndex: + tokens_by_key, + keys_by_token, + ) +- old_candidate_keys = set(candidate_seeds) +- candidate_keys = current_candidate_keys | old_candidate_keys +- +- component_by_key = { +- key: component_id +- for key, component_id in self._component_by_key.items() +- if key not in candidate_keys and key in records +- } +- unaffected_component_ids = set(component_by_key.values()) +- keys_by_component = { +- component_id: keys +- for component_id, keys in self._keys_by_component.items() +- if component_id in unaffected_component_ids +- } +- for keys in _partition_components( ++ candidate_keys = current_candidate_keys | set(candidate_seeds) ++ new_components = _partition_components( + current_candidate_keys, + positions, + tokens_by_key, + keys_by_token, +- ): +- component_id = keys[0] +- keys_by_component[component_id] = keys +- for key in keys: +- component_by_key[key] = component_id ++ ) + + after_affected_keys = _ordered_keys(current_candidate_keys, positions) + _, after_affected_projections = _build_forest( +@@ -1250,13 +1381,31 @@ class IncrementalThreadIndex: + after_affected_projections, + ) + +- self._records = records +- self._positions = positions +- self._next_position = next_position +- self._tokens_by_key = tokens_by_key +- self._keys_by_token = keys_by_token +- self._component_by_key = component_by_key +- self._keys_by_component = keys_by_component ++ for key in removal_keys: ++ self._records.pop(key) ++ self._positions.pop(key) ++ self._tokens_by_key.pop(key, None) ++ self._records.update(record_updates) ++ self._positions.update(position_updates) ++ self._tokens_by_key.update(token_updates) ++ _commit_bucket_updates(self._keys_by_token, token_bucket_updates) ++ _commit_external_identity_updates( ++ self._email_id_states, ++ email_state_updates, ++ self._thread_id_counts, ++ thread_count_updates, ++ ) ++ ++ for component_id in old_component_ids: ++ self._keys_by_component.pop(component_id, None) ++ for key in candidate_keys: ++ self._component_by_key.pop(key, None) ++ for keys in new_components: ++ component_id = keys[0] ++ self._keys_by_component[component_id] = keys ++ for key in keys: ++ self._component_by_key[key] = component_id ++ ++ self._next_position = next_position + self._roots = None + self._projections = None + self._version = version +diff --git a/tests/test_incremental_private_graph.py b/tests/test_incremental_private_graph.py +index 325ce7a..99bbe6c 100644 +--- a/tests/test_incremental_private_graph.py ++++ b/tests/test_incremental_private_graph.py +@@ -153,30 +153,29 @@ def test_reverse_token_buckets_are_copied_only_when_mutated(): + """Atomic changes preserve every shared pre-transaction bucket.""" + original = {"token": {"a", "b"}} +- overlay = dict(original) +- tokens_by_key = {"a": frozenset({"token"})} +- copied_tokens: set[str] = set() ++ updates: dict[str, set[str]] = {} + + incremental._remove_key_from_buckets( + "a", +- tokens_by_key, +- overlay, +- copied_tokens, ++ ("token",), ++ original, ++ updates, + ) + incremental._add_key_to_buckets( + "c", +- frozenset({"token"}), +- tokens_by_key, +- overlay, +- copied_tokens, ++ ("token",), ++ original, ++ updates, + ) + + assert original == {"token": {"a", "b"}} +- assert overlay == {"token": {"b", "c"}} +- assert copied_tokens == {"token"} ++ assert updates == {"token": {"b", "c"}} ++ incremental._commit_bucket_updates(original, updates) ++ assert original == {"token": {"b", "c"}} ++ incremental._commit_bucket_updates(original, {"token": set()}) ++ assert original == {} + + + def test_component_partition_reads_positions_linearly_for_disconnected_keys(): +@@ -238,3 +237,133 @@ def test_replacement_recovers_when_derived_component_mapping_is_missing(): + ) + assert index.projections[0].message_keys == ("a",) ++ ++ ++def test_default_delta_does_not_copy_or_iterate_unrelated_state_maps(): ++ """A small default-mode update touches bounded state instead of the whole mailbox.""" ++ ++ class NoFullIterationDict(dict[str, object]): ++ """Permit keyed access and mutation while rejecting whole-map scans.""" ++ ++ def __iter__(self): ++ """Reject direct iteration over unrelated state.""" ++ raise AssertionError("unexpected full-state iteration") ++ ++ def keys(self): ++ """Reject key-view scans over unrelated state.""" ++ raise AssertionError("unexpected full-state key scan") ++ ++ def items(self): ++ """Reject item-view scans over unrelated state.""" ++ raise AssertionError("unexpected full-state item scan") ++ ++ def values(self): ++ """Reject value-view scans over unrelated state.""" ++ raise AssertionError("unexpected full-state value scan") ++ ++ records = tuple( ++ _record( ++ f"message_{index}", ++ Message(message_id=f"message_{index}"), ++ email_id=f"Email_{index}", ++ thread_id=f"Thread_{index}", ++ ) ++ for index in range(128) ++ ) ++ index = IncrementalThreadIndex() ++ index.apply(MailboxChangeSet(expected_version=0, additions=records)) ++ ++ state_names = ( ++ "_records", ++ "_positions", ++ "_tokens_by_key", ++ "_keys_by_token", ++ "_email_id_states", ++ "_thread_id_counts", ++ "_component_by_key", ++ "_keys_by_component", ++ ) ++ for name in state_names: ++ setattr(index, name, NoFullIterationDict(getattr(index, name))) ++ state_identities = {name: id(getattr(index, name)) for name in state_names} ++ ++ delta = index.apply( ++ MailboxChangeSet( ++ expected_version=1, ++ replacements=( ++ _record( ++ "message_0", ++ Message(message_id="message_0", subject="updated"), ++ email_id="Email_0", ++ thread_id="Thread_0", ++ ), ++ ), ++ ) ++ ) ++ ++ assert delta.affected_message_keys == ("message_0",) ++ assert index.version == 2 ++ assert {name: id(getattr(index, name)) for name in state_names} == state_identities ++ ++ ++def test_external_identity_indexes_update_only_touched_namespaces(): ++ """Identity removals and additions remain atomic without a mailbox-wide scan.""" ++ index = IncrementalThreadIndex() ++ index.apply( ++ MailboxChangeSet( ++ expected_version=0, ++ additions=( ++ _record("a", email_id="Mail_A", thread_id="Thread_A"), ++ _record("b", email_id="Mail_B", thread_id="Thread_B"), ++ ), ++ ) ++ ) ++ ++ index.apply( ++ MailboxChangeSet( ++ expected_version=1, ++ removals=("a",), ++ additions=( ++ _record("c", email_id="Thread_A", thread_id="Mail_A"), ++ ), ++ ) ++ ) ++ ++ assert index._email_id_states == { ++ "Mail_B": ("Thread_B", 1), ++ "Thread_A": ("Mail_A", 1), ++ } ++ assert index._thread_id_counts == {"Thread_B": 1, "Mail_A": 1} ++ ++ before = index.snapshot() ++ with pytest.raises(IncrementalThreadError, match="disjoint ObjectID"): ++ index.apply( ++ MailboxChangeSet( ++ expected_version=2, ++ additions=( ++ _record("d", email_id="Thread_B", thread_id="Thread_D"), ++ ), ++ ) ++ ) ++ assert index.snapshot() == before ++ ++ ++def test_external_identity_index_corruption_fails_before_commit(): ++ """Broken private identity indexes cannot be published by a transaction.""" ++ index = IncrementalThreadIndex() ++ index.apply( ++ MailboxChangeSet( ++ expected_version=0, ++ additions=( ++ _record("a", email_id="Mail_A", thread_id="Thread_A"), ++ ), ++ ) ++ ) ++ index._email_id_states.clear() ++ with pytest.raises(IncrementalThreadError, match="EMAILID index"): ++ index.apply(MailboxChangeSet(expected_version=1, removals=("a",))) ++ ++ index._email_id_states["Mail_A"] = ("Thread_A", 1) ++ index._thread_id_counts.clear() ++ with pytest.raises(IncrementalThreadError, match="THREADID index"): ++ index.apply(MailboxChangeSet(expected_version=1, removals=("a",))) From acbdc3ccc3ddf1aa3e137227928b92b89396d3b4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 16:58:41 +0900 Subject: [PATCH 79/97] ci: verify bounded incremental state overlays --- .../apply-pr20-bounded-state-overlays.yml | 414 ++++++++++++++++++ 1 file changed, 414 insertions(+) create mode 100644 .github/workflows/apply-pr20-bounded-state-overlays.yml diff --git a/.github/workflows/apply-pr20-bounded-state-overlays.yml b/.github/workflows/apply-pr20-bounded-state-overlays.yml new file mode 100644 index 0000000..0d1eb08 --- /dev/null +++ b/.github/workflows/apply-pr20-bounded-state-overlays.yml @@ -0,0 +1,414 @@ +name: Apply PR 20 bounded state overlays + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/apply-pr20-bounded-state-overlays.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-bounded-state-overlays + cancel-in-progress: false + +jobs: + red-green-verify: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 35 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Verify the reviewed baseline and patch + shell: bash + run: | + set -euo pipefail + test "$(git hash-object src/threadweave/incremental.py)" = \ + '674d0978648c90d81e35a845b441fa367bb424d2' + test "$(git hash-object tests/test_incremental_private_graph.py)" = \ + '325ce7a1b07a7f0cf6674bcde93e03d5aa6e5fd6' + test "$(git hash-object CHANGELOG.md)" = \ + '78cec06aee3d679b8fe2d19f801de64ee7117ed3' + test "$(git hash-object README.md)" = \ + '648b017a8ceebe28db8739419541d7d4a1482d6e' + test "$(git hash-object docs/incremental-threading.md)" = \ + 'e0851f170870542030fff2ef34c71763e38e969d' + test "$(git hash-object ARCHITECTURE.md)" = \ + 'e7c1badd6155e85b83729236723e30e6487d2b4e' + test "$(git hash-object tools/pr20-bounded-state-overlays.patch)" = \ + '58f0da771cfc692f863d0d8381080f92574ae1bf' + git apply --check tools/pr20-bounded-state-overlays.patch + + - name: Prove default updates scan and replace mailbox-wide maps before the fix + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + ) + + class NoFullIterationDict(dict): + def __iter__(self): + raise AssertionError("unexpected full-state iteration") + + def keys(self): + raise AssertionError("unexpected full-state key scan") + + def items(self): + raise AssertionError("unexpected full-state item scan") + + def values(self): + raise AssertionError("unexpected full-state value scan") + + records = tuple( + IndexedMessage( + f"message_{index}", + Message(message_id=f"message_{index}"), + ) + for index in range(256) + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + for name in ( + "_records", + "_positions", + "_tokens_by_key", + "_keys_by_token", + "_component_by_key", + "_keys_by_component", + ): + setattr(index, name, NoFullIterationDict(getattr(index, name))) + + try: + index.apply( + MailboxChangeSet( + expected_version=1, + replacements=( + IndexedMessage( + "message_0", + Message(message_id="message_0", subject="updated"), + ), + ), + ) + ) + except AssertionError as error: + if "full-state" not in str(error): + raise + else: + raise SystemExit("expected the pre-fix delta to scan unrelated state") + PYCODE + echo 'Observed the expected pre-fix mailbox-wide state scan.' \ + >>"$GITHUB_STEP_SUMMARY" + + - name: Apply the bounded transaction overlay patch + shell: bash + run: | + set -euo pipefail + git apply tools/pr20-bounded-state-overlays.patch + rm tools/pr20-bounded-state-overlays.patch + rmdir tools 2>/dev/null || true + git diff --check + + - name: Verify focused state, identity, component, and RFC behavior + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + pytest -q \ + tests/test_incremental_private_graph.py \ + tests/test_incremental_rfc8474.py \ + tests/test_incremental_components.py \ + tests/test_incremental_parity.py \ + tests/test_incremental_concurrency.py + + - name: Verify 100,000-message default delta allocations are bounded + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + import gc + import json + import time + import tracemalloc + + from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + ) + + message_count = 100_000 + records = tuple( + IndexedMessage( + f"key_{index}", + Message(message_id=f"message_{index}"), + ) + for index in range(message_count) + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + gc.collect() + tracemalloc.start() + baseline_current, _ = tracemalloc.get_traced_memory() + started = time.perf_counter() + delta = index.apply( + MailboxChangeSet( + expected_version=1, + replacements=( + IndexedMessage( + "key_0", + Message(message_id="message_0", subject="updated"), + ), + ), + ) + ) + elapsed_seconds = time.perf_counter() - started + current_bytes, peak_bytes = tracemalloc.get_traced_memory() + tracemalloc.stop() + transient_peak_bytes = peak_bytes - baseline_current + retained_bytes = current_bytes - baseline_current + if delta.affected_message_keys != ("key_0",): + raise SystemExit(delta.affected_message_keys) + if transient_peak_bytes >= 1_000_000: + raise SystemExit( + f"default delta allocated {transient_peak_bytes} transient bytes" + ) + evidence = { + "message_count": message_count, + "affected_message_count": len(delta.affected_message_keys), + "delta_apply_seconds": elapsed_seconds, + "retained_delta_bytes": retained_bytes, + "transient_peak_delta_bytes": transient_peak_bytes, + } + print(json.dumps(evidence, sort_keys=True)) + with open("$GITHUB_STEP_SUMMARY", "a", encoding="utf-8") as summary: + summary.write("## Bounded default delta evidence\n\n") + summary.write("```json\n") + summary.write(json.dumps(evidence, indent=2, sort_keys=True)) + summary.write("\n```\n") + PYCODE + + - name: Verify bounded randomized batch and identity parity + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + import random + + from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + thread_messages, + ) + + def projection(roots): + result = [] + for root in roots: + keys = [] + seen = set() + stack = [root] + while stack: + node = stack.pop() + if id(node) in seen: + continue + seen.add(id(node)) + if node.message is not None: + keys.append(node.message.payload) + stack.extend(reversed(node.children)) + result.append(tuple(keys)) + return tuple(result) + + transition_count = 0 + subjects = (None, "Topic", "Re: Topic", "Ｔｏｐｉｃ", "Other") + dates = ( + None, + "1 Jan 2026 00:00:00 +0000", + "2 Jan 2026 09:00:00 +0900", + "bad date", + ) + for group_by_subject in (False, True): + for sort_by_sent_date in (False, True): + for seed in range(4): + source = random.Random( + 801_000 + + seed + + (100 if group_by_subject else 0) + + (1_000 if sort_by_sent_date else 0) + ) + index = IncrementalThreadIndex( + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + ) + records = {} + ordered_keys = [] + next_key = 0 + for step in range(160): + roll = source.random() + if not records or roll < 0.48: + key = f"seed_{seed}_message_{next_key}" + next_key += 1 + operation = "add" + elif roll < 0.78: + key = source.choice(ordered_keys) + operation = "replace" + else: + key = source.choice(ordered_keys) + operation = "remove" + + if operation != "remove": + existing_ids = [ + record.message.message_id + for record in records.values() + if record.message.message_id is not None + ] + mode = source.randrange(6) + if mode == 0: + message_id = None + elif mode == 1 and existing_ids: + message_id = source.choice(existing_ids) + else: + message_id = f"{key}@example.test" + candidates = existing_ids + [ + f"missing_{index}@example.test" + for index in range(3) + ] + references = tuple( + source.choice(candidates) + for _ in range(source.randrange(3)) + ) if candidates else () + record = IndexedMessage( + key, + Message( + message_id=message_id, + references=references, + subject=source.choice(subjects), + sent_date=source.choice(dates), + payload=key, + ), + email_id=f"Email_{key}", + thread_id=f"Thread_{key}", + ) + + if operation == "add": + change = MailboxChangeSet( + index.version, + additions=(record,), + ) + records[key] = record + ordered_keys.append(key) + elif operation == "replace": + change = MailboxChangeSet( + index.version, + replacements=(record,), + ) + records[key] = record + else: + change = MailboxChangeSet( + index.version, + removals=(key,), + ) + del records[key] + ordered_keys.remove(key) + + index.apply(change) + expected = projection( + thread_messages( + (records[key].message for key in ordered_keys), + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + ) + ) + observed = tuple( + item.message_keys for item in index.projections + ) + if observed != expected: + raise SystemExit( + (group_by_subject, sort_by_sent_date, seed, step) + ) + transition_count += 1 + print(f"validated {transition_count} randomized transitions") + PYCODE + + - name: Verify the complete repository + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit the verified product update + shell: bash + run: | + set -euo pipefail + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add \ + src/threadweave/incremental.py \ + tests/test_incremental_private_graph.py \ + CHANGELOG.md \ + README.md \ + docs/incremental-threading.md \ + ARCHITECTURE.md + git add -u tools/pr20-bounded-state-overlays.patch + git diff --cached --check + git commit -m 'perf: bound default incremental state updates' + git push origin HEAD:feature/incremental-thread-index From fc78fe6148f28d44c331427406dfee08cab32f7f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 17:53:39 +0900 Subject: [PATCH 80/97] ci: repair bounded overlay patch metadata --- .../workflows/repair-pr20-bounded-patch.yml | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 .github/workflows/repair-pr20-bounded-patch.yml diff --git a/.github/workflows/repair-pr20-bounded-patch.yml b/.github/workflows/repair-pr20-bounded-patch.yml new file mode 100644 index 0000000..837b168 --- /dev/null +++ b/.github/workflows/repair-pr20-bounded-patch.yml @@ -0,0 +1,71 @@ +name: Repair PR 20 bounded overlay patch + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/repair-pr20-bounded-patch.yml + +permissions: + contents: write + +concurrency: + group: repair-pr20-bounded-overlay-patch + cancel-in-progress: false + +jobs: + repair-and-retrigger: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Repair and validate the patch + shell: bash + run: | + set -euo pipefail + test "$(git hash-object tools/pr20-bounded-state-overlays.patch)" = \ + '58f0da771cfc692f863d0d8381080f92574ae1bf' + test "$(git hash-object .github/workflows/apply-pr20-bounded-state-overlays.yml)" = \ + '0d1eb08d4425db241b1de8704134f261f418d5c7' + python - <<'PYCODE' + from pathlib import Path + + patch_path = Path('tools/pr20-bounded-state-overlays.patch') + patch = patch_path.read_text(encoding='utf-8') + old = '@@ -1094,64 +1197,103 @@ class IncrementalThreadIndex:' + new = '@@ -1094,64 +1197,87 @@ class IncrementalThreadIndex:' + if patch.count(old) != 1: + raise SystemExit('unexpected bounded-overlay patch header state') + patch_path.write_text(patch.replace(old, new), encoding='utf-8') + + apply_path = Path('.github/workflows/apply-pr20-bounded-state-overlays.yml') + workflow = apply_path.read_text(encoding='utf-8') + marker = '# corrected-patch-trigger-v1\n' + if marker in workflow: + raise SystemExit('existing apply workflow already contains repair marker') + apply_path.write_text(workflow + '\n' + marker, encoding='utf-8') + PYCODE + git apply --check tools/pr20-bounded-state-overlays.patch + rm .github/workflows/repair-pr20-bounded-patch.yml + git diff --check + + - name: Commit the repaired patch and retrigger product verification + shell: bash + run: | + set -euo pipefail + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add \ + tools/pr20-bounded-state-overlays.patch \ + .github/workflows/apply-pr20-bounded-state-overlays.yml + git add -u .github/workflows/repair-pr20-bounded-patch.yml + git diff --cached --check + git commit -m 'ci: repair bounded overlay patch metadata' + git push origin HEAD:feature/incremental-thread-index From 0855822fda2aab87213845bf5f3611c08bc573ce Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 17:59:37 +0900 Subject: [PATCH 81/97] ci: repair all bounded overlay hunk counts --- .../workflows/repair-pr20-bounded-patch.yml | 68 +++++++++++++++---- 1 file changed, 55 insertions(+), 13 deletions(-) diff --git a/.github/workflows/repair-pr20-bounded-patch.yml b/.github/workflows/repair-pr20-bounded-patch.yml index 837b168..aa52276 100644 --- a/.github/workflows/repair-pr20-bounded-patch.yml +++ b/.github/workflows/repair-pr20-bounded-patch.yml @@ -26,7 +26,7 @@ jobs: ref: feature/incremental-thread-index fetch-depth: 1 - - name: Repair and validate the patch + - name: Recalculate every unified-diff hunk count and validate the patch shell: bash run: | set -euo pipefail @@ -35,22 +35,64 @@ jobs: test "$(git hash-object .github/workflows/apply-pr20-bounded-state-overlays.yml)" = \ '0d1eb08d4425db241b1de8704134f261f418d5c7' python - <<'PYCODE' + from __future__ import annotations + + import re from pathlib import Path - patch_path = Path('tools/pr20-bounded-state-overlays.patch') - patch = patch_path.read_text(encoding='utf-8') - old = '@@ -1094,64 +1197,103 @@ class IncrementalThreadIndex:' - new = '@@ -1094,64 +1197,87 @@ class IncrementalThreadIndex:' - if patch.count(old) != 1: - raise SystemExit('unexpected bounded-overlay patch header state') - patch_path.write_text(patch.replace(old, new), encoding='utf-8') + patch_path = Path("tools/pr20-bounded-state-overlays.patch") + lines = patch_path.read_text(encoding="utf-8").splitlines(keepends=True) + header_pattern = re.compile( + r"^@@ -(\d+)(?:,\d+)? \+(\d+)(?:,\d+)? @@(.*?)(\n)?$" + ) + repaired: list[str] = [] + index = 0 + repaired_hunks = 0 + while index < len(lines): + line = lines[index] + match = header_pattern.match(line) + if match is None: + repaired.append(line) + index += 1 + continue + + body_start = index + 1 + body_end = body_start + while body_end < len(lines): + candidate = lines[body_end] + if candidate.startswith("@@ ") or candidate.startswith("diff --git "): + break + body_end += 1 + + body = lines[body_start:body_end] + old_count = sum( + 1 for body_line in body if body_line.startswith((" ", "-")) + ) + new_count = sum( + 1 for body_line in body if body_line.startswith((" ", "+")) + ) + suffix = match.group(3) + newline = match.group(4) or "" + repaired.append( + f"@@ -{match.group(1)},{old_count} " + f"+{match.group(2)},{new_count} @@{suffix}{newline}" + ) + repaired.extend(body) + repaired_hunks += 1 + index = body_end + + if repaired_hunks < 10: + raise SystemExit(f"unexpected hunk count: {repaired_hunks}") + patch_path.write_text("".join(repaired), encoding="utf-8") - apply_path = Path('.github/workflows/apply-pr20-bounded-state-overlays.yml') - workflow = apply_path.read_text(encoding='utf-8') - marker = '# corrected-patch-trigger-v1\n' + apply_path = Path( + ".github/workflows/apply-pr20-bounded-state-overlays.yml" + ) + workflow = apply_path.read_text(encoding="utf-8") + marker = "# corrected-patch-trigger-v2\n" if marker in workflow: - raise SystemExit('existing apply workflow already contains repair marker') - apply_path.write_text(workflow + '\n' + marker, encoding='utf-8') + raise SystemExit("existing apply workflow already contains repair marker") + apply_path.write_text(workflow + "\n" + marker, encoding="utf-8") PYCODE git apply --check tools/pr20-bounded-state-overlays.patch rm .github/workflows/repair-pr20-bounded-patch.yml From 1593bd72f39480d77936a47db6fff41843062234 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:04:14 +0900 Subject: [PATCH 82/97] ci: apply bounded overlays without workflow mutation --- .../workflows/repair-pr20-bounded-patch.yml | 406 +++++++++++++++++- 1 file changed, 383 insertions(+), 23 deletions(-) diff --git a/.github/workflows/repair-pr20-bounded-patch.yml b/.github/workflows/repair-pr20-bounded-patch.yml index aa52276..7766e63 100644 --- a/.github/workflows/repair-pr20-bounded-patch.yml +++ b/.github/workflows/repair-pr20-bounded-patch.yml @@ -1,4 +1,4 @@ -name: Repair PR 20 bounded overlay patch +name: Repair and apply PR 20 bounded state overlays on: push: @@ -11,14 +11,14 @@ permissions: contents: write concurrency: - group: repair-pr20-bounded-overlay-patch + group: repair-pr20-bounded-state-overlays cancel-in-progress: false jobs: - repair-and-retrigger: + red-green-verify-and-apply: if: github.repository == 'ContextualWisdomLab/ThreadWeave' runs-on: ubuntu-24.04 - timeout-minutes: 10 + timeout-minutes: 45 steps: - name: Check out the exact feature branch uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -26,14 +26,34 @@ jobs: ref: feature/incremental-thread-index fetch-depth: 1 + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + - name: Recalculate every unified-diff hunk count and validate the patch shell: bash run: | set -euo pipefail + test "$(git hash-object src/threadweave/incremental.py)" = \ + '674d0978648c90d81e35a845b441fa367bb424d2' + test "$(git hash-object tests/test_incremental_private_graph.py)" = \ + '325ce7a1b07a7f0cf6674bcde93e03d5aa6e5fd6' + test "$(git hash-object CHANGELOG.md)" = \ + '78cec06aee3d679b8fe2d19f801de64ee7117ed3' + test "$(git hash-object README.md)" = \ + '648b017a8ceebe28db8739419541d7d4a1482d6e' + test "$(git hash-object docs/incremental-threading.md)" = \ + 'e0851f170870542030fff2ef34c71763e38e969d' + test "$(git hash-object ARCHITECTURE.md)" = \ + 'e7c1badd6155e85b83729236723e30e6487d2b4e' test "$(git hash-object tools/pr20-bounded-state-overlays.patch)" = \ '58f0da771cfc692f863d0d8381080f92574ae1bf' - test "$(git hash-object .github/workflows/apply-pr20-bounded-state-overlays.yml)" = \ - '0d1eb08d4425db241b1de8704134f261f418d5c7' python - <<'PYCODE' from __future__ import annotations @@ -71,11 +91,10 @@ jobs: new_count = sum( 1 for body_line in body if body_line.startswith((" ", "+")) ) - suffix = match.group(3) - newline = match.group(4) or "" repaired.append( f"@@ -{match.group(1)},{old_count} " - f"+{match.group(2)},{new_count} @@{suffix}{newline}" + f"+{match.group(2)},{new_count} @@" + f"{match.group(3)}{match.group(4) or ''}" ) repaired.extend(body) repaired_hunks += 1 @@ -84,30 +103,371 @@ jobs: if repaired_hunks < 10: raise SystemExit(f"unexpected hunk count: {repaired_hunks}") patch_path.write_text("".join(repaired), encoding="utf-8") + PYCODE + git apply --check tools/pr20-bounded-state-overlays.patch + + - name: Prove default updates scan unrelated state before the fix + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + ) + + class NoFullIterationDict(dict): + def __iter__(self): + raise AssertionError("unexpected full-state iteration") + + def keys(self): + raise AssertionError("unexpected full-state key scan") + + def items(self): + raise AssertionError("unexpected full-state item scan") + + def values(self): + raise AssertionError("unexpected full-state value scan") - apply_path = Path( - ".github/workflows/apply-pr20-bounded-state-overlays.yml" + records = tuple( + IndexedMessage( + f"message_{index}", + Message(message_id=f"message_{index}"), + ) + for index in range(256) ) - workflow = apply_path.read_text(encoding="utf-8") - marker = "# corrected-patch-trigger-v2\n" - if marker in workflow: - raise SystemExit("existing apply workflow already contains repair marker") - apply_path.write_text(workflow + "\n" + marker, encoding="utf-8") + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + for name in ( + "_records", + "_positions", + "_tokens_by_key", + "_keys_by_token", + "_component_by_key", + "_keys_by_component", + ): + setattr(index, name, NoFullIterationDict(getattr(index, name))) + + try: + index.apply( + MailboxChangeSet( + expected_version=1, + replacements=( + IndexedMessage( + "message_0", + Message(message_id="message_0", subject="updated"), + ), + ), + ) + ) + except AssertionError as error: + if "full-state" not in str(error): + raise + else: + raise SystemExit("expected the pre-fix delta to scan unrelated state") PYCODE - git apply --check tools/pr20-bounded-state-overlays.patch - rm .github/workflows/repair-pr20-bounded-patch.yml + echo 'Observed the expected pre-fix mailbox-wide state scan.' \ + >>"$GITHUB_STEP_SUMMARY" + + - name: Apply the bounded transaction overlay patch + shell: bash + run: | + set -euo pipefail + git apply tools/pr20-bounded-state-overlays.patch + rm tools/pr20-bounded-state-overlays.patch + rmdir tools 2>/dev/null || true git diff --check - - name: Commit the repaired patch and retrigger product verification + - name: Verify focused state, identity, component, and RFC behavior + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + pytest -q \ + tests/test_incremental_private_graph.py \ + tests/test_incremental_rfc8474.py \ + tests/test_incremental_components.py \ + tests/test_incremental_parity.py \ + tests/test_incremental_concurrency.py + + - name: Verify 100,000-message default delta allocations are bounded + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + import gc + import json + import os + import time + import tracemalloc + + from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + ) + + message_count = 100_000 + records = tuple( + IndexedMessage( + f"key_{index}", + Message(message_id=f"message_{index}"), + ) + for index in range(message_count) + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + gc.collect() + tracemalloc.start() + baseline_current, _ = tracemalloc.get_traced_memory() + started = time.perf_counter() + delta = index.apply( + MailboxChangeSet( + expected_version=1, + replacements=( + IndexedMessage( + "key_0", + Message(message_id="message_0", subject="updated"), + ), + ), + ) + ) + elapsed_seconds = time.perf_counter() - started + current_bytes, peak_bytes = tracemalloc.get_traced_memory() + tracemalloc.stop() + transient_peak_bytes = peak_bytes - baseline_current + retained_bytes = current_bytes - baseline_current + if delta.affected_message_keys != ("key_0",): + raise SystemExit(delta.affected_message_keys) + if transient_peak_bytes >= 1_000_000: + raise SystemExit( + f"default delta allocated {transient_peak_bytes} transient bytes" + ) + evidence = { + "message_count": message_count, + "affected_message_count": len(delta.affected_message_keys), + "delta_apply_seconds": elapsed_seconds, + "retained_delta_bytes": retained_bytes, + "transient_peak_delta_bytes": transient_peak_bytes, + } + print(json.dumps(evidence, sort_keys=True)) + with open(os.environ["GITHUB_STEP_SUMMARY"], "a", encoding="utf-8") as summary: + summary.write("## Bounded default delta evidence\n\n") + summary.write("```json\n") + summary.write(json.dumps(evidence, indent=2, sort_keys=True)) + summary.write("\n```\n") + PYCODE + + - name: Verify randomized canonical parity + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + import random + + from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + thread_messages, + ) + + def projection(roots): + result = [] + for root in roots: + keys = [] + seen = set() + stack = [root] + while stack: + node = stack.pop() + if id(node) in seen: + continue + seen.add(id(node)) + if node.message is not None: + keys.append(node.message.payload) + stack.extend(reversed(node.children)) + result.append(tuple(keys)) + return tuple(result) + + transition_count = 0 + subjects = (None, "Topic", "Re: Topic", "Ｔｏｐｉｃ", "Other") + dates = ( + None, + "1 Jan 2026 00:00:00 +0000", + "2 Jan 2026 09:00:00 +0900", + "bad date", + ) + for group_by_subject in (False, True): + for sort_by_sent_date in (False, True): + for seed in range(4): + source = random.Random( + 801_000 + + seed + + (100 if group_by_subject else 0) + + (1_000 if sort_by_sent_date else 0) + ) + index = IncrementalThreadIndex( + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + ) + records = {} + ordered_keys = [] + next_key = 0 + for step in range(160): + roll = source.random() + if not records or roll < 0.48: + key = f"seed_{seed}_message_{next_key}" + next_key += 1 + operation = "add" + elif roll < 0.78: + key = source.choice(ordered_keys) + operation = "replace" + else: + key = source.choice(ordered_keys) + operation = "remove" + + if operation != "remove": + existing_ids = [ + record.message.message_id + for record in records.values() + if record.message.message_id is not None + ] + mode = source.randrange(6) + if mode == 0: + message_id = None + elif mode == 1 and existing_ids: + message_id = source.choice(existing_ids) + else: + message_id = f"{key}@example.test" + candidates = existing_ids + [ + f"missing_{item}@example.test" + for item in range(3) + ] + references = ( + tuple( + source.choice(candidates) + for _ in range(source.randrange(3)) + ) + if candidates + else () + ) + record = IndexedMessage( + key, + Message( + message_id=message_id, + references=references, + subject=source.choice(subjects), + sent_date=source.choice(dates), + payload=key, + ), + email_id=f"Email_{key}", + thread_id=f"Thread_{key}", + ) + + if operation == "add": + change = MailboxChangeSet( + index.version, + additions=(record,), + ) + records[key] = record + ordered_keys.append(key) + elif operation == "replace": + change = MailboxChangeSet( + index.version, + replacements=(record,), + ) + records[key] = record + else: + change = MailboxChangeSet( + index.version, + removals=(key,), + ) + del records[key] + ordered_keys.remove(key) + + index.apply(change) + expected = projection( + thread_messages( + (records[key].message for key in ordered_keys), + group_by_subject=group_by_subject, + sort_by_sent_date=sort_by_sent_date, + ) + ) + observed = tuple( + item.message_keys for item in index.projections + ) + if observed != expected: + raise SystemExit( + (group_by_subject, sort_by_sent_date, seed, step) + ) + transition_count += 1 + print(f"validated {transition_count} randomized transitions") + PYCODE + + - name: Verify the complete repository + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Commit only verified product changes shell: bash run: | set -euo pipefail git config user.name 'ThreadWeave verification bot' git config user.email '41898282+github-actions[bot]@users.noreply.github.com' git add \ - tools/pr20-bounded-state-overlays.patch \ - .github/workflows/apply-pr20-bounded-state-overlays.yml - git add -u .github/workflows/repair-pr20-bounded-patch.yml + src/threadweave/incremental.py \ + tests/test_incremental_private_graph.py \ + CHANGELOG.md \ + README.md \ + docs/incremental-threading.md \ + ARCHITECTURE.md + git add -u tools/pr20-bounded-state-overlays.patch git diff --cached --check - git commit -m 'ci: repair bounded overlay patch metadata' + if git diff --cached --name-only | grep -q '^\.github/workflows/'; then + echo 'Refusing to publish workflow changes from the one-shot verifier.' >&2 + exit 1 + fi + git commit -m 'perf: bound default incremental state updates' git push origin HEAD:feature/incremental-thread-index From db66b4036d5b90eaef171538db96ad60344fe888 Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 6 Aug 2026 09:04:56 +0000 Subject: [PATCH 83/97] perf: bound default incremental state updates --- ARCHITECTURE.md | 12 +- CHANGELOG.md | 4 + README.md | 7 +- docs/incremental-threading.md | 12 +- src/threadweave/incremental.py | 442 +++++++++---- tests/test_incremental_private_graph.py | 154 ++++- tools/pr20-bounded-state-overlays.patch | 821 ------------------------ 7 files changed, 482 insertions(+), 970 deletions(-) delete mode 100644 tools/pr20-bounded-state-overlays.patch diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index e7c1bad..26cbec1 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -48,8 +48,13 @@ is maintained in incremental code. commit; the later transaction observes the new version and fails explicitly. - The process-local lock is not a distributed lock. Naruon or another host must serialize durable writes across processes and persist the optimistic version. -- `apply` validates and computes on isolated transaction state, then commits once. -- Reverse connectivity buckets use copy-on-write mutation. +- `apply` validates and computes on isolated transaction overlays, then commits once. +- Default-mode updates retain the existing state-map objects and publish only touched + record, position, token, component, EMAILID, and THREADID entries. +- Reverse connectivity buckets use copy-on-write mutation; RFC 8474 indexes retain + compact association/count state rather than one set object per message. +- RFC sent-date ordering may still scan the mailbox to derive global ranks and reject + effective sequence-number collisions; the default first-appearance mode does not. - Complete root/projection views are lazy caches invalidated by a successful change. - `roots` returns a defensive structural copy; payload references stay caller-owned. - Snapshot schema version 1 contains structural metadata only, never payload objects @@ -70,7 +75,8 @@ The deterministic benchmark runs incremental and full-rebuild workers in separat processes. It compares projection SHA-256 values and records initial-build time, delta-application time, full-view materialization time, full-rebuild time, affected message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 -existing messages. +existing messages. Focused performance contracts additionally reject default-mode +small-delta implementations that iterate or replace unrelated state maps. ## Integration policy diff --git a/CHANGELOG.md b/CHANGELOG.md index 78cec06..726e79f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Apply default-mode incremental deltas through bounded overlay mappings and + touched reverse indexes instead of cloning or scanning every mailbox state map; + unchanged records, positions, components, and RFC 8474 identity namespaces stay + in place until one validated commit. - Preflight incremental snapshot root fields and record counts before nested traversal, and cap plain-container validation by the configured byte limit so oversized hostile trees fail before JSON encoder construction. diff --git a/README.md b/README.md index 648b017..b03c420 100644 --- a/README.md +++ b/README.md @@ -216,8 +216,11 @@ assert IncrementalThreadIndex.restore(index.snapshot()).projections == ( ``` Every affected component is recomputed through the canonical batch threader, and -full-rebuild parity is the correctness oracle. Structural merges and splits are -reported explicitly. `roots` returns a defensive structural copy, so callers may +full-rebuild parity is the correctness oracle. Default-mode transactions stage +records, positions, connectivity, component ownership, and RFC 8474 identity +changes through bounded overlays; they do not copy or iterate unrelated mailbox +state before one validated commit. Structural merges and splits are reported +explicitly. `roots` returns a defensive structural copy, so callers may traverse or edit that graph without corrupting reusable index state; payload objects remain caller-owned references. Internal sent-date tie-break positions never become public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index e0851f1..320191b 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -143,9 +143,15 @@ old component; a new bridge message includes every old component touched by its new tokens. The candidate region is repartitioned iteratively and each resulting component is processed by `thread_messages`. -Unchanged components retain their existing internal `Container` roots. Applying a -change does not eagerly rebuild the complete public forest: only the affected old -and new component views are composed for `ThreadDelta`. The complete ordered forest +Unchanged components retain their existing internal `Container` roots. In the +default first-appearance ordering mode, a transaction uses overlay mappings and +copy-on-write reverse buckets for only the changed records and affected components. +It does not clone or iterate the unrelated record, position, token, component, +EMAILID, or THREADID maps before the single commit point. Sent-date ordering still +requires a global rank and effective-sequence validation because IMAP ordering is a +mailbox-wide contract. Applying a change does not eagerly rebuild the complete +public forest: only the affected old and new component views are composed for +`ThreadDelta`. The complete ordered forest is materialized once, on demand, when `roots` or `projections` is requested. Public roots are defensive structural copies; editing their parent, child, or message metadata cannot corrupt internal state, while caller payload objects remain shared by diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py index 674d097..b52fb88 100644 --- a/src/threadweave/incremental.py +++ b/src/threadweave/incremental.py @@ -10,11 +10,11 @@ from __future__ import annotations import json -from collections.abc import Iterable, Mapping, Sequence +from collections.abc import Iterable, Iterator, Mapping, Sequence from dataclasses import dataclass from datetime import datetime from _thread import RLock -from typing import Literal +from typing import Literal, TypeVar from threadweave.collation import unicode_casemap_key from threadweave.container import Container @@ -45,6 +45,57 @@ _DEFAULT_MAX_SNAPSHOT_BYTES = 256 * 1024 * 1024 _SNAPSHOT_SCHEMA_VERSION = 1 +_KeyT = TypeVar("_KeyT") +_ValueT = TypeVar("_ValueT") + + +class _OverlayMapping(Mapping[_KeyT, _ValueT]): + """Expose staged updates over a base mapping without copying unrelated entries.""" + + def __init__( + self, + base: Mapping[_KeyT, _ValueT], + updates: Mapping[_KeyT, _ValueT], + removals: frozenset[_KeyT] = frozenset(), + ) -> None: + """Retain immutable transaction views of one base mapping and its delta.""" + self._base = base + self._updates = updates + self._removals = removals + + def __getitem__(self, key: _KeyT) -> _ValueT: + """Return a staged value, excluding keys removed by the transaction.""" + if key in self._updates: + return self._updates[key] + if key in self._removals: + raise KeyError(key) + return self._base[key] + + def __iter__(self) -> Iterator[_KeyT]: + """Iterate the logical mapping only when a global operation requires it.""" + for key in self._base: + if key not in self._removals and key not in self._updates: + yield key + yield from self._updates + + def __len__(self) -> int: + """Return logical size from the bounded transaction delta.""" + removed = sum( + 1 + for key in self._removals + if key in self._base and key not in self._updates + ) + added = sum(1 for key in self._updates if key not in self._base) + return len(self._base) - removed + added + + def __contains__(self, key: object) -> bool: + """Test logical membership without iterating the base mapping.""" + if key in self._updates: + return True + if key in self._removals: + return False + return key in self._base + class IncrementalThreadError(ValueError): """Raised when an incremental change or snapshot violates its contract.""" @@ -369,47 +420,172 @@ def _connectivity_tokens( return frozenset(tokens) -def _writable_token_bucket( - token: str, - keys_by_token: dict[str, set[str]], - copied_tokens: set[str], +def _writable_bucket( + bucket_key: str, + base_buckets: Mapping[str, set[str]], + bucket_updates: dict[str, set[str]], ) -> set[str]: - """Return one copy-on-write reverse bucket owned by the transaction.""" - if token not in copied_tokens: - keys_by_token[token] = set(keys_by_token.get(token, set())) - copied_tokens.add(token) - elif token not in keys_by_token: - keys_by_token[token] = set() - return keys_by_token[token] + """Return one transaction-owned copy of a reverse-index bucket.""" + if bucket_key not in bucket_updates: + bucket_updates[bucket_key] = set(base_buckets.get(bucket_key, set())) + return bucket_updates[bucket_key] def _remove_key_from_buckets( key: str, - tokens_by_key: dict[str, frozenset[str]], - keys_by_token: dict[str, set[str]], - copied_tokens: set[str], -) -> frozenset[str]: - """Remove one key from transaction-owned token buckets.""" - old_tokens = tokens_by_key.pop(key, frozenset()) - for token in old_tokens: - bucket = _writable_token_bucket(token, keys_by_token, copied_tokens) - bucket.discard(key) - if not bucket: - del keys_by_token[token] - return old_tokens + bucket_keys: Iterable[str], + base_buckets: Mapping[str, set[str]], + bucket_updates: dict[str, set[str]], +) -> None: + """Stage removal of one record key from selected reverse-index buckets.""" + for bucket_key in bucket_keys: + _writable_bucket(bucket_key, base_buckets, bucket_updates).discard(key) def _add_key_to_buckets( key: str, - tokens: frozenset[str], - tokens_by_key: dict[str, frozenset[str]], - keys_by_token: dict[str, set[str]], - copied_tokens: set[str], + bucket_keys: Iterable[str], + base_buckets: Mapping[str, set[str]], + bucket_updates: dict[str, set[str]], ) -> None: - """Insert one key through transaction-owned copy-on-write buckets.""" - tokens_by_key[key] = tokens - for token in tokens: - _writable_token_bucket(token, keys_by_token, copied_tokens).add(key) + """Stage insertion of one record key into selected reverse-index buckets.""" + for bucket_key in bucket_keys: + _writable_bucket(bucket_key, base_buckets, bucket_updates).add(key) + + +def _commit_bucket_updates( + target: dict[str, set[str]], + bucket_updates: Mapping[str, set[str]], +) -> None: + """Publish touched reverse buckets and discard buckets that became empty.""" + for bucket_key, values in bucket_updates.items(): + if values: + target[bucket_key] = values + else: + target.pop(bucket_key, None) + + +def _email_state_after( + email_id: str, + base_states: Mapping[str, tuple[str | None, int]], + state_updates: Mapping[str, tuple[str | None, int] | None], +) -> tuple[str | None, int] | None: + """Return one staged EMAILID association and reference count.""" + if email_id in state_updates: + return state_updates[email_id] + return base_states.get(email_id) + + +def _thread_count_after( + thread_id: str, + base_counts: Mapping[str, int], + count_updates: Mapping[str, int], +) -> int: + """Return one staged THREADID reference count.""" + return count_updates.get(thread_id, base_counts.get(thread_id, 0)) + + +def _stage_external_identity( + record: IndexedMessage, + adjustment: Literal[-1, 1], + base_email_states: Mapping[str, tuple[str | None, int]], + email_state_updates: dict[str, tuple[str | None, int] | None], + base_thread_counts: Mapping[str, int], + thread_count_updates: dict[str, int], + touched_values: set[str], +) -> None: + """Stage one record's RFC 8474 identity contribution without a global scan.""" + email_id = record.email_id + if email_id is not None: + touched_values.add(email_id) + current_state = _email_state_after( + email_id, + base_email_states, + email_state_updates, + ) + if adjustment < 0: + if current_state is None or current_state[0] != record.thread_id: + raise IncrementalThreadError("internal EMAILID index is inconsistent") + next_count = current_state[1] - 1 + email_state_updates[email_id] = ( + None + if next_count == 0 + else (current_state[0], next_count) + ) + else: + if current_state is not None and current_state[0] != record.thread_id: + raise ExternalIdentityError( + f"messages with EMAILID {email_id!r} must expose the same THREADID" + ) + email_state_updates[email_id] = ( + record.thread_id, + 1 if current_state is None else current_state[1] + 1, + ) + + thread_id = record.thread_id + if thread_id is not None: + touched_values.add(thread_id) + next_count = ( + _thread_count_after( + thread_id, + base_thread_counts, + thread_count_updates, + ) + + adjustment + ) + if next_count < 0: + raise IncrementalThreadError("internal THREADID index is inconsistent") + thread_count_updates[thread_id] = next_count + + +def _validate_touched_identity_namespaces( + touched_values: Iterable[str], + base_email_states: Mapping[str, tuple[str | None, int]], + email_state_updates: Mapping[str, tuple[str | None, int] | None], + base_thread_counts: Mapping[str, int], + thread_count_updates: Mapping[str, int], +) -> None: + """Reject touched ObjectID values present in both RFC 8474 namespaces.""" + reused_values = sorted( + identity_value + for identity_value in touched_values + if _email_state_after( + identity_value, + base_email_states, + email_state_updates, + ) + is not None + and _thread_count_after( + identity_value, + base_thread_counts, + thread_count_updates, + ) + > 0 + ) + if reused_values: + raise ExternalIdentityError( + "EMAILID and THREADID must use disjoint ObjectID values: " + f"{reused_values!r}" + ) + + +def _commit_external_identity_updates( + email_states: dict[str, tuple[str | None, int]], + email_state_updates: Mapping[str, tuple[str | None, int] | None], + thread_counts: dict[str, int], + thread_count_updates: Mapping[str, int], +) -> None: + """Publish touched compact RFC 8474 indexes after transaction validation.""" + for email_id, state in email_state_updates.items(): + if state is None: + email_states.pop(email_id, None) + else: + email_states[email_id] = state + for thread_id, count in thread_count_updates.items(): + if count == 0: + thread_counts.pop(thread_id, None) + else: + thread_counts[thread_id] = count def _ordered_keys(keys: Iterable[str], positions: Mapping[str, int]) -> tuple[str, ...]: @@ -442,32 +618,6 @@ def _validate_effective_sequence_numbers( used[sequence_number] = key -def _validate_external_identities(records: Mapping[str, IndexedMessage]) -> None: - """Enforce RFC 8474 EMAILID/THREADID consistency and namespace separation.""" - thread_id_by_email_id: dict[str, str | None] = {} - email_ids: set[str] = set() - thread_ids: set[str] = set() - for record in records.values(): - email_id = record.email_id - thread_id = record.thread_id - if email_id is not None: - email_ids.add(email_id) - if email_id not in thread_id_by_email_id: - thread_id_by_email_id[email_id] = thread_id - elif thread_id_by_email_id[email_id] != thread_id: - raise ExternalIdentityError( - f"messages with EMAILID {email_id!r} must expose the same THREADID" - ) - if thread_id is not None: - thread_ids.add(thread_id) - reused_values = email_ids & thread_ids - if reused_values: - raise ExternalIdentityError( - "EMAILID and THREADID must use disjoint ObjectID values: " - f"{sorted(reused_values)!r}" - ) - - def _validate_replacement_identity( old_record: IndexedMessage, new_record: IndexedMessage, @@ -965,6 +1115,8 @@ def __init__( self._next_position = 1 self._tokens_by_key: dict[str, frozenset[str]] = {} self._keys_by_token: dict[str, set[str]] = {} + self._email_id_states: dict[str, tuple[str | None, int]] = {} + self._thread_id_counts: dict[str, int] = {} self._component_by_key: dict[str, str] = {} self._keys_by_component: dict[str, tuple[str, ...]] = {} self._roots: tuple[Container, ...] | None = () @@ -1065,10 +1217,11 @@ def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: addition_keys = {record.message_key for record in change_set.additions} replacement_keys = {record.message_key for record in change_set.replacements} removal_keys = set(change_set.removals) - existing_keys = set(self._records) - already_present = addition_keys & existing_keys - missing_replacements = replacement_keys - existing_keys - missing_removals = removal_keys - existing_keys + already_present = {key for key in addition_keys if key in self._records} + missing_replacements = { + key for key in replacement_keys if key not in self._records + } + missing_removals = {key for key in removal_keys if key not in self._records} if already_present: raise IncrementalThreadError( f"addition keys already exist: {sorted(already_present)!r}" @@ -1094,64 +1247,87 @@ def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: replacement, ) - records = dict(self._records) - positions = dict(self._positions) - tokens_by_key = dict(self._tokens_by_key) - keys_by_token = dict(self._keys_by_token) - copied_tokens: set[str] = set() + record_updates = { + record.message_key: record + for record in (*copied_replacements, *copied_additions) + } + records = _OverlayMapping( + self._records, + record_updates, + frozenset(removal_keys), + ) + position_updates: dict[str, int] = {} next_position = self._next_position + for addition in copied_additions: + position_updates[addition.message_key] = next_position + next_position += 1 + positions = _OverlayMapping( + self._positions, + position_updates, + frozenset(removal_keys), + ) + + token_updates: dict[str, frozenset[str]] = {} + token_bucket_updates: dict[str, set[str]] = {} + email_state_updates: dict[str, tuple[str | None, int] | None] = {} + thread_count_updates: dict[str, int] = {} changed_existing_keys = replacement_keys | removal_keys candidate_seeds: set[str] = set() touched_tokens: set[str] = set() + touched_identity_values: set[str] = set() for key in changed_existing_keys: component_id = self._component_by_key.get(key) if component_id is not None: candidate_seeds.update(self._keys_by_component[component_id]) - touched_tokens.update( - _remove_key_from_buckets( - key, - tokens_by_key, - keys_by_token, - copied_tokens, - ) + old_tokens = self._tokens_by_key.get(key, frozenset()) + touched_tokens.update(old_tokens) + _remove_key_from_buckets( + key, + old_tokens, + self._keys_by_token, + token_bucket_updates, + ) + _stage_external_identity( + self._records[key], + -1, + self._email_id_states, + email_state_updates, + self._thread_id_counts, + thread_count_updates, + touched_identity_values, ) - for key in removal_keys: - records.pop(key) - positions.pop(key) - for replacement in copied_replacements: - records[replacement.message_key] = replacement + for record in (*copied_replacements, *copied_additions): tokens = _connectivity_tokens( - replacement, + record, group_by_subject=self._group_by_subject, ) + token_updates[record.message_key] = tokens touched_tokens.update(tokens) _add_key_to_buckets( - replacement.message_key, + record.message_key, tokens, - tokens_by_key, - keys_by_token, - copied_tokens, + self._keys_by_token, + token_bucket_updates, ) - candidate_seeds.add(replacement.message_key) - for addition in copied_additions: - records[addition.message_key] = addition - positions[addition.message_key] = next_position - next_position += 1 - tokens = _connectivity_tokens( - addition, - group_by_subject=self._group_by_subject, - ) - touched_tokens.update(tokens) - _add_key_to_buckets( - addition.message_key, - tokens, - tokens_by_key, - keys_by_token, - copied_tokens, + _stage_external_identity( + record, + 1, + self._email_id_states, + email_state_updates, + self._thread_id_counts, + thread_count_updates, + touched_identity_values, ) - candidate_seeds.add(addition.message_key) + candidate_seeds.add(record.message_key) + + tokens_by_key = _OverlayMapping( + self._tokens_by_key, + token_updates, + frozenset(removal_keys), + ) + keys_by_token = _OverlayMapping(self._keys_by_token, token_bucket_updates) for token in touched_tokens: candidate_seeds.update(keys_by_token.get(token, set())) @@ -1186,7 +1362,13 @@ def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: sort_by_sent_date=self._sort_by_sent_date, ) - _validate_external_identities(records) + _validate_touched_identity_namespaces( + touched_identity_values, + self._email_id_states, + email_state_updates, + self._thread_id_counts, + thread_count_updates, + ) ranks = ( _current_ranks(positions) if self._sort_by_sent_date @@ -1200,30 +1382,13 @@ def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: tokens_by_key, keys_by_token, ) - old_candidate_keys = set(candidate_seeds) - candidate_keys = current_candidate_keys | old_candidate_keys - - component_by_key = { - key: component_id - for key, component_id in self._component_by_key.items() - if key not in candidate_keys and key in records - } - unaffected_component_ids = set(component_by_key.values()) - keys_by_component = { - component_id: keys - for component_id, keys in self._keys_by_component.items() - if component_id in unaffected_component_ids - } - for keys in _partition_components( + candidate_keys = current_candidate_keys | set(candidate_seeds) + new_components = _partition_components( current_candidate_keys, positions, tokens_by_key, keys_by_token, - ): - component_id = keys[0] - keys_by_component[component_id] = keys - for key in keys: - component_by_key[key] = component_id + ) after_affected_keys = _ordered_keys(current_candidate_keys, positions) _, after_affected_projections = _build_forest( @@ -1250,13 +1415,32 @@ def _apply_locked(self, change_set: MailboxChangeSet) -> ThreadDelta: after_affected_projections, ) - self._records = records - self._positions = positions + for key in removal_keys: + self._records.pop(key) + self._positions.pop(key) + self._tokens_by_key.pop(key, None) + self._records.update(record_updates) + self._positions.update(position_updates) + self._tokens_by_key.update(token_updates) + _commit_bucket_updates(self._keys_by_token, token_bucket_updates) + _commit_external_identity_updates( + self._email_id_states, + email_state_updates, + self._thread_id_counts, + thread_count_updates, + ) + + for component_id in old_component_ids: + self._keys_by_component.pop(component_id, None) + for key in candidate_keys: + self._component_by_key.pop(key, None) + for keys in new_components: + component_id = keys[0] + self._keys_by_component[component_id] = keys + for key in keys: + self._component_by_key[key] = component_id + self._next_position = next_position - self._tokens_by_key = tokens_by_key - self._keys_by_token = keys_by_token - self._component_by_key = component_by_key - self._keys_by_component = keys_by_component self._roots = None self._projections = None self._version = version diff --git a/tests/test_incremental_private_graph.py b/tests/test_incremental_private_graph.py index 325ce7a..e2f45a8 100644 --- a/tests/test_incremental_private_graph.py +++ b/tests/test_incremental_private_graph.py @@ -152,27 +152,27 @@ def test_projection_membership_rejects_duplicate_keys_between_roots(): def test_reverse_token_buckets_are_copied_only_when_mutated(): """Atomic changes preserve every shared pre-transaction bucket.""" original = {"token": {"a", "b"}} - overlay = dict(original) - tokens_by_key = {"a": frozenset({"token"})} - copied_tokens: set[str] = set() + updates: dict[str, set[str]] = {} incremental._remove_key_from_buckets( "a", - tokens_by_key, - overlay, - copied_tokens, + ("token",), + original, + updates, ) incremental._add_key_to_buckets( "c", - frozenset({"token"}), - tokens_by_key, - overlay, - copied_tokens, + ("token",), + original, + updates, ) assert original == {"token": {"a", "b"}} - assert overlay == {"token": {"b", "c"}} - assert copied_tokens == {"token"} + assert updates == {"token": {"b", "c"}} + incremental._commit_bucket_updates(original, updates) + assert original == {"token": {"b", "c"}} + incremental._commit_bucket_updates(original, {"token": set()}) + assert original == {} def test_component_partition_reads_positions_linearly_for_disconnected_keys(): @@ -244,3 +244,133 @@ def test_replacement_recovers_when_derived_component_mapping_is_missing(): ) ) assert index.projections[0].message_keys == ("a",) + + +def test_default_delta_does_not_copy_or_iterate_unrelated_state_maps(): + """A small default-mode update touches bounded state instead of the whole mailbox.""" + + class NoFullIterationDict(dict[str, object]): + """Permit keyed access and mutation while rejecting whole-map scans.""" + + def __iter__(self): + """Reject direct iteration over unrelated state.""" + raise AssertionError("unexpected full-state iteration") + + def keys(self): + """Reject key-view scans over unrelated state.""" + raise AssertionError("unexpected full-state key scan") + + def items(self): + """Reject item-view scans over unrelated state.""" + raise AssertionError("unexpected full-state item scan") + + def values(self): + """Reject value-view scans over unrelated state.""" + raise AssertionError("unexpected full-state value scan") + + records = tuple( + _record( + f"message_{index}", + Message(message_id=f"message_{index}"), + email_id=f"Email_{index}", + thread_id=f"Thread_{index}", + ) + for index in range(128) + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + + state_names = ( + "_records", + "_positions", + "_tokens_by_key", + "_keys_by_token", + "_email_id_states", + "_thread_id_counts", + "_component_by_key", + "_keys_by_component", + ) + for name in state_names: + setattr(index, name, NoFullIterationDict(getattr(index, name))) + state_identities = {name: id(getattr(index, name)) for name in state_names} + + delta = index.apply( + MailboxChangeSet( + expected_version=1, + replacements=( + _record( + "message_0", + Message(message_id="message_0", subject="updated"), + email_id="Email_0", + thread_id="Thread_0", + ), + ), + ) + ) + + assert delta.affected_message_keys == ("message_0",) + assert index.version == 2 + assert {name: id(getattr(index, name)) for name in state_names} == state_identities + + +def test_external_identity_indexes_update_only_touched_namespaces(): + """Identity removals and additions remain atomic without a mailbox-wide scan.""" + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record("a", email_id="Mail_A", thread_id="Thread_A"), + _record("b", email_id="Mail_B", thread_id="Thread_B"), + ), + ) + ) + + index.apply( + MailboxChangeSet( + expected_version=1, + removals=("a",), + additions=( + _record("c", email_id="Thread_A", thread_id="Mail_A"), + ), + ) + ) + + assert index._email_id_states == { + "Mail_B": ("Thread_B", 1), + "Thread_A": ("Mail_A", 1), + } + assert index._thread_id_counts == {"Thread_B": 1, "Mail_A": 1} + + before = index.snapshot() + with pytest.raises(IncrementalThreadError, match="disjoint ObjectID"): + index.apply( + MailboxChangeSet( + expected_version=2, + additions=( + _record("d", email_id="Thread_B", thread_id="Thread_D"), + ), + ) + ) + assert index.snapshot() == before + + +def test_external_identity_index_corruption_fails_before_commit(): + """Broken private identity indexes cannot be published by a transaction.""" + index = IncrementalThreadIndex() + index.apply( + MailboxChangeSet( + expected_version=0, + additions=( + _record("a", email_id="Mail_A", thread_id="Thread_A"), + ), + ) + ) + index._email_id_states.clear() + with pytest.raises(IncrementalThreadError, match="EMAILID index"): + index.apply(MailboxChangeSet(expected_version=1, removals=("a",))) + + index._email_id_states["Mail_A"] = ("Thread_A", 1) + index._thread_id_counts.clear() + with pytest.raises(IncrementalThreadError, match="THREADID index"): + index.apply(MailboxChangeSet(expected_version=1, removals=("a",))) diff --git a/tools/pr20-bounded-state-overlays.patch b/tools/pr20-bounded-state-overlays.patch deleted file mode 100644 index 58f0da7..0000000 --- a/tools/pr20-bounded-state-overlays.patch +++ /dev/null @@ -1,821 +0,0 @@ -diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md -index e7c1bad..26cbec1 100644 ---- a/ARCHITECTURE.md -+++ b/ARCHITECTURE.md -@@ -48,8 +48,13 @@ is maintained in incremental code. - commit; the later transaction observes the new version and fails explicitly. - - The process-local lock is not a distributed lock. Naruon or another host must - serialize durable writes across processes and persist the optimistic version. --- `apply` validates and computes on isolated transaction state, then commits once. --- Reverse connectivity buckets use copy-on-write mutation. -+- `apply` validates and computes on isolated transaction overlays, then commits once. -+- Default-mode updates retain the existing state-map objects and publish only touched -+ record, position, token, component, EMAILID, and THREADID entries. -+- Reverse connectivity buckets use copy-on-write mutation; RFC 8474 indexes retain -+ compact association/count state rather than one set object per message. -+- RFC sent-date ordering may still scan the mailbox to derive global ranks and reject -+ effective sequence-number collisions; the default first-appearance mode does not. - - Complete root/projection views are lazy caches invalidated by a successful change. - - `roots` returns a defensive structural copy; payload references stay caller-owned. - - Snapshot schema version 1 contains structural metadata only, never payload objects -@@ -70,7 +75,8 @@ The deterministic benchmark runs incremental and full-rebuild workers in separat - processes. It compares projection SHA-256 values and records initial-build time, - delta-application time, full-view materialization time, full-rebuild time, affected - message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 --existing messages. -+existing messages. Focused performance contracts additionally reject default-mode -+small-delta implementations that iterate or replace unrelated state maps. - - ## Integration policy - -diff --git a/CHANGELOG.md b/CHANGELOG.md -index 78cec06..726e79f 100644 ---- a/CHANGELOG.md -+++ b/CHANGELOG.md -@@ -6,6 +6,10 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - - ## Unreleased - -+- Apply default-mode incremental deltas through bounded overlay mappings and -+ touched reverse indexes instead of cloning or scanning every mailbox state map; -+ unchanged records, positions, components, and RFC 8474 identity namespaces stay -+ in place until one validated commit. - - Preflight incremental snapshot root fields and record counts before nested - traversal, and cap plain-container validation by the configured byte limit so - oversized hostile trees fail before JSON encoder construction. -diff --git a/README.md b/README.md -index 648b017..b03c420 100644 ---- a/README.md -+++ b/README.md -@@ -216,8 +216,11 @@ assert IncrementalThreadIndex.restore(index.snapshot()).projections == ( - ``` - - Every affected component is recomputed through the canonical batch threader, and --full-rebuild parity is the correctness oracle. Structural merges and splits are --reported explicitly. `roots` returns a defensive structural copy, so callers may -+full-rebuild parity is the correctness oracle. Default-mode transactions stage -+records, positions, connectivity, component ownership, and RFC 8474 identity -+changes through bounded overlays; they do not copy or iterate unrelated mailbox -+state before one validated commit. Structural merges and splits are reported -+explicitly. `roots` returns a defensive structural copy, so callers may - traverse or edit that graph without corrupting reusable index state; payload objects - remain caller-owned references. Internal sent-date tie-break positions never become - public IMAP sequence numbers. Versioned snapshots omit arbitrary payloads, reject -diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md -index e0851f1..320191b 100644 ---- a/docs/incremental-threading.md -+++ b/docs/incremental-threading.md -@@ -143,9 +143,15 @@ old component; a new bridge message includes every old component touched by its - new tokens. The candidate region is repartitioned iteratively and each resulting - component is processed by `thread_messages`. - --Unchanged components retain their existing internal `Container` roots. Applying a --change does not eagerly rebuild the complete public forest: only the affected old --and new component views are composed for `ThreadDelta`. The complete ordered forest -+Unchanged components retain their existing internal `Container` roots. In the -+default first-appearance ordering mode, a transaction uses overlay mappings and -+copy-on-write reverse buckets for only the changed records and affected components. -+It does not clone or iterate the unrelated record, position, token, component, -+EMAILID, or THREADID maps before the single commit point. Sent-date ordering still -+requires a global rank and effective-sequence validation because IMAP ordering is a -+mailbox-wide contract. Applying a change does not eagerly rebuild the complete -+public forest: only the affected old and new component views are composed for -+`ThreadDelta`. The complete ordered forest - is materialized once, on demand, when `roots` or `projections` is requested. Public - roots are defensive structural copies; editing their parent, child, or message - metadata cannot corrupt internal state, while caller payload objects remain shared by -diff --git a/src/threadweave/incremental.py b/src/threadweave/incremental.py -index 674d097..44e5616 100644 ---- a/src/threadweave/incremental.py -+++ b/src/threadweave/incremental.py -@@ -10,11 +10,11 @@ payloads. - from __future__ import annotations - - import json --from collections.abc import Iterable, Mapping, Sequence -+from collections.abc import Iterable, Iterator, Mapping, Sequence - from dataclasses import dataclass - from datetime import datetime - from _thread import RLock --from typing import Literal -+from typing import Literal, TypeVar - - from threadweave.collation import unicode_casemap_key - from threadweave.container import Container -@@ -45,6 +45,57 @@ _DEFAULT_MAX_SNAPSHOT_RECORDS = 1_000_000 - _DEFAULT_MAX_SNAPSHOT_BYTES = 256 * 1024 * 1024 - _SNAPSHOT_SCHEMA_VERSION = 1 - -+_KeyT = TypeVar("_KeyT") -+_ValueT = TypeVar("_ValueT") -+ -+ -+class _OverlayMapping(Mapping[_KeyT, _ValueT]): -+ """Expose staged updates over a base mapping without copying unrelated entries.""" -+ -+ def __init__( -+ self, -+ base: Mapping[_KeyT, _ValueT], -+ updates: Mapping[_KeyT, _ValueT], -+ removals: frozenset[_KeyT] = frozenset(), -+ ) -> None: -+ """Retain immutable transaction views of one base mapping and its delta.""" -+ self._base = base -+ self._updates = updates -+ self._removals = removals -+ -+ def __getitem__(self, key: _KeyT) -> _ValueT: -+ """Return a staged value, excluding keys removed by the transaction.""" -+ if key in self._updates: -+ return self._updates[key] -+ if key in self._removals: -+ raise KeyError(key) -+ return self._base[key] -+ -+ def __iter__(self) -> Iterator[_KeyT]: -+ """Iterate the logical mapping only when a global operation requires it.""" -+ for key in self._base: -+ if key not in self._removals and key not in self._updates: -+ yield key -+ yield from self._updates -+ -+ def __len__(self) -> int: -+ """Return logical size from the bounded transaction delta.""" -+ removed = sum( -+ 1 -+ for key in self._removals -+ if key in self._base and key not in self._updates -+ ) -+ added = sum(1 for key in self._updates if key not in self._base) -+ return len(self._base) - removed + added -+ -+ def __contains__(self, key: object) -> bool: -+ """Test logical membership without iterating the base mapping.""" -+ if key in self._updates: -+ return True -+ if key in self._removals: -+ return False -+ return key in self._base -+ - - class IncrementalThreadError(ValueError): - """Raised when an incremental change or snapshot violates its contract.""" -@@ -369,47 +420,172 @@ def _connectivity_tokens( - return frozenset(tokens) - - --def _writable_token_bucket( -- token: str, -- keys_by_token: dict[str, set[str]], -- copied_tokens: set[str], -+def _writable_bucket( -+ bucket_key: str, -+ base_buckets: Mapping[str, set[str]], -+ bucket_updates: dict[str, set[str]], - ) -> set[str]: -- """Return one copy-on-write reverse bucket owned by the transaction.""" -- if token not in copied_tokens: -- keys_by_token[token] = set(keys_by_token.get(token, set())) -- copied_tokens.add(token) -- elif token not in keys_by_token: -- keys_by_token[token] = set() -- return keys_by_token[token] -+ """Return one transaction-owned copy of a reverse-index bucket.""" -+ if bucket_key not in bucket_updates: -+ bucket_updates[bucket_key] = set(base_buckets.get(bucket_key, set())) -+ return bucket_updates[bucket_key] - - - def _remove_key_from_buckets( - key: str, -- tokens_by_key: dict[str, frozenset[str]], -- keys_by_token: dict[str, set[str]], -- copied_tokens: set[str], --) -> frozenset[str]: -- """Remove one key from transaction-owned token buckets.""" -- old_tokens = tokens_by_key.pop(key, frozenset()) -- for token in old_tokens: -- bucket = _writable_token_bucket(token, keys_by_token, copied_tokens) -- bucket.discard(key) -- if not bucket: -- del keys_by_token[token] -- return old_tokens -+ bucket_keys: Iterable[str], -+ base_buckets: Mapping[str, set[str]], -+ bucket_updates: dict[str, set[str]], -+) -> None: -+ """Stage removal of one record key from selected reverse-index buckets.""" -+ for bucket_key in bucket_keys: -+ _writable_bucket(bucket_key, base_buckets, bucket_updates).discard(key) - - - def _add_key_to_buckets( - key: str, -- tokens: frozenset[str], -- tokens_by_key: dict[str, frozenset[str]], -- keys_by_token: dict[str, set[str]], -- copied_tokens: set[str], -+ bucket_keys: Iterable[str], -+ base_buckets: Mapping[str, set[str]], -+ bucket_updates: dict[str, set[str]], -+) -> None: -+ """Stage insertion of one record key into selected reverse-index buckets.""" -+ for bucket_key in bucket_keys: -+ _writable_bucket(bucket_key, base_buckets, bucket_updates).add(key) -+ -+ -+def _commit_bucket_updates( -+ target: dict[str, set[str]], -+ bucket_updates: Mapping[str, set[str]], - ) -> None: -- """Insert one key through transaction-owned copy-on-write buckets.""" -- tokens_by_key[key] = tokens -- for token in tokens: -- _writable_token_bucket(token, keys_by_token, copied_tokens).add(key) -+ """Publish touched reverse buckets and discard buckets that became empty.""" -+ for bucket_key, values in bucket_updates.items(): -+ if values: -+ target[bucket_key] = values -+ else: -+ target.pop(bucket_key, None) -+ -+ -+def _email_state_after( -+ email_id: str, -+ base_states: Mapping[str, tuple[str | None, int]], -+ state_updates: Mapping[str, tuple[str | None, int] | None], -+) -> tuple[str | None, int] | None: -+ """Return one staged EMAILID association and reference count.""" -+ if email_id in state_updates: -+ return state_updates[email_id] -+ return base_states.get(email_id) -+ -+ -+def _thread_count_after( -+ thread_id: str, -+ base_counts: Mapping[str, int], -+ count_updates: Mapping[str, int], -+) -> int: -+ """Return one staged THREADID reference count.""" -+ return count_updates.get(thread_id, base_counts.get(thread_id, 0)) -+ -+ -+def _stage_external_identity( -+ record: IndexedMessage, -+ adjustment: Literal[-1, 1], -+ base_email_states: Mapping[str, tuple[str | None, int]], -+ email_state_updates: dict[str, tuple[str | None, int] | None], -+ base_thread_counts: Mapping[str, int], -+ thread_count_updates: dict[str, int], -+ touched_values: set[str], -+) -> None: -+ """Stage one record's RFC 8474 identity contribution without a global scan.""" -+ email_id = record.email_id -+ if email_id is not None: -+ touched_values.add(email_id) -+ current_state = _email_state_after( -+ email_id, -+ base_email_states, -+ email_state_updates, -+ ) -+ if adjustment < 0: -+ if current_state is None or current_state[0] != record.thread_id: -+ raise IncrementalThreadError("internal EMAILID index is inconsistent") -+ next_count = current_state[1] - 1 -+ email_state_updates[email_id] = ( -+ None -+ if next_count == 0 -+ else (current_state[0], next_count) -+ ) -+ else: -+ if current_state is not None and current_state[0] != record.thread_id: -+ raise ExternalIdentityError( -+ f"messages with EMAILID {email_id!r} must expose the same THREADID" -+ ) -+ email_state_updates[email_id] = ( -+ record.thread_id, -+ 1 if current_state is None else current_state[1] + 1, -+ ) -+ -+ thread_id = record.thread_id -+ if thread_id is not None: -+ touched_values.add(thread_id) -+ next_count = ( -+ _thread_count_after( -+ thread_id, -+ base_thread_counts, -+ thread_count_updates, -+ ) -+ + adjustment -+ ) -+ if next_count < 0: -+ raise IncrementalThreadError("internal THREADID index is inconsistent") -+ thread_count_updates[thread_id] = next_count -+ -+ -+def _validate_touched_identity_namespaces( -+ touched_values: Iterable[str], -+ base_email_states: Mapping[str, tuple[str | None, int]], -+ email_state_updates: Mapping[str, tuple[str | None, int] | None], -+ base_thread_counts: Mapping[str, int], -+ thread_count_updates: Mapping[str, int], -+) -> None: -+ """Reject touched ObjectID values present in both RFC 8474 namespaces.""" -+ reused_values = sorted( -+ identity_value -+ for identity_value in touched_values -+ if _email_state_after( -+ identity_value, -+ base_email_states, -+ email_state_updates, -+ ) -+ is not None -+ and _thread_count_after( -+ identity_value, -+ base_thread_counts, -+ thread_count_updates, -+ ) -+ > 0 -+ ) -+ if reused_values: -+ raise ExternalIdentityError( -+ "EMAILID and THREADID must use disjoint ObjectID values: " -+ f"{reused_values!r}" -+ ) -+ -+ -+def _commit_external_identity_updates( -+ email_states: dict[str, tuple[str | None, int]], -+ email_state_updates: Mapping[str, tuple[str | None, int] | None], -+ thread_counts: dict[str, int], -+ thread_count_updates: Mapping[str, int], -+) -> None: -+ """Publish touched compact RFC 8474 indexes after transaction validation.""" -+ for email_id, state in email_state_updates.items(): -+ if state is None: -+ email_states.pop(email_id, None) -+ else: -+ email_states[email_id] = state -+ for thread_id, count in thread_count_updates.items(): -+ if count == 0: -+ thread_counts.pop(thread_id, None) -+ else: -+ thread_counts[thread_id] = count - - - def _ordered_keys(keys: Iterable[str], positions: Mapping[str, int]) -> tuple[str, ...]: -@@ -442,32 +568,6 @@ def _validate_effective_sequence_numbers( - used[sequence_number] = key - - --def _validate_external_identities(records: Mapping[str, IndexedMessage]) -> None: -- """Enforce RFC 8474 EMAILID/THREADID consistency and namespace separation.""" -- thread_id_by_email_id: dict[str, str | None] = {} -- email_ids: set[str] = set() -- thread_ids: set[str] = set() -- for record in records.values(): -- email_id = record.email_id -- thread_id = record.thread_id -- if email_id is not None: -- email_ids.add(email_id) -- if email_id not in thread_id_by_email_id: -- thread_id_by_email_id[email_id] = thread_id -- elif thread_id_by_email_id[email_id] != thread_id: -- raise ExternalIdentityError( -- f"messages with EMAILID {email_id!r} must expose the same THREADID" -- ) -- if thread_id is not None: -- thread_ids.add(thread_id) -- reused_values = email_ids & thread_ids -- if reused_values: -- raise ExternalIdentityError( -- "EMAILID and THREADID must use disjoint ObjectID values: " -- f"{sorted(reused_values)!r}" -- ) -- -- - def _validate_replacement_identity( - old_record: IndexedMessage, - new_record: IndexedMessage, -@@ -965,6 +1065,8 @@ class IncrementalThreadIndex: - self._next_position = 1 - self._tokens_by_key: dict[str, frozenset[str]] = {} - self._keys_by_token: dict[str, set[str]] = {} -+ self._email_id_states: dict[str, tuple[str | None, int]] = {} -+ self._thread_id_counts: dict[str, int] = {} - self._component_by_key: dict[str, str] = {} - self._keys_by_component: dict[str, tuple[str, ...]] = {} - self._roots: tuple[Container, ...] | None = () -@@ -1065,10 +1167,11 @@ class IncrementalThreadIndex: - addition_keys = {record.message_key for record in change_set.additions} - replacement_keys = {record.message_key for record in change_set.replacements} - removal_keys = set(change_set.removals) -- existing_keys = set(self._records) -- already_present = addition_keys & existing_keys -- missing_replacements = replacement_keys - existing_keys -- missing_removals = removal_keys - existing_keys -+ already_present = {key for key in addition_keys if key in self._records} -+ missing_replacements = { -+ key for key in replacement_keys if key not in self._records -+ } -+ missing_removals = {key for key in removal_keys if key not in self._records} - if already_present: - raise IncrementalThreadError( - f"addition keys already exist: {sorted(already_present)!r}" -@@ -1094,64 +1197,103 @@ class IncrementalThreadIndex: - replacement, - ) - -- records = dict(self._records) -- positions = dict(self._positions) -- tokens_by_key = dict(self._tokens_by_key) -- keys_by_token = dict(self._keys_by_token) -- copied_tokens: set[str] = set() -+ record_updates = { -+ record.message_key: record -+ for record in (*copied_replacements, *copied_additions) -+ } -+ records = _OverlayMapping( -+ self._records, -+ record_updates, -+ frozenset(removal_keys), -+ ) -+ position_updates: dict[str, int] = {} - next_position = self._next_position -+ for addition in copied_additions: -+ position_updates[addition.message_key] = next_position -+ next_position += 1 -+ positions = _OverlayMapping( -+ self._positions, -+ position_updates, -+ frozenset(removal_keys), -+ ) -+ -+ token_updates: dict[str, frozenset[str]] = {} -+ token_bucket_updates: dict[str, set[str]] = {} -+ email_state_updates: dict[str, tuple[str | None, int] | None] = {} -+ thread_count_updates: dict[str, int] = {} - changed_existing_keys = replacement_keys | removal_keys - candidate_seeds: set[str] = set() - touched_tokens: set[str] = set() -+ touched_identity_values: set[str] = set() - - for key in changed_existing_keys: - component_id = self._component_by_key.get(key) - if component_id is not None: - candidate_seeds.update(self._keys_by_component[component_id]) -- touched_tokens.update( -- _remove_key_from_buckets( -- key, -- tokens_by_key, -- keys_by_token, -- copied_tokens, -- ) -+ old_tokens = self._tokens_by_key.get(key, frozenset()) -+ touched_tokens.update(old_tokens) -+ _remove_key_from_buckets( -+ key, -+ old_tokens, -+ self._keys_by_token, -+ token_bucket_updates, -+ ) -+ _stage_external_identity( -+ self._records[key], -+ -1, -+ self._email_id_states, -+ email_state_updates, -+ self._thread_id_counts, -+ thread_count_updates, -+ touched_identity_values, - ) - -- for key in removal_keys: -- records.pop(key) -- positions.pop(key) -- for replacement in copied_replacements: -- records[replacement.message_key] = replacement -+ for record in (*copied_replacements, *copied_additions): - tokens = _connectivity_tokens( -- replacement, -+ record, - group_by_subject=self._group_by_subject, - ) -+ token_updates[record.message_key] = tokens - touched_tokens.update(tokens) - _add_key_to_buckets( -- replacement.message_key, -+ record.message_key, - tokens, -- tokens_by_key, -- keys_by_token, -- copied_tokens, -+ self._keys_by_token, -+ token_bucket_updates, - ) -- candidate_seeds.add(replacement.message_key) -- for addition in copied_additions: -- records[addition.message_key] = addition -- positions[addition.message_key] = next_position -- next_position += 1 -- tokens = _connectivity_tokens( -- addition, -- group_by_subject=self._group_by_subject, -+ _stage_external_identity( -+ record, -+ 1, -+ self._email_id_states, -+ email_state_updates, -+ self._thread_id_counts, -+ thread_count_updates, -+ touched_identity_values, - ) -- touched_tokens.update(tokens) -- _add_key_to_buckets( -- addition.message_key, -- tokens, -- tokens_by_key, -- keys_by_token, -- copied_tokens, -- ) -- candidate_seeds.add(addition.message_key) -+ candidate_seeds.add(record.message_key) -+ -+ tokens_by_key = _OverlayMapping( -+ self._tokens_by_key, -+ token_updates, -+ frozenset(removal_keys), -+ ) -+ keys_by_token = _OverlayMapping(self._keys_by_token, token_bucket_updates) - - for token in touched_tokens: - candidate_seeds.update(keys_by_token.get(token, set())) -@@ -1186,14 +1328,19 @@ class IncrementalThreadIndex: - group_by_subject=self._group_by_subject, - sort_by_sent_date=self._sort_by_sent_date, - ) - -- _validate_external_identities(records) -+ _validate_touched_identity_namespaces( -+ touched_identity_values, -+ self._email_id_states, -+ email_state_updates, -+ self._thread_id_counts, -+ thread_count_updates, -+ ) - ranks = ( - _current_ranks(positions) - if self._sort_by_sent_date - else positions - ) - if self._sort_by_sent_date: - _validate_effective_sequence_numbers(records, ranks) -@@ -1200,30 +1347,14 @@ class IncrementalThreadIndex: - tokens_by_key, - keys_by_token, - ) -- old_candidate_keys = set(candidate_seeds) -- candidate_keys = current_candidate_keys | old_candidate_keys -- -- component_by_key = { -- key: component_id -- for key, component_id in self._component_by_key.items() -- if key not in candidate_keys and key in records -- } -- unaffected_component_ids = set(component_by_key.values()) -- keys_by_component = { -- component_id: keys -- for component_id, keys in self._keys_by_component.items() -- if component_id in unaffected_component_ids -- } -- for keys in _partition_components( -+ candidate_keys = current_candidate_keys | set(candidate_seeds) -+ new_components = _partition_components( - current_candidate_keys, - positions, - tokens_by_key, - keys_by_token, -- ): -- component_id = keys[0] -- keys_by_component[component_id] = keys -- for key in keys: -- component_by_key[key] = component_id -+ ) - - after_affected_keys = _ordered_keys(current_candidate_keys, positions) - _, after_affected_projections = _build_forest( -@@ -1250,13 +1381,31 @@ class IncrementalThreadIndex: - after_affected_projections, - ) - -- self._records = records -- self._positions = positions -- self._next_position = next_position -- self._tokens_by_key = tokens_by_key -- self._keys_by_token = keys_by_token -- self._component_by_key = component_by_key -- self._keys_by_component = keys_by_component -+ for key in removal_keys: -+ self._records.pop(key) -+ self._positions.pop(key) -+ self._tokens_by_key.pop(key, None) -+ self._records.update(record_updates) -+ self._positions.update(position_updates) -+ self._tokens_by_key.update(token_updates) -+ _commit_bucket_updates(self._keys_by_token, token_bucket_updates) -+ _commit_external_identity_updates( -+ self._email_id_states, -+ email_state_updates, -+ self._thread_id_counts, -+ thread_count_updates, -+ ) -+ -+ for component_id in old_component_ids: -+ self._keys_by_component.pop(component_id, None) -+ for key in candidate_keys: -+ self._component_by_key.pop(key, None) -+ for keys in new_components: -+ component_id = keys[0] -+ self._keys_by_component[component_id] = keys -+ for key in keys: -+ self._component_by_key[key] = component_id -+ -+ self._next_position = next_position - self._roots = None - self._projections = None - self._version = version -diff --git a/tests/test_incremental_private_graph.py b/tests/test_incremental_private_graph.py -index 325ce7a..99bbe6c 100644 ---- a/tests/test_incremental_private_graph.py -+++ b/tests/test_incremental_private_graph.py -@@ -153,30 +153,29 @@ def test_reverse_token_buckets_are_copied_only_when_mutated(): - """Atomic changes preserve every shared pre-transaction bucket.""" - original = {"token": {"a", "b"}} -- overlay = dict(original) -- tokens_by_key = {"a": frozenset({"token"})} -- copied_tokens: set[str] = set() -+ updates: dict[str, set[str]] = {} - - incremental._remove_key_from_buckets( - "a", -- tokens_by_key, -- overlay, -- copied_tokens, -+ ("token",), -+ original, -+ updates, - ) - incremental._add_key_to_buckets( - "c", -- frozenset({"token"}), -- tokens_by_key, -- overlay, -- copied_tokens, -+ ("token",), -+ original, -+ updates, - ) - - assert original == {"token": {"a", "b"}} -- assert overlay == {"token": {"b", "c"}} -- assert copied_tokens == {"token"} -+ assert updates == {"token": {"b", "c"}} -+ incremental._commit_bucket_updates(original, updates) -+ assert original == {"token": {"b", "c"}} -+ incremental._commit_bucket_updates(original, {"token": set()}) -+ assert original == {} - - - def test_component_partition_reads_positions_linearly_for_disconnected_keys(): -@@ -238,3 +237,133 @@ def test_replacement_recovers_when_derived_component_mapping_is_missing(): - ) - assert index.projections[0].message_keys == ("a",) -+ -+ -+def test_default_delta_does_not_copy_or_iterate_unrelated_state_maps(): -+ """A small default-mode update touches bounded state instead of the whole mailbox.""" -+ -+ class NoFullIterationDict(dict[str, object]): -+ """Permit keyed access and mutation while rejecting whole-map scans.""" -+ -+ def __iter__(self): -+ """Reject direct iteration over unrelated state.""" -+ raise AssertionError("unexpected full-state iteration") -+ -+ def keys(self): -+ """Reject key-view scans over unrelated state.""" -+ raise AssertionError("unexpected full-state key scan") -+ -+ def items(self): -+ """Reject item-view scans over unrelated state.""" -+ raise AssertionError("unexpected full-state item scan") -+ -+ def values(self): -+ """Reject value-view scans over unrelated state.""" -+ raise AssertionError("unexpected full-state value scan") -+ -+ records = tuple( -+ _record( -+ f"message_{index}", -+ Message(message_id=f"message_{index}"), -+ email_id=f"Email_{index}", -+ thread_id=f"Thread_{index}", -+ ) -+ for index in range(128) -+ ) -+ index = IncrementalThreadIndex() -+ index.apply(MailboxChangeSet(expected_version=0, additions=records)) -+ -+ state_names = ( -+ "_records", -+ "_positions", -+ "_tokens_by_key", -+ "_keys_by_token", -+ "_email_id_states", -+ "_thread_id_counts", -+ "_component_by_key", -+ "_keys_by_component", -+ ) -+ for name in state_names: -+ setattr(index, name, NoFullIterationDict(getattr(index, name))) -+ state_identities = {name: id(getattr(index, name)) for name in state_names} -+ -+ delta = index.apply( -+ MailboxChangeSet( -+ expected_version=1, -+ replacements=( -+ _record( -+ "message_0", -+ Message(message_id="message_0", subject="updated"), -+ email_id="Email_0", -+ thread_id="Thread_0", -+ ), -+ ), -+ ) -+ ) -+ -+ assert delta.affected_message_keys == ("message_0",) -+ assert index.version == 2 -+ assert {name: id(getattr(index, name)) for name in state_names} == state_identities -+ -+ -+def test_external_identity_indexes_update_only_touched_namespaces(): -+ """Identity removals and additions remain atomic without a mailbox-wide scan.""" -+ index = IncrementalThreadIndex() -+ index.apply( -+ MailboxChangeSet( -+ expected_version=0, -+ additions=( -+ _record("a", email_id="Mail_A", thread_id="Thread_A"), -+ _record("b", email_id="Mail_B", thread_id="Thread_B"), -+ ), -+ ) -+ ) -+ -+ index.apply( -+ MailboxChangeSet( -+ expected_version=1, -+ removals=("a",), -+ additions=( -+ _record("c", email_id="Thread_A", thread_id="Mail_A"), -+ ), -+ ) -+ ) -+ -+ assert index._email_id_states == { -+ "Mail_B": ("Thread_B", 1), -+ "Thread_A": ("Mail_A", 1), -+ } -+ assert index._thread_id_counts == {"Thread_B": 1, "Mail_A": 1} -+ -+ before = index.snapshot() -+ with pytest.raises(IncrementalThreadError, match="disjoint ObjectID"): -+ index.apply( -+ MailboxChangeSet( -+ expected_version=2, -+ additions=( -+ _record("d", email_id="Thread_B", thread_id="Thread_D"), -+ ), -+ ) -+ ) -+ assert index.snapshot() == before -+ -+ -+def test_external_identity_index_corruption_fails_before_commit(): -+ """Broken private identity indexes cannot be published by a transaction.""" -+ index = IncrementalThreadIndex() -+ index.apply( -+ MailboxChangeSet( -+ expected_version=0, -+ additions=( -+ _record("a", email_id="Mail_A", thread_id="Thread_A"), -+ ), -+ ) -+ ) -+ index._email_id_states.clear() -+ with pytest.raises(IncrementalThreadError, match="EMAILID index"): -+ index.apply(MailboxChangeSet(expected_version=1, removals=("a",))) -+ -+ index._email_id_states["Mail_A"] = ("Thread_A", 1) -+ index._thread_id_counts.clear() -+ with pytest.raises(IncrementalThreadError, match="THREADID index"): -+ index.apply(MailboxChangeSet(expected_version=1, removals=("a",))) From 5a4621613a95b163130f2b14e8bdcd0c70854467 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:06:27 +0900 Subject: [PATCH 84/97] ci: remove completed bounded-overlay repair verifier --- .../workflows/repair-pr20-bounded-patch.yml | 473 ------------------ 1 file changed, 473 deletions(-) delete mode 100644 .github/workflows/repair-pr20-bounded-patch.yml diff --git a/.github/workflows/repair-pr20-bounded-patch.yml b/.github/workflows/repair-pr20-bounded-patch.yml deleted file mode 100644 index 7766e63..0000000 --- a/.github/workflows/repair-pr20-bounded-patch.yml +++ /dev/null @@ -1,473 +0,0 @@ -name: Repair and apply PR 20 bounded state overlays - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - .github/workflows/repair-pr20-bounded-patch.yml - -permissions: - contents: write - -concurrency: - group: repair-pr20-bounded-state-overlays - cancel-in-progress: false - -jobs: - red-green-verify-and-apply: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' - runs-on: ubuntu-24.04 - timeout-minutes: 45 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Recalculate every unified-diff hunk count and validate the patch - shell: bash - run: | - set -euo pipefail - test "$(git hash-object src/threadweave/incremental.py)" = \ - '674d0978648c90d81e35a845b441fa367bb424d2' - test "$(git hash-object tests/test_incremental_private_graph.py)" = \ - '325ce7a1b07a7f0cf6674bcde93e03d5aa6e5fd6' - test "$(git hash-object CHANGELOG.md)" = \ - '78cec06aee3d679b8fe2d19f801de64ee7117ed3' - test "$(git hash-object README.md)" = \ - '648b017a8ceebe28db8739419541d7d4a1482d6e' - test "$(git hash-object docs/incremental-threading.md)" = \ - 'e0851f170870542030fff2ef34c71763e38e969d' - test "$(git hash-object ARCHITECTURE.md)" = \ - 'e7c1badd6155e85b83729236723e30e6487d2b4e' - test "$(git hash-object tools/pr20-bounded-state-overlays.patch)" = \ - '58f0da771cfc692f863d0d8381080f92574ae1bf' - python - <<'PYCODE' - from __future__ import annotations - - import re - from pathlib import Path - - patch_path = Path("tools/pr20-bounded-state-overlays.patch") - lines = patch_path.read_text(encoding="utf-8").splitlines(keepends=True) - header_pattern = re.compile( - r"^@@ -(\d+)(?:,\d+)? \+(\d+)(?:,\d+)? @@(.*?)(\n)?$" - ) - repaired: list[str] = [] - index = 0 - repaired_hunks = 0 - while index < len(lines): - line = lines[index] - match = header_pattern.match(line) - if match is None: - repaired.append(line) - index += 1 - continue - - body_start = index + 1 - body_end = body_start - while body_end < len(lines): - candidate = lines[body_end] - if candidate.startswith("@@ ") or candidate.startswith("diff --git "): - break - body_end += 1 - - body = lines[body_start:body_end] - old_count = sum( - 1 for body_line in body if body_line.startswith((" ", "-")) - ) - new_count = sum( - 1 for body_line in body if body_line.startswith((" ", "+")) - ) - repaired.append( - f"@@ -{match.group(1)},{old_count} " - f"+{match.group(2)},{new_count} @@" - f"{match.group(3)}{match.group(4) or ''}" - ) - repaired.extend(body) - repaired_hunks += 1 - index = body_end - - if repaired_hunks < 10: - raise SystemExit(f"unexpected hunk count: {repaired_hunks}") - patch_path.write_text("".join(repaired), encoding="utf-8") - PYCODE - git apply --check tools/pr20-bounded-state-overlays.patch - - - name: Prove default updates scan unrelated state before the fix - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - from threadweave import ( - IncrementalThreadIndex, - IndexedMessage, - MailboxChangeSet, - Message, - ) - - class NoFullIterationDict(dict): - def __iter__(self): - raise AssertionError("unexpected full-state iteration") - - def keys(self): - raise AssertionError("unexpected full-state key scan") - - def items(self): - raise AssertionError("unexpected full-state item scan") - - def values(self): - raise AssertionError("unexpected full-state value scan") - - records = tuple( - IndexedMessage( - f"message_{index}", - Message(message_id=f"message_{index}"), - ) - for index in range(256) - ) - index = IncrementalThreadIndex() - index.apply(MailboxChangeSet(expected_version=0, additions=records)) - for name in ( - "_records", - "_positions", - "_tokens_by_key", - "_keys_by_token", - "_component_by_key", - "_keys_by_component", - ): - setattr(index, name, NoFullIterationDict(getattr(index, name))) - - try: - index.apply( - MailboxChangeSet( - expected_version=1, - replacements=( - IndexedMessage( - "message_0", - Message(message_id="message_0", subject="updated"), - ), - ), - ) - ) - except AssertionError as error: - if "full-state" not in str(error): - raise - else: - raise SystemExit("expected the pre-fix delta to scan unrelated state") - PYCODE - echo 'Observed the expected pre-fix mailbox-wide state scan.' \ - >>"$GITHUB_STEP_SUMMARY" - - - name: Apply the bounded transaction overlay patch - shell: bash - run: | - set -euo pipefail - git apply tools/pr20-bounded-state-overlays.patch - rm tools/pr20-bounded-state-overlays.patch - rmdir tools 2>/dev/null || true - git diff --check - - - name: Verify focused state, identity, component, and RFC behavior - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - pytest -q \ - tests/test_incremental_private_graph.py \ - tests/test_incremental_rfc8474.py \ - tests/test_incremental_components.py \ - tests/test_incremental_parity.py \ - tests/test_incremental_concurrency.py - - - name: Verify 100,000-message default delta allocations are bounded - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - import gc - import json - import os - import time - import tracemalloc - - from threadweave import ( - IncrementalThreadIndex, - IndexedMessage, - MailboxChangeSet, - Message, - ) - - message_count = 100_000 - records = tuple( - IndexedMessage( - f"key_{index}", - Message(message_id=f"message_{index}"), - ) - for index in range(message_count) - ) - index = IncrementalThreadIndex() - index.apply(MailboxChangeSet(expected_version=0, additions=records)) - gc.collect() - tracemalloc.start() - baseline_current, _ = tracemalloc.get_traced_memory() - started = time.perf_counter() - delta = index.apply( - MailboxChangeSet( - expected_version=1, - replacements=( - IndexedMessage( - "key_0", - Message(message_id="message_0", subject="updated"), - ), - ), - ) - ) - elapsed_seconds = time.perf_counter() - started - current_bytes, peak_bytes = tracemalloc.get_traced_memory() - tracemalloc.stop() - transient_peak_bytes = peak_bytes - baseline_current - retained_bytes = current_bytes - baseline_current - if delta.affected_message_keys != ("key_0",): - raise SystemExit(delta.affected_message_keys) - if transient_peak_bytes >= 1_000_000: - raise SystemExit( - f"default delta allocated {transient_peak_bytes} transient bytes" - ) - evidence = { - "message_count": message_count, - "affected_message_count": len(delta.affected_message_keys), - "delta_apply_seconds": elapsed_seconds, - "retained_delta_bytes": retained_bytes, - "transient_peak_delta_bytes": transient_peak_bytes, - } - print(json.dumps(evidence, sort_keys=True)) - with open(os.environ["GITHUB_STEP_SUMMARY"], "a", encoding="utf-8") as summary: - summary.write("## Bounded default delta evidence\n\n") - summary.write("```json\n") - summary.write(json.dumps(evidence, indent=2, sort_keys=True)) - summary.write("\n```\n") - PYCODE - - - name: Verify randomized canonical parity - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - import random - - from threadweave import ( - IncrementalThreadIndex, - IndexedMessage, - MailboxChangeSet, - Message, - thread_messages, - ) - - def projection(roots): - result = [] - for root in roots: - keys = [] - seen = set() - stack = [root] - while stack: - node = stack.pop() - if id(node) in seen: - continue - seen.add(id(node)) - if node.message is not None: - keys.append(node.message.payload) - stack.extend(reversed(node.children)) - result.append(tuple(keys)) - return tuple(result) - - transition_count = 0 - subjects = (None, "Topic", "Re: Topic", "Ｔｏｐｉｃ", "Other") - dates = ( - None, - "1 Jan 2026 00:00:00 +0000", - "2 Jan 2026 09:00:00 +0900", - "bad date", - ) - for group_by_subject in (False, True): - for sort_by_sent_date in (False, True): - for seed in range(4): - source = random.Random( - 801_000 - + seed - + (100 if group_by_subject else 0) - + (1_000 if sort_by_sent_date else 0) - ) - index = IncrementalThreadIndex( - group_by_subject=group_by_subject, - sort_by_sent_date=sort_by_sent_date, - ) - records = {} - ordered_keys = [] - next_key = 0 - for step in range(160): - roll = source.random() - if not records or roll < 0.48: - key = f"seed_{seed}_message_{next_key}" - next_key += 1 - operation = "add" - elif roll < 0.78: - key = source.choice(ordered_keys) - operation = "replace" - else: - key = source.choice(ordered_keys) - operation = "remove" - - if operation != "remove": - existing_ids = [ - record.message.message_id - for record in records.values() - if record.message.message_id is not None - ] - mode = source.randrange(6) - if mode == 0: - message_id = None - elif mode == 1 and existing_ids: - message_id = source.choice(existing_ids) - else: - message_id = f"{key}@example.test" - candidates = existing_ids + [ - f"missing_{item}@example.test" - for item in range(3) - ] - references = ( - tuple( - source.choice(candidates) - for _ in range(source.randrange(3)) - ) - if candidates - else () - ) - record = IndexedMessage( - key, - Message( - message_id=message_id, - references=references, - subject=source.choice(subjects), - sent_date=source.choice(dates), - payload=key, - ), - email_id=f"Email_{key}", - thread_id=f"Thread_{key}", - ) - - if operation == "add": - change = MailboxChangeSet( - index.version, - additions=(record,), - ) - records[key] = record - ordered_keys.append(key) - elif operation == "replace": - change = MailboxChangeSet( - index.version, - replacements=(record,), - ) - records[key] = record - else: - change = MailboxChangeSet( - index.version, - removals=(key,), - ) - del records[key] - ordered_keys.remove(key) - - index.apply(change) - expected = projection( - thread_messages( - (records[key].message for key in ordered_keys), - group_by_subject=group_by_subject, - sort_by_sent_date=sort_by_sent_date, - ) - ) - observed = tuple( - item.message_keys for item in index.projections - ) - if observed != expected: - raise SystemExit( - (group_by_subject, sort_by_sent_date, seed, step) - ) - transition_count += 1 - print(f"validated {transition_count} randomized transitions") - PYCODE - - - name: Verify the complete repository - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit only verified product changes - shell: bash - run: | - set -euo pipefail - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add \ - src/threadweave/incremental.py \ - tests/test_incremental_private_graph.py \ - CHANGELOG.md \ - README.md \ - docs/incremental-threading.md \ - ARCHITECTURE.md - git add -u tools/pr20-bounded-state-overlays.patch - git diff --cached --check - if git diff --cached --name-only | grep -q '^\.github/workflows/'; then - echo 'Refusing to publish workflow changes from the one-shot verifier.' >&2 - exit 1 - fi - git commit -m 'perf: bound default incremental state updates' - git push origin HEAD:feature/incremental-thread-index From 46e3c5d43c249e3b39197940cd5cbdffb1f7d7a2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:06:42 +0900 Subject: [PATCH 85/97] ci: remove completed bounded-overlay verifier --- .../apply-pr20-bounded-state-overlays.yml | 414 ------------------ 1 file changed, 414 deletions(-) delete mode 100644 .github/workflows/apply-pr20-bounded-state-overlays.yml diff --git a/.github/workflows/apply-pr20-bounded-state-overlays.yml b/.github/workflows/apply-pr20-bounded-state-overlays.yml deleted file mode 100644 index 0d1eb08..0000000 --- a/.github/workflows/apply-pr20-bounded-state-overlays.yml +++ /dev/null @@ -1,414 +0,0 @@ -name: Apply PR 20 bounded state overlays - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - .github/workflows/apply-pr20-bounded-state-overlays.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-bounded-state-overlays - cancel-in-progress: false - -jobs: - red-green-verify: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' - runs-on: ubuntu-24.04 - timeout-minutes: 35 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Verify the reviewed baseline and patch - shell: bash - run: | - set -euo pipefail - test "$(git hash-object src/threadweave/incremental.py)" = \ - '674d0978648c90d81e35a845b441fa367bb424d2' - test "$(git hash-object tests/test_incremental_private_graph.py)" = \ - '325ce7a1b07a7f0cf6674bcde93e03d5aa6e5fd6' - test "$(git hash-object CHANGELOG.md)" = \ - '78cec06aee3d679b8fe2d19f801de64ee7117ed3' - test "$(git hash-object README.md)" = \ - '648b017a8ceebe28db8739419541d7d4a1482d6e' - test "$(git hash-object docs/incremental-threading.md)" = \ - 'e0851f170870542030fff2ef34c71763e38e969d' - test "$(git hash-object ARCHITECTURE.md)" = \ - 'e7c1badd6155e85b83729236723e30e6487d2b4e' - test "$(git hash-object tools/pr20-bounded-state-overlays.patch)" = \ - '58f0da771cfc692f863d0d8381080f92574ae1bf' - git apply --check tools/pr20-bounded-state-overlays.patch - - - name: Prove default updates scan and replace mailbox-wide maps before the fix - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - from threadweave import ( - IncrementalThreadIndex, - IndexedMessage, - MailboxChangeSet, - Message, - ) - - class NoFullIterationDict(dict): - def __iter__(self): - raise AssertionError("unexpected full-state iteration") - - def keys(self): - raise AssertionError("unexpected full-state key scan") - - def items(self): - raise AssertionError("unexpected full-state item scan") - - def values(self): - raise AssertionError("unexpected full-state value scan") - - records = tuple( - IndexedMessage( - f"message_{index}", - Message(message_id=f"message_{index}"), - ) - for index in range(256) - ) - index = IncrementalThreadIndex() - index.apply(MailboxChangeSet(expected_version=0, additions=records)) - for name in ( - "_records", - "_positions", - "_tokens_by_key", - "_keys_by_token", - "_component_by_key", - "_keys_by_component", - ): - setattr(index, name, NoFullIterationDict(getattr(index, name))) - - try: - index.apply( - MailboxChangeSet( - expected_version=1, - replacements=( - IndexedMessage( - "message_0", - Message(message_id="message_0", subject="updated"), - ), - ), - ) - ) - except AssertionError as error: - if "full-state" not in str(error): - raise - else: - raise SystemExit("expected the pre-fix delta to scan unrelated state") - PYCODE - echo 'Observed the expected pre-fix mailbox-wide state scan.' \ - >>"$GITHUB_STEP_SUMMARY" - - - name: Apply the bounded transaction overlay patch - shell: bash - run: | - set -euo pipefail - git apply tools/pr20-bounded-state-overlays.patch - rm tools/pr20-bounded-state-overlays.patch - rmdir tools 2>/dev/null || true - git diff --check - - - name: Verify focused state, identity, component, and RFC behavior - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - pytest -q \ - tests/test_incremental_private_graph.py \ - tests/test_incremental_rfc8474.py \ - tests/test_incremental_components.py \ - tests/test_incremental_parity.py \ - tests/test_incremental_concurrency.py - - - name: Verify 100,000-message default delta allocations are bounded - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - import gc - import json - import time - import tracemalloc - - from threadweave import ( - IncrementalThreadIndex, - IndexedMessage, - MailboxChangeSet, - Message, - ) - - message_count = 100_000 - records = tuple( - IndexedMessage( - f"key_{index}", - Message(message_id=f"message_{index}"), - ) - for index in range(message_count) - ) - index = IncrementalThreadIndex() - index.apply(MailboxChangeSet(expected_version=0, additions=records)) - gc.collect() - tracemalloc.start() - baseline_current, _ = tracemalloc.get_traced_memory() - started = time.perf_counter() - delta = index.apply( - MailboxChangeSet( - expected_version=1, - replacements=( - IndexedMessage( - "key_0", - Message(message_id="message_0", subject="updated"), - ), - ), - ) - ) - elapsed_seconds = time.perf_counter() - started - current_bytes, peak_bytes = tracemalloc.get_traced_memory() - tracemalloc.stop() - transient_peak_bytes = peak_bytes - baseline_current - retained_bytes = current_bytes - baseline_current - if delta.affected_message_keys != ("key_0",): - raise SystemExit(delta.affected_message_keys) - if transient_peak_bytes >= 1_000_000: - raise SystemExit( - f"default delta allocated {transient_peak_bytes} transient bytes" - ) - evidence = { - "message_count": message_count, - "affected_message_count": len(delta.affected_message_keys), - "delta_apply_seconds": elapsed_seconds, - "retained_delta_bytes": retained_bytes, - "transient_peak_delta_bytes": transient_peak_bytes, - } - print(json.dumps(evidence, sort_keys=True)) - with open("$GITHUB_STEP_SUMMARY", "a", encoding="utf-8") as summary: - summary.write("## Bounded default delta evidence\n\n") - summary.write("```json\n") - summary.write(json.dumps(evidence, indent=2, sort_keys=True)) - summary.write("\n```\n") - PYCODE - - - name: Verify bounded randomized batch and identity parity - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - import random - - from threadweave import ( - IncrementalThreadIndex, - IndexedMessage, - MailboxChangeSet, - Message, - thread_messages, - ) - - def projection(roots): - result = [] - for root in roots: - keys = [] - seen = set() - stack = [root] - while stack: - node = stack.pop() - if id(node) in seen: - continue - seen.add(id(node)) - if node.message is not None: - keys.append(node.message.payload) - stack.extend(reversed(node.children)) - result.append(tuple(keys)) - return tuple(result) - - transition_count = 0 - subjects = (None, "Topic", "Re: Topic", "Ｔｏｐｉｃ", "Other") - dates = ( - None, - "1 Jan 2026 00:00:00 +0000", - "2 Jan 2026 09:00:00 +0900", - "bad date", - ) - for group_by_subject in (False, True): - for sort_by_sent_date in (False, True): - for seed in range(4): - source = random.Random( - 801_000 - + seed - + (100 if group_by_subject else 0) - + (1_000 if sort_by_sent_date else 0) - ) - index = IncrementalThreadIndex( - group_by_subject=group_by_subject, - sort_by_sent_date=sort_by_sent_date, - ) - records = {} - ordered_keys = [] - next_key = 0 - for step in range(160): - roll = source.random() - if not records or roll < 0.48: - key = f"seed_{seed}_message_{next_key}" - next_key += 1 - operation = "add" - elif roll < 0.78: - key = source.choice(ordered_keys) - operation = "replace" - else: - key = source.choice(ordered_keys) - operation = "remove" - - if operation != "remove": - existing_ids = [ - record.message.message_id - for record in records.values() - if record.message.message_id is not None - ] - mode = source.randrange(6) - if mode == 0: - message_id = None - elif mode == 1 and existing_ids: - message_id = source.choice(existing_ids) - else: - message_id = f"{key}@example.test" - candidates = existing_ids + [ - f"missing_{index}@example.test" - for index in range(3) - ] - references = tuple( - source.choice(candidates) - for _ in range(source.randrange(3)) - ) if candidates else () - record = IndexedMessage( - key, - Message( - message_id=message_id, - references=references, - subject=source.choice(subjects), - sent_date=source.choice(dates), - payload=key, - ), - email_id=f"Email_{key}", - thread_id=f"Thread_{key}", - ) - - if operation == "add": - change = MailboxChangeSet( - index.version, - additions=(record,), - ) - records[key] = record - ordered_keys.append(key) - elif operation == "replace": - change = MailboxChangeSet( - index.version, - replacements=(record,), - ) - records[key] = record - else: - change = MailboxChangeSet( - index.version, - removals=(key,), - ) - del records[key] - ordered_keys.remove(key) - - index.apply(change) - expected = projection( - thread_messages( - (records[key].message for key in ordered_keys), - group_by_subject=group_by_subject, - sort_by_sent_date=sort_by_sent_date, - ) - ) - observed = tuple( - item.message_keys for item in index.projections - ) - if observed != expected: - raise SystemExit( - (group_by_subject, sort_by_sent_date, seed, step) - ) - transition_count += 1 - print(f"validated {transition_count} randomized transitions") - PYCODE - - - name: Verify the complete repository - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Commit the verified product update - shell: bash - run: | - set -euo pipefail - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add \ - src/threadweave/incremental.py \ - tests/test_incremental_private_graph.py \ - CHANGELOG.md \ - README.md \ - docs/incremental-threading.md \ - ARCHITECTURE.md - git add -u tools/pr20-bounded-state-overlays.patch - git diff --cached --check - git commit -m 'perf: bound default incremental state updates' - git push origin HEAD:feature/incremental-thread-index From 6e1b28992ab54aa14240a53a52207a113261547a Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:30:08 +0900 Subject: [PATCH 86/97] ci: diagnose hourly product workflow validation --- .../diagnose-hourly-product-workflow.yml | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 .github/workflows/diagnose-hourly-product-workflow.yml diff --git a/.github/workflows/diagnose-hourly-product-workflow.yml b/.github/workflows/diagnose-hourly-product-workflow.yml new file mode 100644 index 0000000..c5c25fc --- /dev/null +++ b/.github/workflows/diagnose-hourly-product-workflow.yml @@ -0,0 +1,44 @@ +name: Diagnose hourly product workflow + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/diagnose-hourly-product-workflow.yml + +permissions: + contents: read + +jobs: + actionlint: + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Check out the exact branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + persist-credentials: false + + - name: Install pinned actionlint + shell: bash + run: | + set -euo pipefail + archive="${RUNNER_TEMP}/actionlint.tar.gz" + curl -fsSL \ + -o "$archive" \ + https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz + printf '%s %s\n' \ + 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8 \ + "$archive" | sha256sum -c - + tar -xzf "$archive" -C "$RUNNER_TEMP" actionlint + + - name: Diagnose the hourly product-development workflow + shell: bash + run: | + set -o pipefail + "${RUNNER_TEMP}/actionlint" \ + -color never \ + .github/workflows/hourly-product-development.yml \ + 2>&1 | tee "${RUNNER_TEMP}/actionlint-output.txt" From 9ec4786be2a094aad4c53499ef48dd096a6a868c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:32:02 +0900 Subject: [PATCH 87/97] ci: correct actionlint diagnostic invocation --- .github/workflows/diagnose-hourly-product-workflow.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/diagnose-hourly-product-workflow.yml b/.github/workflows/diagnose-hourly-product-workflow.yml index c5c25fc..88723ef 100644 --- a/.github/workflows/diagnose-hourly-product-workflow.yml +++ b/.github/workflows/diagnose-hourly-product-workflow.yml @@ -39,6 +39,6 @@ jobs: run: | set -o pipefail "${RUNNER_TEMP}/actionlint" \ - -color never \ + -color=false \ .github/workflows/hourly-product-development.yml \ 2>&1 | tee "${RUNNER_TEMP}/actionlint-output.txt" From 15f362e127003b607eb21bda64c2eb0808f9704f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:35:32 +0900 Subject: [PATCH 88/97] ci: diagnose every repository workflow --- .github/workflows/diagnose-hourly-product-workflow.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/diagnose-hourly-product-workflow.yml b/.github/workflows/diagnose-hourly-product-workflow.yml index 88723ef..87db9c3 100644 --- a/.github/workflows/diagnose-hourly-product-workflow.yml +++ b/.github/workflows/diagnose-hourly-product-workflow.yml @@ -1,4 +1,4 @@ -name: Diagnose hourly product workflow +name: Diagnose repository workflows on: push: @@ -34,11 +34,11 @@ jobs: "$archive" | sha256sum -c - tar -xzf "$archive" -C "$RUNNER_TEMP" actionlint - - name: Diagnose the hourly product-development workflow + - name: Diagnose every repository workflow shell: bash run: | set -o pipefail "${RUNNER_TEMP}/actionlint" \ -color=false \ - .github/workflows/hourly-product-development.yml \ + .github/workflows/*.yml \ 2>&1 | tee "${RUNNER_TEMP}/actionlint-output.txt" From 7165a4fb8b075d3761df2440d79b2d36c01f4bfe Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:38:27 +0900 Subject: [PATCH 89/97] ci: diagnose workflows after known YAML correction --- .../diagnose-hourly-product-workflow.yml | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/.github/workflows/diagnose-hourly-product-workflow.yml b/.github/workflows/diagnose-hourly-product-workflow.yml index 87db9c3..b4b4c52 100644 --- a/.github/workflows/diagnose-hourly-product-workflow.yml +++ b/.github/workflows/diagnose-hourly-product-workflow.yml @@ -34,7 +34,23 @@ jobs: "$archive" | sha256sum -c - tar -xzf "$archive" -C "$RUNNER_TEMP" actionlint - - name: Diagnose every repository workflow + - name: Apply the known YAML-indentation correction only in the diagnostic workspace + shell: bash + run: | + set -euo pipefail + python - <<'PY' + from pathlib import Path + + path = Path('.github/workflows/hourly-product-development.yml') + text = path.read_text(encoding='utf-8') + old = " printf 'threadweave @ file://%s \\\n --hash=sha256:%s\\n' \\\n" + new = " printf 'threadweave @ file://%s \\\n --hash=sha256:%s\\n' \\\n" + if text.count(old) != 1: + raise SystemExit('unexpected hourly workflow indentation state') + path.write_text(text.replace(old, new), encoding='utf-8') + PY + + - name: Diagnose every repository workflow after the known correction shell: bash run: | set -o pipefail From db85f2bc256a6bd3e0e69a58b420e689a8289948 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:41:07 +0900 Subject: [PATCH 90/97] ci: diagnose workflows after nested-shell correction --- .../diagnose-hourly-product-workflow.yml | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/.github/workflows/diagnose-hourly-product-workflow.yml b/.github/workflows/diagnose-hourly-product-workflow.yml index b4b4c52..b61432c 100644 --- a/.github/workflows/diagnose-hourly-product-workflow.yml +++ b/.github/workflows/diagnose-hourly-product-workflow.yml @@ -34,7 +34,7 @@ jobs: "$archive" | sha256sum -c - tar -xzf "$archive" -C "$RUNNER_TEMP" actionlint - - name: Apply the known YAML-indentation correction only in the diagnostic workspace + - name: Apply known corrections only in the diagnostic workspace shell: bash run: | set -euo pipefail @@ -43,14 +43,23 @@ jobs: path = Path('.github/workflows/hourly-product-development.yml') text = path.read_text(encoding='utf-8') - old = " printf 'threadweave @ file://%s \\\n --hash=sha256:%s\\n' \\\n" - new = " printf 'threadweave @ file://%s \\\n --hash=sha256:%s\\n' \\\n" - if text.count(old) != 1: + indentation_old = " printf 'threadweave @ file://%s \\\n --hash=sha256:%s\\n' \\\n" + indentation_new = " printf 'threadweave @ file://%s \\\n --hash=sha256:%s\\n' \\\n" + if text.count(indentation_old) != 1: raise SystemExit('unexpected hourly workflow indentation state') - path.write_text(text.replace(old, new), encoding='utf-8') + text = text.replace(indentation_old, indentation_new) + + shellcheck_old = " sudo -u '#65532' -g '#65532' env -i \\\n" + shellcheck_new = ( + " # shellcheck disable=SC2016 -- inner bash expands positional parameters.\n" + " sudo -u '#65532' -g '#65532' env -i \\\n" + ) + if text.count(shellcheck_old) != 1: + raise SystemExit('unexpected nested-shell state') + path.write_text(text.replace(shellcheck_old, shellcheck_new), encoding='utf-8') PY - - name: Diagnose every repository workflow after the known correction + - name: Diagnose every repository workflow after known corrections shell: bash run: | set -o pipefail From cc686492ce6875faa4e6d5e085990be141429d0c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:42:57 +0900 Subject: [PATCH 91/97] ci: move nested-shell actionlint suppression --- .github/workflows/diagnose-hourly-product-workflow.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/diagnose-hourly-product-workflow.yml b/.github/workflows/diagnose-hourly-product-workflow.yml index b61432c..99aa587 100644 --- a/.github/workflows/diagnose-hourly-product-workflow.yml +++ b/.github/workflows/diagnose-hourly-product-workflow.yml @@ -49,10 +49,10 @@ jobs: raise SystemExit('unexpected hourly workflow indentation state') text = text.replace(indentation_old, indentation_new) - shellcheck_old = " sudo -u '#65532' -g '#65532' env -i \\\n" + shellcheck_old = " if timeout --kill-after=30s \"${OPENCODE_RUN_TIMEOUT_SECONDS}s\" \\\n" shellcheck_new = ( " # shellcheck disable=SC2016 -- inner bash expands positional parameters.\n" - " sudo -u '#65532' -g '#65532' env -i \\\n" + " if timeout --kill-after=30s \"${OPENCODE_RUN_TIMEOUT_SECONDS}s\" \\\n" ) if text.count(shellcheck_old) != 1: raise SystemExit('unexpected nested-shell state') From d35b1f4d2577f385485535d99b608e56bb5968f7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 18:44:49 +0900 Subject: [PATCH 92/97] ci: correct nested-shell actionlint directive --- .github/workflows/diagnose-hourly-product-workflow.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/diagnose-hourly-product-workflow.yml b/.github/workflows/diagnose-hourly-product-workflow.yml index 99aa587..3da59a9 100644 --- a/.github/workflows/diagnose-hourly-product-workflow.yml +++ b/.github/workflows/diagnose-hourly-product-workflow.yml @@ -51,7 +51,7 @@ jobs: shellcheck_old = " if timeout --kill-after=30s \"${OPENCODE_RUN_TIMEOUT_SECONDS}s\" \\\n" shellcheck_new = ( - " # shellcheck disable=SC2016 -- inner bash expands positional parameters.\n" + " # shellcheck disable=SC2016\n" " if timeout --kill-after=30s \"${OPENCODE_RUN_TIMEOUT_SECONDS}s\" \\\n" ) if text.count(shellcheck_old) != 1: From ea0e54ac4163e52ea66c00a2785a93e93bce4ba9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 19:01:49 +0900 Subject: [PATCH 93/97] ci: finalize bounded incremental state overlays --- .../finalize-pr20-bounded-state-overlays.yml | 195 ++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 .github/workflows/finalize-pr20-bounded-state-overlays.yml diff --git a/.github/workflows/finalize-pr20-bounded-state-overlays.yml b/.github/workflows/finalize-pr20-bounded-state-overlays.yml new file mode 100644 index 0000000..2a1d68f --- /dev/null +++ b/.github/workflows/finalize-pr20-bounded-state-overlays.yml @@ -0,0 +1,195 @@ +name: Finalize PR 20 bounded state overlays + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/finalize-pr20-bounded-state-overlays.yml + +permissions: + contents: write + +concurrency: + group: apply-pr20-bounded-state-overlays + cancel-in-progress: false + +jobs: + verify-and-clean: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 40 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Ensure the bounded overlay product patch is present + shell: bash + run: | + set -euo pipefail + if grep -q '^class _OverlayMapping' src/threadweave/incremental.py; then + echo 'Product patch already present.' >>"$GITHUB_STEP_SUMMARY" + elif [ -f tools/pr20-bounded-state-overlays.patch ]; then + git apply --check tools/pr20-bounded-state-overlays.patch + git apply tools/pr20-bounded-state-overlays.patch + echo 'Product patch applied by finalizer.' >>"$GITHUB_STEP_SUMMARY" + else + echo 'Bounded overlay implementation and patch are both missing.' >&2 + exit 1 + fi + git diff --check + + - name: Verify focused product behavior + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + pytest -q \ + tests/test_incremental_private_graph.py \ + tests/test_incremental_rfc8474.py \ + tests/test_incremental_components.py \ + tests/test_incremental_parity.py \ + tests/test_incremental_concurrency.py + + - name: Verify bounded 100,000-message default delta allocations + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python - <<'PYCODE' + import gc + import json + import os + import time + import tracemalloc + + from threadweave import ( + IncrementalThreadIndex, + IndexedMessage, + MailboxChangeSet, + Message, + ) + + message_count = 100_000 + records = tuple( + IndexedMessage( + f"key_{index}", + Message(message_id=f"message_{index}"), + ) + for index in range(message_count) + ) + index = IncrementalThreadIndex() + index.apply(MailboxChangeSet(expected_version=0, additions=records)) + gc.collect() + tracemalloc.start() + baseline_current, _ = tracemalloc.get_traced_memory() + started = time.perf_counter() + delta = index.apply( + MailboxChangeSet( + expected_version=1, + replacements=( + IndexedMessage( + "key_0", + Message(message_id="message_0", subject="updated"), + ), + ), + ) + ) + elapsed_seconds = time.perf_counter() - started + current_bytes, peak_bytes = tracemalloc.get_traced_memory() + tracemalloc.stop() + transient_peak_bytes = peak_bytes - baseline_current + retained_bytes = current_bytes - baseline_current + if delta.affected_message_keys != ("key_0",): + raise SystemExit(delta.affected_message_keys) + if transient_peak_bytes >= 1_000_000: + raise SystemExit( + f"default delta allocated {transient_peak_bytes} transient bytes" + ) + evidence = { + "message_count": message_count, + "affected_message_count": len(delta.affected_message_keys), + "delta_apply_seconds": elapsed_seconds, + "retained_delta_bytes": retained_bytes, + "transient_peak_delta_bytes": transient_peak_bytes, + } + with open( + os.environ["GITHUB_STEP_SUMMARY"], + "a", + encoding="utf-8", + ) as summary: + summary.write("## Bounded default delta evidence\n\n") + summary.write("```json\n") + summary.write(json.dumps(evidence, indent=2, sort_keys=True)) + summary.write("\n```\n") + PYCODE + + - name: Verify the complete repository + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Remove one-shot assets and publish the verified branch state + shell: bash + run: | + set -euo pipefail + rm -f \ + tools/pr20-bounded-state-overlays.patch \ + .github/workflows/apply-pr20-bounded-state-overlays.yml \ + .github/workflows/finalize-pr20-bounded-state-overlays.yml + rmdir tools 2>/dev/null || true + git diff --check + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add -A + if git diff --cached --quiet; then + echo 'No cleanup commit required.' >>"$GITHUB_STEP_SUMMARY" + exit 0 + fi + git commit -m 'ci: finalize bounded incremental state overlays' + git push origin HEAD:feature/incremental-thread-index From a2daf91a4c2c19d50a3cb56a729e214225318b5b Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 6 Aug 2026 10:02:30 +0000 Subject: [PATCH 94/97] ci: finalize bounded incremental state overlays --- .../finalize-pr20-bounded-state-overlays.yml | 195 ------------------ 1 file changed, 195 deletions(-) delete mode 100644 .github/workflows/finalize-pr20-bounded-state-overlays.yml diff --git a/.github/workflows/finalize-pr20-bounded-state-overlays.yml b/.github/workflows/finalize-pr20-bounded-state-overlays.yml deleted file mode 100644 index 2a1d68f..0000000 --- a/.github/workflows/finalize-pr20-bounded-state-overlays.yml +++ /dev/null @@ -1,195 +0,0 @@ -name: Finalize PR 20 bounded state overlays - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - .github/workflows/finalize-pr20-bounded-state-overlays.yml - -permissions: - contents: write - -concurrency: - group: apply-pr20-bounded-state-overlays - cancel-in-progress: false - -jobs: - verify-and-clean: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' - runs-on: ubuntu-24.04 - timeout-minutes: 40 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Ensure the bounded overlay product patch is present - shell: bash - run: | - set -euo pipefail - if grep -q '^class _OverlayMapping' src/threadweave/incremental.py; then - echo 'Product patch already present.' >>"$GITHUB_STEP_SUMMARY" - elif [ -f tools/pr20-bounded-state-overlays.patch ]; then - git apply --check tools/pr20-bounded-state-overlays.patch - git apply tools/pr20-bounded-state-overlays.patch - echo 'Product patch applied by finalizer.' >>"$GITHUB_STEP_SUMMARY" - else - echo 'Bounded overlay implementation and patch are both missing.' >&2 - exit 1 - fi - git diff --check - - - name: Verify focused product behavior - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - pytest -q \ - tests/test_incremental_private_graph.py \ - tests/test_incremental_rfc8474.py \ - tests/test_incremental_components.py \ - tests/test_incremental_parity.py \ - tests/test_incremental_concurrency.py - - - name: Verify bounded 100,000-message default delta allocations - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python - <<'PYCODE' - import gc - import json - import os - import time - import tracemalloc - - from threadweave import ( - IncrementalThreadIndex, - IndexedMessage, - MailboxChangeSet, - Message, - ) - - message_count = 100_000 - records = tuple( - IndexedMessage( - f"key_{index}", - Message(message_id=f"message_{index}"), - ) - for index in range(message_count) - ) - index = IncrementalThreadIndex() - index.apply(MailboxChangeSet(expected_version=0, additions=records)) - gc.collect() - tracemalloc.start() - baseline_current, _ = tracemalloc.get_traced_memory() - started = time.perf_counter() - delta = index.apply( - MailboxChangeSet( - expected_version=1, - replacements=( - IndexedMessage( - "key_0", - Message(message_id="message_0", subject="updated"), - ), - ), - ) - ) - elapsed_seconds = time.perf_counter() - started - current_bytes, peak_bytes = tracemalloc.get_traced_memory() - tracemalloc.stop() - transient_peak_bytes = peak_bytes - baseline_current - retained_bytes = current_bytes - baseline_current - if delta.affected_message_keys != ("key_0",): - raise SystemExit(delta.affected_message_keys) - if transient_peak_bytes >= 1_000_000: - raise SystemExit( - f"default delta allocated {transient_peak_bytes} transient bytes" - ) - evidence = { - "message_count": message_count, - "affected_message_count": len(delta.affected_message_keys), - "delta_apply_seconds": elapsed_seconds, - "retained_delta_bytes": retained_bytes, - "transient_peak_delta_bytes": transient_peak_bytes, - } - with open( - os.environ["GITHUB_STEP_SUMMARY"], - "a", - encoding="utf-8", - ) as summary: - summary.write("## Bounded default delta evidence\n\n") - summary.write("```json\n") - summary.write(json.dumps(evidence, indent=2, sort_keys=True)) - summary.write("\n```\n") - PYCODE - - - name: Verify the complete repository - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Remove one-shot assets and publish the verified branch state - shell: bash - run: | - set -euo pipefail - rm -f \ - tools/pr20-bounded-state-overlays.patch \ - .github/workflows/apply-pr20-bounded-state-overlays.yml \ - .github/workflows/finalize-pr20-bounded-state-overlays.yml - rmdir tools 2>/dev/null || true - git diff --check - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add -A - if git diff --cached --quiet; then - echo 'No cleanup commit required.' >>"$GITHUB_STEP_SUMMARY" - exit 0 - fi - git commit -m 'ci: finalize bounded incremental state overlays' - git push origin HEAD:feature/incremental-thread-index From fc626fa5f5852367383bf9e87fa6bc6d072c9441 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Thu, 6 Aug 2026 19:35:15 +0900 Subject: [PATCH 95/97] ci: ensure incremental delta memory evidence --- .../ensure-pr20-delta-memory-evidence.yml | 297 ++++++++++++++++++ 1 file changed, 297 insertions(+) create mode 100644 .github/workflows/ensure-pr20-delta-memory-evidence.yml diff --git a/.github/workflows/ensure-pr20-delta-memory-evidence.yml b/.github/workflows/ensure-pr20-delta-memory-evidence.yml new file mode 100644 index 0000000..4f08a57 --- /dev/null +++ b/.github/workflows/ensure-pr20-delta-memory-evidence.yml @@ -0,0 +1,297 @@ +name: Ensure PR 20 delta memory evidence + +on: + push: + branches: + - feature/incremental-thread-index + paths: + - .github/workflows/ensure-pr20-delta-memory-evidence.yml + +permissions: + contents: write + +concurrency: + group: ensure-pr20-delta-memory-evidence + cancel-in-progress: false + +jobs: + verify-and-finalize: + if: github.repository == 'ContextualWisdomLab/ThreadWeave' + runs-on: ubuntu-24.04 + timeout-minutes: 45 + steps: + - name: Check out the exact feature branch + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: feature/incremental-thread-index + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + cache: pip + cache-dependency-path: requirements/ci.lock + + - name: Install the reviewed CI lock + run: python -m pip install --require-hashes -r requirements/ci.lock + + - name: Add the failing benchmark contract when absent + id: contract + shell: bash + run: | + set -euo pipefail + if grep -q 'delta_transient_peak_bytes' benchmarks/incremental_mailbox.py; then + echo 'already_applied=true' >>"$GITHUB_OUTPUT" + exit 0 + fi + echo 'already_applied=false' >>"$GITHUB_OUTPUT" + python - <<'PY' + from pathlib import Path + + path = Path('tests/test_incremental_benchmark.py') + text = path.read_text(encoding='utf-8') + old = ''' assert result["incremental"]["delta_apply_seconds"] >= 0 + assert result["incremental"]["peak_rss_bytes"] > 0 + ''' + new = ''' assert result["incremental"]["delta_apply_seconds"] >= 0 + assert result["incremental"]["delta_retained_bytes"] >= 0 + assert result["incremental"]["delta_transient_peak_bytes"] >= 0 + assert result["incremental"]["peak_rss_bytes"] > 0 + ''' + if text.count(old) != 1: + raise SystemExit('benchmark assertion anchor is not unique') + path.write_text(text.replace(old, new), encoding='utf-8') + PY + set +e + PYTHONPATH=src pytest -q \ + tests/test_incremental_benchmark.py::test_small_benchmark_proves_projection_parity_and_reports_resources \ + >"$RUNNER_TEMP/delta-memory-red.log" 2>&1 + red_status="$?" + set -e + cat "$RUNNER_TEMP/delta-memory-red.log" + if [ "$red_status" -eq 0 ]; then + echo 'Expected the memory evidence contract to fail before implementation.' >&2 + exit 1 + fi + if ! grep -Eq 'delta_retained_bytes|delta_transient_peak_bytes' \ + "$RUNNER_TEMP/delta-memory-red.log"; then + echo 'The RED failure did not come from the missing memory fields.' >&2 + exit 1 + fi + + - name: Implement bounded delta allocation evidence when absent + if: steps.contract.outputs.already_applied == 'false' + shell: bash + run: | + set -euo pipefail + python - <<'PY' + from pathlib import Path + + path = Path('benchmarks/incremental_mailbox.py') + text = path.read_text(encoding='utf-8') + old = '''import argparse + import hashlib + import json + import resource + import subprocess + import sys + ''' + new = '''import argparse + import gc + import hashlib + import json + import resource + import subprocess + import sys + import tracemalloc + ''' + if text.count(old) != 1: + raise SystemExit('benchmark import anchor is not unique') + text = text.replace(old, new) + old = ''' bridge = _bridge_record(thread_size) + started = perf_counter() + delta = index.apply( + MailboxChangeSet(expected_version=1, additions=(bridge,)) + ) + delta_seconds = perf_counter() - started + + started = perf_counter() + ''' + new = ''' bridge = _bridge_record(thread_size) + gc.collect() + tracemalloc.start() + baseline_current, _ = tracemalloc.get_traced_memory() + started = perf_counter() + delta = index.apply( + MailboxChangeSet(expected_version=1, additions=(bridge,)) + ) + delta_seconds = perf_counter() - started + current_bytes, peak_bytes = tracemalloc.get_traced_memory() + tracemalloc.stop() + + started = perf_counter() + ''' + if text.count(old) != 1: + raise SystemExit('benchmark delta anchor is not unique') + text = text.replace(old, new) + old = ''' "delta_apply_seconds": delta_seconds, + "materialize_seconds": materialize_seconds, + "peak_rss_bytes": _peak_rss_bytes(), + ''' + new = ''' "delta_apply_seconds": delta_seconds, + "delta_retained_bytes": current_bytes - baseline_current, + "delta_transient_peak_bytes": peak_bytes - baseline_current, + "materialize_seconds": materialize_seconds, + "peak_rss_bytes": _peak_rss_bytes(), + ''' + if text.count(old) != 1: + raise SystemExit('benchmark result anchor is not unique') + path.write_text(text.replace(old, new), encoding='utf-8') + + path = Path('docs/incremental-threading.md') + text = path.read_text(encoding='utf-8') + old = '''RSS, affected-message count, and full-view materialization time. The scheduled/manual + workflow defaults to 100,000 existing messages and stores the JSON evidence for 90 + ''' + new = '''RSS, affected-message count, delta retained/transient traced bytes, and full-view + materialization time. The scheduled/manual workflow defaults to 100,000 existing + messages and stores the JSON evidence for 90 + ''' + if text.count(old) != 1: + raise SystemExit('incremental documentation anchor is not unique') + path.write_text(text.replace(old, new), encoding='utf-8') + + path = Path('README.md') + text = path.read_text(encoding='utf-8') + old = '''and records affected-message count, wall time, and peak RSS as JSON evidence. + ''' + new = '''and records affected-message count, wall time, delta retained/transient traced + bytes, and peak RSS as JSON evidence. + ''' + if text.count(old) != 1: + raise SystemExit('README benchmark anchor is not unique') + path.write_text(text.replace(old, new), encoding='utf-8') + + path = Path('ARCHITECTURE.md') + text = path.read_text(encoding='utf-8') + old = '''message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 + existing messages. Focused performance contracts additionally reject default-mode + ''' + new = '''message count, root count, delta retained/transient traced bytes, and peak RSS. + Scheduled evidence defaults to 100,000 existing messages. Focused performance + contracts additionally reject default-mode + ''' + if text.count(old) != 1: + raise SystemExit('architecture benchmark anchor is not unique') + path.write_text(text.replace(old, new), encoding='utf-8') + + path = Path('CHANGELOG.md') + text = path.read_text(encoding='utf-8') + entry = '''- Record retained and transient traced allocation for each isolated incremental + benchmark delta so scheduled 100,000-message evidence detects mailbox-wide + copying regressions alongside wall time, affected-message count, and peak RSS. + ''' + if entry not in text: + anchor = '## Unreleased\n\n' + if text.count(anchor) != 1: + raise SystemExit('CHANGELOG anchor is not unique') + text = text.replace(anchor, anchor + entry, 1) + path.write_text(text, encoding='utf-8') + PY + + - name: Verify the focused benchmark contract + shell: bash + env: + PYTHONPATH: src + run: pytest -q tests/test_incremental_benchmark.py + + - name: Verify exact 100,000-message evidence + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + python benchmarks/incremental_mailbox.py \ + --messages 100000 \ + --thread-size 10 \ + --output "$RUNNER_TEMP/incremental-benchmark.json" + python - <<'PY' + import json + import os + from pathlib import Path + + result = json.loads( + Path(os.environ['RUNNER_TEMP'], 'incremental-benchmark.json').read_text( + encoding='utf-8' + ) + ) + incremental = result['incremental'] + full_rebuild = result['full_rebuild'] + if incremental['projection_sha256'] != full_rebuild['projection_sha256']: + raise SystemExit('projection digest mismatch') + if incremental['affected_message_count'] != 21: + raise SystemExit('unexpected affected-message count') + if incremental['delta_transient_peak_bytes'] >= 1_000_000: + raise SystemExit('delta transient allocation exceeds the 1 MB contract') + with open( + os.environ['GITHUB_STEP_SUMMARY'], + 'a', + encoding='utf-8', + ) as summary: + summary.write('## Exact 100,000-message delta evidence\n\n') + summary.write('```json\n') + summary.write(json.dumps(result, indent=2, sort_keys=True)) + summary.write('\n```\n') + PY + + - name: Verify the complete repository + shell: bash + env: + PYTHONPATH: src + run: | + set -euo pipefail + ruff check . + python -m compileall -q src tests scripts benchmarks + python -m doctest \ + src/threadweave/collation.py \ + src/threadweave/dates.py \ + src/threadweave/headers.py \ + src/threadweave/subject.py + coverage erase + coverage run --branch --source=scripts/ci -m pytest -q \ + tests/test_autonomous_documentation.py \ + tests/test_dependency_lock_contract.py \ + tests/test_hourly_product_guard.py \ + tests/test_hourly_product_guard_coverage.py \ + tests/test_hourly_product_guard_return.py \ + tests/test_nim_proxy.py \ + tests/test_nim_tls_context.py \ + tests/test_release_contract.py \ + tests/test_release_contract_coverage.py \ + tests/test_release_workflow.py + coverage report \ + --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ + --fail-under=100 \ + --show-missing + coverage erase + coverage run -m pytest -q + coverage report --fail-under=100 --show-missing + python -m build --no-isolation + python -m pip check + + - name: Publish the verified product state and remove the one-shot workflow + shell: bash + run: | + set -euo pipefail + rm .github/workflows/ensure-pr20-delta-memory-evidence.yml + git diff --check + git config user.name 'ThreadWeave verification bot' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add -A + if git diff --cached --quiet; then + echo 'No product or cleanup commit required.' >>"$GITHUB_STEP_SUMMARY" + exit 0 + fi + git commit -m 'perf: retain incremental delta allocation evidence' + git push origin HEAD:feature/incremental-thread-index From da4b8598e1164f83437989c7e0cca72f3ba5420c Mon Sep 17 00:00:00 2001 From: ThreadWeave verification bot <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 6 Aug 2026 10:36:01 +0000 Subject: [PATCH 96/97] perf: retain incremental delta allocation evidence --- .../ensure-pr20-delta-memory-evidence.yml | 297 ------------------ ARCHITECTURE.md | 5 +- CHANGELOG.md | 3 + README.md | 3 +- benchmarks/incremental_mailbox.py | 9 + docs/incremental-threading.md | 5 +- tests/test_incremental_benchmark.py | 2 + 7 files changed, 22 insertions(+), 302 deletions(-) delete mode 100644 .github/workflows/ensure-pr20-delta-memory-evidence.yml diff --git a/.github/workflows/ensure-pr20-delta-memory-evidence.yml b/.github/workflows/ensure-pr20-delta-memory-evidence.yml deleted file mode 100644 index 4f08a57..0000000 --- a/.github/workflows/ensure-pr20-delta-memory-evidence.yml +++ /dev/null @@ -1,297 +0,0 @@ -name: Ensure PR 20 delta memory evidence - -on: - push: - branches: - - feature/incremental-thread-index - paths: - - .github/workflows/ensure-pr20-delta-memory-evidence.yml - -permissions: - contents: write - -concurrency: - group: ensure-pr20-delta-memory-evidence - cancel-in-progress: false - -jobs: - verify-and-finalize: - if: github.repository == 'ContextualWisdomLab/ThreadWeave' - runs-on: ubuntu-24.04 - timeout-minutes: 45 - steps: - - name: Check out the exact feature branch - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: feature/incremental-thread-index - fetch-depth: 1 - - - name: Set up Python - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 - with: - python-version: "3.13" - cache: pip - cache-dependency-path: requirements/ci.lock - - - name: Install the reviewed CI lock - run: python -m pip install --require-hashes -r requirements/ci.lock - - - name: Add the failing benchmark contract when absent - id: contract - shell: bash - run: | - set -euo pipefail - if grep -q 'delta_transient_peak_bytes' benchmarks/incremental_mailbox.py; then - echo 'already_applied=true' >>"$GITHUB_OUTPUT" - exit 0 - fi - echo 'already_applied=false' >>"$GITHUB_OUTPUT" - python - <<'PY' - from pathlib import Path - - path = Path('tests/test_incremental_benchmark.py') - text = path.read_text(encoding='utf-8') - old = ''' assert result["incremental"]["delta_apply_seconds"] >= 0 - assert result["incremental"]["peak_rss_bytes"] > 0 - ''' - new = ''' assert result["incremental"]["delta_apply_seconds"] >= 0 - assert result["incremental"]["delta_retained_bytes"] >= 0 - assert result["incremental"]["delta_transient_peak_bytes"] >= 0 - assert result["incremental"]["peak_rss_bytes"] > 0 - ''' - if text.count(old) != 1: - raise SystemExit('benchmark assertion anchor is not unique') - path.write_text(text.replace(old, new), encoding='utf-8') - PY - set +e - PYTHONPATH=src pytest -q \ - tests/test_incremental_benchmark.py::test_small_benchmark_proves_projection_parity_and_reports_resources \ - >"$RUNNER_TEMP/delta-memory-red.log" 2>&1 - red_status="$?" - set -e - cat "$RUNNER_TEMP/delta-memory-red.log" - if [ "$red_status" -eq 0 ]; then - echo 'Expected the memory evidence contract to fail before implementation.' >&2 - exit 1 - fi - if ! grep -Eq 'delta_retained_bytes|delta_transient_peak_bytes' \ - "$RUNNER_TEMP/delta-memory-red.log"; then - echo 'The RED failure did not come from the missing memory fields.' >&2 - exit 1 - fi - - - name: Implement bounded delta allocation evidence when absent - if: steps.contract.outputs.already_applied == 'false' - shell: bash - run: | - set -euo pipefail - python - <<'PY' - from pathlib import Path - - path = Path('benchmarks/incremental_mailbox.py') - text = path.read_text(encoding='utf-8') - old = '''import argparse - import hashlib - import json - import resource - import subprocess - import sys - ''' - new = '''import argparse - import gc - import hashlib - import json - import resource - import subprocess - import sys - import tracemalloc - ''' - if text.count(old) != 1: - raise SystemExit('benchmark import anchor is not unique') - text = text.replace(old, new) - old = ''' bridge = _bridge_record(thread_size) - started = perf_counter() - delta = index.apply( - MailboxChangeSet(expected_version=1, additions=(bridge,)) - ) - delta_seconds = perf_counter() - started - - started = perf_counter() - ''' - new = ''' bridge = _bridge_record(thread_size) - gc.collect() - tracemalloc.start() - baseline_current, _ = tracemalloc.get_traced_memory() - started = perf_counter() - delta = index.apply( - MailboxChangeSet(expected_version=1, additions=(bridge,)) - ) - delta_seconds = perf_counter() - started - current_bytes, peak_bytes = tracemalloc.get_traced_memory() - tracemalloc.stop() - - started = perf_counter() - ''' - if text.count(old) != 1: - raise SystemExit('benchmark delta anchor is not unique') - text = text.replace(old, new) - old = ''' "delta_apply_seconds": delta_seconds, - "materialize_seconds": materialize_seconds, - "peak_rss_bytes": _peak_rss_bytes(), - ''' - new = ''' "delta_apply_seconds": delta_seconds, - "delta_retained_bytes": current_bytes - baseline_current, - "delta_transient_peak_bytes": peak_bytes - baseline_current, - "materialize_seconds": materialize_seconds, - "peak_rss_bytes": _peak_rss_bytes(), - ''' - if text.count(old) != 1: - raise SystemExit('benchmark result anchor is not unique') - path.write_text(text.replace(old, new), encoding='utf-8') - - path = Path('docs/incremental-threading.md') - text = path.read_text(encoding='utf-8') - old = '''RSS, affected-message count, and full-view materialization time. The scheduled/manual - workflow defaults to 100,000 existing messages and stores the JSON evidence for 90 - ''' - new = '''RSS, affected-message count, delta retained/transient traced bytes, and full-view - materialization time. The scheduled/manual workflow defaults to 100,000 existing - messages and stores the JSON evidence for 90 - ''' - if text.count(old) != 1: - raise SystemExit('incremental documentation anchor is not unique') - path.write_text(text.replace(old, new), encoding='utf-8') - - path = Path('README.md') - text = path.read_text(encoding='utf-8') - old = '''and records affected-message count, wall time, and peak RSS as JSON evidence. - ''' - new = '''and records affected-message count, wall time, delta retained/transient traced - bytes, and peak RSS as JSON evidence. - ''' - if text.count(old) != 1: - raise SystemExit('README benchmark anchor is not unique') - path.write_text(text.replace(old, new), encoding='utf-8') - - path = Path('ARCHITECTURE.md') - text = path.read_text(encoding='utf-8') - old = '''message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 - existing messages. Focused performance contracts additionally reject default-mode - ''' - new = '''message count, root count, delta retained/transient traced bytes, and peak RSS. - Scheduled evidence defaults to 100,000 existing messages. Focused performance - contracts additionally reject default-mode - ''' - if text.count(old) != 1: - raise SystemExit('architecture benchmark anchor is not unique') - path.write_text(text.replace(old, new), encoding='utf-8') - - path = Path('CHANGELOG.md') - text = path.read_text(encoding='utf-8') - entry = '''- Record retained and transient traced allocation for each isolated incremental - benchmark delta so scheduled 100,000-message evidence detects mailbox-wide - copying regressions alongside wall time, affected-message count, and peak RSS. - ''' - if entry not in text: - anchor = '## Unreleased\n\n' - if text.count(anchor) != 1: - raise SystemExit('CHANGELOG anchor is not unique') - text = text.replace(anchor, anchor + entry, 1) - path.write_text(text, encoding='utf-8') - PY - - - name: Verify the focused benchmark contract - shell: bash - env: - PYTHONPATH: src - run: pytest -q tests/test_incremental_benchmark.py - - - name: Verify exact 100,000-message evidence - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - python benchmarks/incremental_mailbox.py \ - --messages 100000 \ - --thread-size 10 \ - --output "$RUNNER_TEMP/incremental-benchmark.json" - python - <<'PY' - import json - import os - from pathlib import Path - - result = json.loads( - Path(os.environ['RUNNER_TEMP'], 'incremental-benchmark.json').read_text( - encoding='utf-8' - ) - ) - incremental = result['incremental'] - full_rebuild = result['full_rebuild'] - if incremental['projection_sha256'] != full_rebuild['projection_sha256']: - raise SystemExit('projection digest mismatch') - if incremental['affected_message_count'] != 21: - raise SystemExit('unexpected affected-message count') - if incremental['delta_transient_peak_bytes'] >= 1_000_000: - raise SystemExit('delta transient allocation exceeds the 1 MB contract') - with open( - os.environ['GITHUB_STEP_SUMMARY'], - 'a', - encoding='utf-8', - ) as summary: - summary.write('## Exact 100,000-message delta evidence\n\n') - summary.write('```json\n') - summary.write(json.dumps(result, indent=2, sort_keys=True)) - summary.write('\n```\n') - PY - - - name: Verify the complete repository - shell: bash - env: - PYTHONPATH: src - run: | - set -euo pipefail - ruff check . - python -m compileall -q src tests scripts benchmarks - python -m doctest \ - src/threadweave/collation.py \ - src/threadweave/dates.py \ - src/threadweave/headers.py \ - src/threadweave/subject.py - coverage erase - coverage run --branch --source=scripts/ci -m pytest -q \ - tests/test_autonomous_documentation.py \ - tests/test_dependency_lock_contract.py \ - tests/test_hourly_product_guard.py \ - tests/test_hourly_product_guard_coverage.py \ - tests/test_hourly_product_guard_return.py \ - tests/test_nim_proxy.py \ - tests/test_nim_tls_context.py \ - tests/test_release_contract.py \ - tests/test_release_contract_coverage.py \ - tests/test_release_workflow.py - coverage report \ - --include=scripts/ci/hourly_product_guard.py,scripts/ci/nim_proxy.py,scripts/ci/release_contract.py \ - --fail-under=100 \ - --show-missing - coverage erase - coverage run -m pytest -q - coverage report --fail-under=100 --show-missing - python -m build --no-isolation - python -m pip check - - - name: Publish the verified product state and remove the one-shot workflow - shell: bash - run: | - set -euo pipefail - rm .github/workflows/ensure-pr20-delta-memory-evidence.yml - git diff --check - git config user.name 'ThreadWeave verification bot' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git add -A - if git diff --cached --quiet; then - echo 'No product or cleanup commit required.' >>"$GITHUB_STEP_SUMMARY" - exit 0 - fi - git commit -m 'perf: retain incremental delta allocation evidence' - git push origin HEAD:feature/incremental-thread-index diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 26cbec1..1fb9ac6 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -74,8 +74,9 @@ sequence numbers or UIDs. The deterministic benchmark runs incremental and full-rebuild workers in separate processes. It compares projection SHA-256 values and records initial-build time, delta-application time, full-view materialization time, full-rebuild time, affected -message count, root count, and peak RSS. Scheduled evidence defaults to 100,000 -existing messages. Focused performance contracts additionally reject default-mode +message count, root count, delta retained/transient traced bytes, and peak RSS. +Scheduled evidence defaults to 100,000 existing messages. Focused performance +contracts additionally reject default-mode small-delta implementations that iterate or replace unrelated state maps. ## Integration policy diff --git a/CHANGELOG.md b/CHANGELOG.md index 726e79f..34aa4ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## Unreleased +- Record retained and transient traced allocation for each isolated incremental + benchmark delta so scheduled 100,000-message evidence detects mailbox-wide + copying regressions alongside wall time, affected-message count, and peak RSS. - Apply default-mode incremental deltas through bounded overlay mappings and touched reverse indexes instead of cloning or scanning every mailbox state map; unchanged records, positions, components, and RFC 8474 identity namespaces stay diff --git a/README.md b/README.md index b03c420..62f5eba 100644 --- a/README.md +++ b/README.md @@ -276,7 +276,8 @@ identity, snapshot, complexity, and RFC boundaries. indefinitely. - `benchmarks/incremental_mailbox.py` compares isolated incremental and full rebuild processes at 100,000 messages, verifies an identical projection digest, - and records affected-message count, wall time, and peak RSS as JSON evidence. + and records affected-message count, wall time, delta retained/transient traced +bytes, and peak RSS as JSON evidence. ## Reproducible CI supply chain diff --git a/benchmarks/incremental_mailbox.py b/benchmarks/incremental_mailbox.py index 063aed0..fdaedc5 100644 --- a/benchmarks/incremental_mailbox.py +++ b/benchmarks/incremental_mailbox.py @@ -9,11 +9,13 @@ from __future__ import annotations import argparse +import gc import hashlib import json import resource import subprocess import sys +import tracemalloc from collections.abc import Iterable from time import perf_counter @@ -108,11 +110,16 @@ def _incremental_worker(message_count: int, thread_size: int) -> dict[str, objec initial_seconds = perf_counter() - started bridge = _bridge_record(thread_size) + gc.collect() + tracemalloc.start() + baseline_current, _ = tracemalloc.get_traced_memory() started = perf_counter() delta = index.apply( MailboxChangeSet(expected_version=1, additions=(bridge,)) ) delta_seconds = perf_counter() - started + current_bytes, peak_bytes = tracemalloc.get_traced_memory() + tracemalloc.stop() started = perf_counter() projections = tuple( @@ -126,6 +133,8 @@ def _incremental_worker(message_count: int, thread_size: int) -> dict[str, objec "affected_message_count": len(delta.affected_message_keys), "initial_build_seconds": initial_seconds, "delta_apply_seconds": delta_seconds, + "delta_retained_bytes": current_bytes - baseline_current, + "delta_transient_peak_bytes": peak_bytes - baseline_current, "materialize_seconds": materialize_seconds, "peak_rss_bytes": _peak_rss_bytes(), "projection_sha256": _projection_digest(projections), diff --git a/docs/incremental-threading.md b/docs/incremental-threading.md index 320191b..aa967fa 100644 --- a/docs/incremental-threading.md +++ b/docs/incremental-threading.md @@ -232,8 +232,9 @@ optimistic conflicts, hostile snapshots, and payload omission. `benchmarks/incremental_mailbox.py` runs incremental and full-rebuild scenarios in separate processes, requires identical projection hashes, and reports wall time, peak -RSS, affected-message count, and full-view materialization time. The scheduled/manual -workflow defaults to 100,000 existing messages and stores the JSON evidence for 90 +RSS, affected-message count, delta retained/transient traced bytes, and full-view +materialization time. The scheduled/manual workflow defaults to 100,000 existing +messages and stores the JSON evidence for 90 days. The update contract promises that unrelated records are not passed to the batch threader; full-view materialization remains proportional to the number of roots and is therefore reported separately from delta application. diff --git a/tests/test_incremental_benchmark.py b/tests/test_incremental_benchmark.py index 4dbe6eb..0163ee5 100644 --- a/tests/test_incremental_benchmark.py +++ b/tests/test_incremental_benchmark.py @@ -32,6 +32,8 @@ def test_small_benchmark_proves_projection_parity_and_reports_resources(): ) assert result["incremental"]["affected_message_count"] == 21 assert result["incremental"]["delta_apply_seconds"] >= 0 + assert result["incremental"]["delta_retained_bytes"] >= 0 + assert result["incremental"]["delta_transient_peak_bytes"] >= 0 assert result["incremental"]["peak_rss_bytes"] > 0 assert result["full_rebuild"]["full_rebuild_seconds"] >= 0 assert result["full_rebuild"]["peak_rss_bytes"] > 0 From 871b0e5712fa056a991b4d1bebb3697fbcc1f431 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 7 Aug 2026 13:19:50 +0900 Subject: [PATCH 97/97] docs: restore benchmark bullet indentation --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 62f5eba..5106ba4 100644 --- a/README.md +++ b/README.md @@ -277,7 +277,7 @@ identity, snapshot, complexity, and RFC boundaries. - `benchmarks/incremental_mailbox.py` compares isolated incremental and full rebuild processes at 100,000 messages, verifies an identical projection digest, and records affected-message count, wall time, delta retained/transient traced -bytes, and peak RSS as JSON evidence. + bytes, and peak RSS as JSON evidence. ## Reproducible CI supply chain