feat: add incremental mailbox threading with stable identity handoff

[seonghobae]

Buyer-visible product gap

ThreadWeave's standards-grounded batch threader otherwise requires a complete rebuild after every arrival, expunge, metadata correction, or delayed ancestor. This PR adds an atomic incremental state boundary for mail servers, migration products, archive viewers, and naruon-style control planes without replacing the canonical batch engine.

Change

• Add IncrementalThreadIndex, IndexedMessage, MailboxChangeSet, deterministic projections/deltas, and explicit merge/split transitions.
• Use immutable caller-owned message keys rather than mutable IMAP sequence numbers.
• Apply additions, replacements, and removals atomically with optimistic version checking and fail-closed rollback.
• Serialize public reads, snapshots, and writes on one index with a process-local reentrant lock so same-version concurrent writers produce one commit and one explicit VersionConflictError; hosts still own distributed write serialization.
• Recompute affected reference or optional RFC 5051 subject components through the canonical thread_messages implementation.
• Materialize the complete public forest lazily and delegate final ordering to the batch oracle.
• Keep implicit sent-date tie-break positions internal instead of exposing invented IMAP sequence numbers.
• Return defensive Container/Message structural copies from roots while retaining caller payloads by reference.
• Replace repeated-minimum component partitioning and pairwise thread-delta overlap scans with indexed passes.
• Apply default-mode transactions through bounded overlay mappings, touched reverse buckets, and compact RFC 8474 association/count indexes rather than copying or scanning every unrelated mailbox state map.
• Preserve RFC 5256 fallback, subject grouping, sent-date ordering, ordinary THREAD, and UID THREAD behavior.
• Validate RFC 8474 EMAILID/THREADID values with exact ObjectID grammar, immutable reported values, equal-EMAILID consistency, and disjoint active namespaces.
• Add deterministic schema-version-1 JSON-safe snapshots that omit arbitrary payloads and require an exact non-boolean integer schema version.
• Reject executable container, key, and scalar subclasses before sorted JSON encoding can invoke attacker-controlled behavior.
• Reject cyclic or reused container identities iteratively so compact Python graphs cannot expand exponentially during encoding.
• Count UTF-8 output without calling str.encode on a complete JSON chunk or allocating a second full bytes copy.
• Preflight the exact root/options schema and record-count limit before nested traversal, then cap the plain-container walk with the configured byte limit before encoder construction.
• Add ARCHITECTURE.md, CLAUDE.md, user/research documentation, CHANGELOG entries, and a weekly/manual isolated mailbox benchmark workflow.

Exact-head verification

Exact head: 46e3c5d43c249e3b39197940cd5cbdffb1f7d7a2

All GitHub-hosted gates on that exact head completed successfully:

• ci on Python 3.10, 3.11, 3.12, and 3.13;
• byte-for-byte lock regeneration and integrity;
• Ruff, compileall, and production doctests;
• package build, hash-install, installed-wheel smoke, and pip check;
• SAST Semgrep;
• Security Scan, including Scorecard, dependency-review policy, OSV, and Trivy jobs.

The exact-head Python 3.13 run reports:

328 tests passed
2,101 combined package/trust-boundary statements: 100%
784 combined package/trust-boundary branches: 100%

The autonomous and release trust boundary separately reports:

74 tests passed
750 statements: 100%
224 branches: 100%

src/threadweave/incremental.py reports 713 statements and 300 branches at 100% coverage. Every published GitHub code-quality review thread is resolved.

The exact-head wheel and sdist were independently hashed and the wheel was installed outside the source tree. Extended smoke verification covered incremental THREAD/UID THREAD, snapshot round-trip, optimistic continuation, identity handoff, and fail-closed record preflight:

wheel SHA-256: e45134ce30305a0e31752bc4d981533bf3f3033c87d3f2ec83890e0e54d33e7b
sdist SHA-256: 1e9ca057d4a10497839350d8221990083c3d3c8e3e9565aba771d9f737f3ea6c

The bounded-state change followed explicit red/green proof:

• before the fix, a one-message default-mode replacement attempted to iterate or clone unrelated state maps;
• after the fix, the same operation retains the established state-map objects and touches only the affected record/component and identity buckets;
• 2,560 randomized additions, replacements, and removals across all group_by_subject / sort_by_sent_date combinations matched complete batch reconstruction at every transition;
• all focused identity, component, concurrency, and RFC regression tests passed before the full suite.

A current-head CodeRabbit review is requested separately. No independent approval is inferred from a skipped, rate-limited, or status-only response.

Mailbox-scale evidence

A 100,000-message default-mode mailbox with one metadata replacement produced the following isolated transaction evidence on the verified product code:

affected messages: 1
delta apply: 0.000848659 s
retained delta allocation: 4,821 bytes
transient peak delta allocation: 11,245 bytes

The scheduled/manual workflow separately compares incremental bridge updates with canonical full rebuilds, requires identical ordered projection digests, reports wall time and peak RSS, and retains JSON evidence for 90 days. The release-order contract requires the mailbox-scale benchmark to be rerun after issue #17 closes and the branch is updated to released main.

These results support bounded small-delta work and canonical parity. They are not a claim of lower initial-build memory, lower full-view materialization cost, or constant-time sent-date ordering.

Standards and references — APA 7th

Bray, T. (Ed.). (2017). The JavaScript Object Notation (JSON) data interchange format (RFC 8259). RFC Editor. https://doi.org/10.17487/RFC8259

Gondwana, B. (2018). IMAP extension for object identifiers (RFC 8474). RFC Editor. https://doi.org/10.17487/RFC8474

Jenkins, N., & Newman, C. (2019). The JSON Meta Application Protocol (JMAP) for mail (RFC 8621). RFC Editor. https://doi.org/10.17487/RFC8621

Melnikov, A., & Cridland, D. (2014). IMAP extensions: Quick flag changes resynchronization (CONDSTORE) and quick mailbox resynchronization (QRESYNC) (RFC 7162). RFC Editor. https://doi.org/10.17487/RFC7162

Melnikov, A., & Leiba, B. (Eds.). (2021). Internet Message Access Protocol (IMAP)—Version 4rev2 (RFC 9051). RFC Editor. https://doi.org/10.17487/RFC9051

The existing RFC 5256 batch implementation remains the structural correctness oracle.

Compatibility and release freeze

• Runtime dependencies remain zero; the batch API and package version remain 0.2.0.
• The incremental layer contains no database, network, authentication, session, tenant, or provider state and remains usable standalone or as a naruon module.
• Issue Release operations: configure the PyPI Trusted Publisher for 0.2.0 #17 remains the account-side 0.2.0 Trusted Publishing blocker. This PR must stay draft and must not merge until the verified 0.2.0 release closes Release operations: configure the PyPI Trusted Publisher for 0.2.0 #17.
• After Release operations: configure the PyPI Trusted Publisher for 0.2.0 #17 closes, update the branch to current main, rerun every exact-head gate and the mailbox-scale benchmark, obtain current-head independent review, and merge only if no blocking finding remains.

Closes #19 only after those post-release conditions are satisfied.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1d06eb60-feeb-40b4-b505-a306a6fc0e6a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[seonghobae]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[seonghobae]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.