[Post-0.2.0 Product Gap] Incremental mailbox threading with stable identity handoff

[seonghobae]

Buyer-visible problem

ThreadWeave currently rebuilds a complete thread forest from a flat iterable. That is correct for batch ingestion, but mail servers, migration products, archive viewers, and naruon-style control planes receive mailbox deltas continuously. Re-threading an entire large mailbox after every arrival, expunge, metadata change, or delayed ancestor is operationally expensive and gives integrators no explicit contract for stable message/thread identity handoff.

This work MUST NOT begin or merge while release-blocker #17 is open. Release 0.2.0 first from the frozen main, then develop this as the first bounded 0.3.0 product slice.

Product outcome

Add a standalone incremental layer that consumes mailbox change sets, updates only affected reference components and subject buckets, and returns an explicit deterministic delta while preserving the existing batch API unchanged.

The layer must remain usable independently and as a naruon module. The transport-neutral batch functions stay pure; IMAP/JMAP synchronization metadata belongs in a separate adapter/state boundary.

Proposed public boundary

The exact names may be refined in a reviewed design, but the contract must support:

• immutable caller-owned message keys independent of IMAP sequence numbers;
• optional RFC 8474 EMAILID and caller/server-owned THREADID metadata;
• added, replaced, and removed messages in one atomic change set;
• out-of-order ancestry that can join previously separate components;
• a deterministic result describing added, updated, removed, merged, or split thread projections;
• snapshot/restore of versioned, JSON-safe state without serializing caller payloads;
• explicit aliases or remap events when a managed correlation policy cannot preserve an already exposed thread identifier;
• full rebuild parity as the correctness oracle.

Do not silently invent stable server identifiers from mutable sequence numbers. External identifiers must remain caller-owned; any optional managed identifier policy must be explicit and must never hide a merge/remap event.

Standards constraints

• RFC 7162 defines change-oriented IMAP resynchronization through CONDSTORE/QRESYNC, including mailbox state changes, vanished messages, and mod-sequences.
• RFC 8474 defines immutable EMAILID values and a THREADID correlator; once reported, a message's THREADID must not silently change.
• RFC 8621 models every Email as belonging to a Thread, exposes immutable server-set threadId, and acknowledges that delayed messages may reveal relationships that join earlier groups.
• RFC 9051 remains the current IMAP4rev2 base protocol boundary.

Required correctness tests

Use test-driven development and observe each focused regression fail before implementation.

1. Applying additions to an empty index produces the same forest and RFC 5256 serialization as thread_messages over the complete set.
2. Adding a delayed missing ancestor reparents only the affected component and emits a deterministic delta.
3. Adding one bridge message that joins two existing components emits an explicit merge/remap event rather than silently changing exposed identities.
4. Removing a root, internal message, leaf, or duplicate-ID message matches full rebuild semantics, including dummy pruning and promotion.
5. Replacing References, In-Reply-To, subject, sent date, sequence number, UID, or caller metadata updates exactly the affected structural/order/projection state.
6. RFC 5051 subject buckets, RFC 5256 sent-date ordering, THREAD/UID THREAD output, one-shot input behavior, and malformed-cycle safety remain identical to batch results.
7. Snapshot/restore is deterministic, versioned, rejects malformed or oversized input, and never persists arbitrary payload objects.
8. Reapplying the same change set is idempotent or fails with an explicit optimistic-version error; it must never duplicate messages or edges.
9. Realistic mailbox fixtures cover cross-mailbox IDs, delayed delivery, expunge/re-add, duplicate Message-ID, missing IDs, mailing-list subjects, UTF-8 headers, and large deep/split trees.

Performance evidence

• Add a deterministic CI workload large enough to catch accidental full-mailbox scans.
• Add a scheduled/manual benchmark with at least 100,000 messages and bounded delta sizes.
• Report wall time, peak RSS, affected-node count, and full-rebuild comparison.
• The acceptance target is work proportional to the changed reference components and affected subject buckets, not the entire unrelated mailbox.
• Keep all graph operations iterative and avoid recursion or unbounded retained history.

Quality and architecture gates

• zero runtime dependencies;
• 100% production statement and branch coverage;
• complete beginner-readable module/class/function/method/property docstrings;
• Python 3.10–3.13 compatibility unless the release policy changes first;
• exact package-build and installed-wheel smoke coverage;
• CHANGELOG and user/research documentation;
• no database, socket, authentication, workflow, or provider credential inside the core incremental module;
• database-backed adapters, if added later, must use multi-word snake_case object names;
• current-head CI, SAST, Security Scan, independent review, and no unresolved threads before merge.

APA 7th references

Gondwana, B. (2018). IMAP extension for object identifiers (RFC 8474). RFC Editor. https://doi.org/10.17487/RFC8474

Jenkins, N., & Newman, C. (2019). The JSON Meta Application Protocol (JMAP) for mail (RFC 8621). RFC Editor. https://doi.org/10.17487/RFC8621

Melnikov, A., & Cridland, D. (2014). IMAP extensions: Quick flag changes resynchronization (CONDSTORE) and quick mailbox resynchronization (QRESYNC) (RFC 7162). RFC Editor. https://doi.org/10.17487/RFC7162

Melnikov, A., & Leiba, B. (Eds.). (2021). Internet Message Access Protocol (IMAP)—Version 4rev2 (RFC 9051). RFC Editor. https://doi.org/10.17487/RFC9051

Definition of done

The feature is complete only when incremental results and full rebuilds are property-equivalent across the adversarial corpus, stable-identity transitions are explicit, performance evidence shows unrelated mailbox regions are not rescanned, documentation contains the standards and operational boundaries above, and the exact PR head passes all repository gates.