Skip to content

Decide the token/segment identity re-anchoring strategy #136

Description

@imnasnainaec

Token.ref embeds the verse SID and character offset ("GEN 1:1:0"), and Segment.id is the verse SID. Every analysis link, phrase link, and alignment endpoint joins on these strings. Any change that shifts offsets — an upstream text edit, a tokenizer improvement, or the user actions proposed in #43/#49 — re-keys every subsequent token in the segment and orphans its links. Drift is currently detected (surface-text snapshots flip links to stale) but never healed, and the WebView silently filters dangling links (analysisSlice.ts resolveApprovedAnalysis).

Decide between (at least):

  1. Stable synthetic token ids (UUID or content-position hybrid) with a ref-mapping layer built at tokenization time. Clean identity, but requires persisting the token layer (today it is rebuilt from USJ and never stored).
  2. A documented re-anchoring algorithm that runs on retokenization: match stored snapshots (surfaceText + neighborhood) against new tokens within the segment, rewrite refs, and mark unmatched links stale. Keeps the model as-is; the algorithm becomes the contract.
  3. Hybrid: keep derived refs but persist a per-project re-anchor journal of user token operations (Split/join tokens that were wrongly tokenized #43's splits/joins) that replays after tokenization.

Deliverable: a short design doc in docs/, agreed by the team, plus a spike validating the chosen approach on a real text edit. #43 and #49 should not start until this lands.

Size: M (design doc + spike)

Priority: P1 — blocks #43 and #49

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions