If we tokenize two separate words as a single token, the user should be able to split them, and that should be remembered when the book is retokenized in the future.
Dependency update: "remembered when the book is retokenized" is the crux. Token refs are derived from character offsets, so a split/join shifts every subsequent token's ref in the segment and orphans their analysis links. This issue is blocked on the re-anchoring strategy decision (#136); the likely shape here is a persisted per-project list of token operations (split/join at snapshot positions) replayed after tokenization.
Size: L
If we tokenize two separate words as a single token, the user should be able to split them, and that should be remembered when the book is retokenized in the future.
Dependency update: "remembered when the book is retokenized" is the crux. Token refs are derived from character offsets, so a split/join shifts every subsequent token's ref in the segment and orphans their analysis links. This issue is blocked on the re-anchoring strategy decision (#136); the likely shape here is a persisted per-project list of token operations (split/join at snapshot positions) replayed after tokenization.
Size: L