Skip to content

Split/join tokens that were wrongly tokenized #43

Description

@imnasnainaec

If we tokenize two separate words as a single token, the user should be able to split them, and that should be remembered when the book is retokenized in the future.

Dependency update: "remembered when the book is retokenized" is the crux. Token refs are derived from character offsets, so a split/join shifts every subsequent token's ref in the segment and orphans their analysis links. This issue is blocked on the re-anchoring strategy decision (#136); the likely shape here is a persisted per-project list of token operations (split/join at snapshot positions) replayed after tokenization.

Size: L

Metadata

Metadata

Assignees

No one assigned

    Labels

    up nextAuto adds an issue to the PT Lexical Extensions project

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions