Unicode apostrophe confuses tokenizer

[pokey]

When the tokenizer sees the unicode apostrophe, it doesn't tokenize correctly. For example:

import spacy

nlp = spacy.load('en', parser=False)

print(list(nlp.tokenizer("I'm hungry")))
print(list(nlp.tokenizer("I\u2019m hungry")))

outputs

[I, 'm, hungry]
[I’m, hungry]

My Environment

• OS X 10.11.6
• Python 3.5.2
• spacy 1.2.0

[ines]

Thanks, good catch! This should probably be added to the tokenizer exceptions.

I'm currently in the process of reorganising the language data and we're gonna merge the changes pretty soon, so there's little point in fixing this in the old, messy format now. I'll do it it on the new organize-language-data branch instead so it will definitely be fixed in the next release.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.