Allow tokenizers 0.23.1 - #46381
Conversation
Closes #45736
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
tokenizers 0.23.1 made RobertaProcessing args positional-only. Convert the 5 call sites (convert_slow_tokenizer x2, clip, layoutlmv3, roberta) to positional (sep, cls, trim_offsets, add_prefix_space) — works on 0.22.x and 0.23.1.
tokenizers 0.23.1 early-exits right-direction truncation (huggingface/tokenizers#1990), so return_overflowing_tokens may omit the trailing overflow segment when it is shorter than the window. Validate the overflow content only when that segment is present; truncation is always checked. Keeps passing on 0.22.x (segment always present).
The trailing overflow segment is not essential to validate here, and 0.23.1's truncation early-exit (huggingface/tokenizers#1990) may omit it. Keep the truncation checks; remove the now-dead overflow expectations.
tokenizers.processors.BertProcessing.__new__ does not accept 'sep' and 'cls' keyword arguments, raising TypeError on tokenizer init. Pass the (token, id) tuples positionally instead.
Run with tokenizers 0.23.1 (new type stubs). Fixes: - add missing '| None' to params defaulting to None (truncation, udop text/text_target) across layoutlmv2/v3, layoutxlm, luke, markuplm, mluke, roc_bert, pop2piano, udop (invalid-parameter-default) - pop2piano: offset_sec annotated int but defaults to 0.0 -> float - bertweet: import html.entities (submodule not pulled in by 'import html') - tapas: tuple[float, Ellipsis] -> tuple[float, ...] (invalid-type-form)
d951dc9 to
6c8ef7a
Compare
|
View the CircleCI Test Summary for this PR: https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46381&sha=6c8ef7 |
|
Oups doing this now |
Did you push it to Thanks a lot! |
|
nope sorry |
|
I was a bit busy with designing tokenizers v1 |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: bertweet, clip, herbert, layoutlmv2, layoutlmv3, layoutxlm, luke, markuplm, mluke, mpnet, pop2piano, roberta, roc_bert, tapas, udop |
CI recapDashboard: View test results in Grafana |
3 participants
Closes #45736 — allow
tokenizers==0.23.1.Two 0.23.1 changes handled:
RobertaProcessingargs are now positional-only → fixed 5 call sites (convert_slow ×2, clip, layoutlmv3, roberta).Tested: clip/roberta/layoutlmv3/bert/gpt2/t5 tokenizer suites pass on both 0.22.2 and 0.23.1. AI-assisted, human-reviewed.