Fix CodeLlama tokenizer stripping real leading spaces on decode (#46491)

[kpal002]

What this fixes

Closes #46491.

decoders.Strip(content=" ", left=1) in CodeLlamaTokenizer.__init__ was unconditionally removing one leading space on every decode call. For inputs that begin with real spaces (e.g. indented Python code), this silently corrupted the output — decode(encode(" if x:\n")) returned " if x:\n" instead of " if x:\n".

The stripping was intended to remove the synthetic ▁ prefix that Metaspace(prepend_scheme="first") prepends before the first content token. The bug: after Replace(▁ → " "), the synthetic prefix and real user-provided leading spaces are indistinguishable in the decoded string, so Strip(left=1) ate one real space whenever the input started with one or more spaces.

Root cause

# before (tokenization_code_llama.py line 163-164)
self._tokenizer.decoder = decoders.Sequence(
    [decoders.Replace("▁", " "), decoders.ByteFallback(), decoders.Fuse(), decoders.Strip(content=" ", left=1)]
)

Fix

Remove Strip from the decoder sequence and override _decode to perform the strip conditionally, based on the first real (non-special) token:

• If the first real token consists entirely of ▁ characters (▁=29871, ▁▁=259, ▁▁▁=1678, …), it was produced by real user-provided leading spaces → do not strip.
• Otherwise, the leading space in the decoded text came from the synthetic Metaspace prefix → strip one space.

This handles 2, 3, 4, and N leading spaces, and is correct in the presence of a prepended BOS token (BOS is in all_special_ids and is skipped when looking for the first real token).

Note: a single leading space (" hello" vs "hello") is inherently ambiguous — both encode to the same token IDs — and cannot round-trip. This was true before this fix and remains true; it is documented in the new test.

Before / after

# transformers 5.10.x (before)
BUG: '  hello'           -> ' hello'
BUG: '   leading spaces' -> '  leading spaces'
BUG: '    indented_line' -> '   indented_line'

# after this fix
OK:  '  hello'           -> '  hello'
OK:  '   leading spaces' -> '   leading spaces'
OK:  '    indented_line' -> '    indented_line'
OK:  'hello world'       -> 'hello world'   # unaffected

Tests

Added test_leading_space_roundtrip to tests/models/code_llama/test_tokenization_code_llama.py covering 2-, 3-, and 4-space-prefixed strings via encode(s, add_special_tokens=False) → decode(…, skip_special_tokens=True).

Ran locally on this PR's branch (pytest tests/models/code_llama/test_tokenization_code_llama.py, Python 3.12.13, transformers 5.10.0.dev0 editable install):

60 passed, 8 skipped in 34.11s

including CodeLlamaTokenizationTest::test_leading_space_roundtrip. Also green on CI: https://circleci.com/gh/huggingface/transformers/2354143

Disclosure

This fix was drafted with AI-agent assistance (diagnosis, patch, and test). I reviewed every changed line, confirmed the root cause by tracing the super().__init__() overwrite of self._tokenizer and the BPE merge behavior for multi-▁ tokens, and ran the test suite locally (output above) before opening this PR.

[Rocketknight1]

cc @ArthurZucker @itazap

[github-actions]

CI Dashboard: View test results in Grafana

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: code_llama

[qgallouedec]

closing as superseded, see #47487 (comment), thanks!