Add Siglip2Tokenizer to enforce training-time text preprocessing defaults by vaibhav-research · Pull Request #43101 · huggingface/transformers

vaibhav-research · 2026-01-04T22:59:05Z

What does this PR do?

This PR makes SigLIP2 text preprocessing explicit and consistent with how the model was trained.

It introduces a model specific tokenizer (Siglip2Tokenizer) that wraps the existing GemmaTokenizer while enforcing SigLIP2’s training-time defaults (lowercasing and fixed padding/truncation to 64 tokens). AutoTokenizer and AutoProcessor are updated to use this tokenizer for SigLIP2 checkpoints, so users get the correct behavior automatically without relying on implicit processor logic.

The underlying tokenization and vocabulary remain unchanged. This is a lightweight wrapper that improves correctness, reproducibility, and clarity, especially for text embedding and retrieval use cases.

This addresses the behavior reported in #43054.

Fixes # 43054

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
[] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@ArthurZucker @itazap
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

vaibhav-research · 2026-01-04T23:01:38Z

here are the test results including the one that I added in this PR.

pytest -q tests/models/siglip2 -k processing

tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_call_numpy PASSED         [  4%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_call_numpy_4_channels SKIPPED [  8%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_call_pil PASSED           [ 13%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_call_pytorch PASSED       [ 17%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_can_compile_fast_image_processor SKIPPED [ 21%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_cast_dtype_device PASSED  [ 26%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_fast_is_faster_than_slow SKIPPED [ 30%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_image_processor_from_and_save_pretrained PASSED [ 34%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_image_processor_from_dict_with_kwargs PASSED [ 39%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_image_processor_preprocess_arguments PASSED [ 43%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_image_processor_properties PASSED [ 47%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_image_processor_save_load_with_autoimageprocessor PASSED [ 52%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_image_processor_to_json_file PASSED [ 56%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_image_processor_to_json_string PASSED [ 60%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_init_without_params PASSED [ 65%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_is_fast PASSED            [ 69%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_new_models_require_fast_image_processor PASSED [ 73%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_override_instance_attributes_does_not_affect_other_instances SKIPPED [ 78%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_save_load_fast_slow PASSED [ 82%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_save_load_fast_slow_auto PASSED [ 86%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_slow_fast_equivalence PASSED [ 91%]
tests/models/siglip2/test_image_processing_siglip2.py::Siglip2ImageProcessingTest::test_slow_fast_equivalence_batched PASSED [ 95%]
tests/models/siglip2/test_processing_siglip2.py::Siglip2ProcessorTest::test_siglip2_text_padding_length_64 PASSED [100%]

=================================================== warnings summary ====================================================
<frozen importlib._bootstrap>:241
  <frozen importlib._bootstrap>:241: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:241
  <frozen importlib._bootstrap>:241: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================== 19 passed, 4 skipped, 561 deselected, 2 warnings in 9.15s ===============================

vaibhav-research · 2026-01-04T23:24:10Z

I verified locally that SigLIP2 text embeddings behave as expected once the training-time preprocessing is applied (lowercasing + fixed-length padding with max_length=64).

This PR intentionally avoids changing processor defaults and instead documents the correct usage and adds a small test to prevent regressions.

fancyerii

It's ok.

zucchini-nlp

I think it should be done on a tokenizer level. Siglip model for example loads a SiglipTokenizer class which has a lower case called on inputs before tokenizing

@itazap can we use SiglipTokenizer here as well or maybe add a special Siglip2Tokenizer class?

vaibhav-research · 2026-01-05T14:24:06Z

I think it should be done on a tokenizer level. Siglip model for example loads a SiglipTokenizer class which has a lower case called on inputs before tokenizing

@itazap can we use SiglipTokenizer here as well or maybe add a special Siglip2Tokenizer class?

@zucchini-nlp Thanks for reviewing this PR
SigLIP2 currently uses GemmaTokenizerFast for the text branch (see Siglip2Processor and convert_siglip2_to_hf.py). So IMHO we can’t directly reuse SiglipTokenizer (different tokenization backend).
To avoid relying on user-side lowercasing, I can add automatic lowercasing in Siglip2Processor.call before passing text to the tokenizer (covers the main multimodal usage). If we strongly prefer tokenizer-level like SigLIP v1, we’d likely need a Siglip2TokenizerFast wrapper around GemmaTokenizerFast + AutoTokenizer mapping changes. Let me know which direction you’d prefer.
cc @ArthurZucker @itazap

ArthurZucker · 2026-01-05T15:44:15Z

I can add automatic lowercasing in Siglip2Processor.call before passing text to the tokenizer (covers the main multimodal usage)

this sounds like the most reasonable solution no?

vaibhav-research · 2026-01-05T18:50:35Z

I can add automatic lowercasing in Siglip2Processor.call before passing text to the tokenizer (covers the main multimodal usage)

this sounds like the most reasonable solution no?

thanks a lot for reviewing the PR @ArthurZucker
Yeah, agreed. Since SigLIP2 currently uses GemmaTokenizerFast, adding lowercasing at the tokenizer level would either affect Gemma globally or require a new tokenizer class + mapping. Implementing it in Siglip2Processor.call keeps the behavior SigLIP2-specific and covers the main multimodal usage. I’ll push a patch + regression test.

vaibhav-research · 2026-01-05T20:27:05Z

I can add automatic lowercasing in Siglip2Processor.call before passing text to the tokenizer (covers the main multimodal usage)

this sounds like the most reasonable solution no?

thanks a lot for reviewing the PR @ArthurZucker Yeah, agreed. Since SigLIP2 currently uses GemmaTokenizerFast, adding lowercasing at the tokenizer level would either affect Gemma globally or require a new tokenizer class + mapping. Implementing it in Siglip2Processor.call keeps the behavior SigLIP2-specific and covers the main multimodal usage. I’ll push a patch + regression test.

Thanks for the guidance @ArthurZucker , this is now addressed

I implemented automatic lowercasing and the SigLIP2 default text padding/truncation directly in SiglipProcessor.call, gated to SigLIP2-branded checkpoints. This avoids changing GemmaTokenizerFast globally or introducing a new tokenizer class, while ensuring SigLIP2 users get the correct preprocessing by default via AutoProcessor.

I have also updated the SigLIP2 docs to clarify the preprocessing expectations and note that these defaults are now handled automatically by the processor and added regression tests covering the following:
• identical outputs for upper/lower-case inputs
• default padding to length 64

please let me know if you’d prefer this logic to live elsewhere, but this seemed like the least intrusive fix.

vaibhav-research · 2026-01-07T18:02:00Z

looking for your inputs here as well @ArthurZucker
TIA

ArthurZucker

Hey! I really don't think we should do any of the fixes here, but rather on the hub.
We can add a siglip2 tokenizer but it should be explicit that it uses a lowercase normalizer, and should be for all Siglip2 / siglip_2 tokenizer. If its just for 1-2 then it should be updated on the hub directly !

vaibhav-research · 2026-01-09T14:23:58Z

Thanks for detailed review of this @zucchini-nlp @ArthurZucker
I got a lot of pointers to fix :)
will address each of these and confirm once done.
thanks again

vaibhav-research · 2026-01-15T23:22:51Z

all the test are passing except the glm_image test which is unrelated to this change. I am not sure why but I tested the same locally in main and PR branch and both were successful .

(transformer) vaibhavpandey@Vaibhavs-MacBook-Pro transformers % git switch main
git reset --hard upstream/main
pytest -q tests/models/glm_image/test_modeling_glm_image.py::GlmImageModelTest::test_prompt_lookup_decoding_matches_greedy_search -vv

git switch siglip2_text_embedding
pytest -q tests/models/glm_image/test_modeling_glm_image.py::GlmImageModelTest::test_prompt_lookup_decoding_matches_greedy_search -vv

Already on 'main'
Your branch is ahead of 'origin/main' by 127 commits.
  (use "git push" to publish your local commits)
HEAD is now at 9eea1b0bd3 [`FA`] Generalize fa config checks and fix flags (#43121)
================================================== test session starts ==================================================
platform darwin -- Python 3.11.9, pytest-8.4.2, pluggy-1.6.0 -- /Users/vaibhavpandey/.pyenv/versions/transformer/bin/python
cachedir: .pytest_cache
hypothesis profile 'default'
rootdir: /Users/vaibhavpandey/Projects/transformers
configfile: pyproject.toml
plugins: anyio-4.12.0, xdist-3.8.0, timeout-2.4.0, order-1.3.0, rerunfailures-15.1, asyncio-1.3.0, rich-0.2.0, hypothesis-6.148.8
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 1 item                                                                                                        

tests/models/glm_image/test_modeling_glm_image.py::GlmImageModelTest::test_prompt_lookup_decoding_matches_greedy_search PASSED [100%]

=================================================== 1 passed in 0.94s ===================================================

Switched to branch 'siglip2_text_embedding'
Your branch is up to date with 'origin/siglip2_text_embedding'.
================================================== test session starts ==================================================
platform darwin -- Python 3.11.9, pytest-8.4.2, pluggy-1.6.0 -- /Users/vaibhavpandey/.pyenv/versions/transformer/bin/python
cachedir: .pytest_cache
hypothesis profile 'default'
rootdir: /Users/vaibhavpandey/Projects/transformers
configfile: pyproject.toml
plugins: anyio-4.12.0, xdist-3.8.0, timeout-2.4.0, order-1.3.0, rerunfailures-15.1, asyncio-1.3.0, rich-0.2.0, hypothesis-6.148.8
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 1 item                                                                                                        

tests/models/glm_image/test_modeling_glm_image.py::GlmImageModelTest::test_prompt_lookup_decoding_matches_greedy_search PASSED [100%]

=================================================== 1 passed in 1.07s ==================================================

please take a look at the PR for the change it is intended to solve. I am not sure why the CI test is failing on glm_image. sorry to bother you guys again
@ArthurZucker @zucchini-nlp

zucchini-nlp · 2026-01-16T08:40:34Z

The failing glm-image tests is supposed to be skipped. I see that there is a typo in test file and it's not being skipped, so feel free to ignore it

vaibhav-research · 2026-01-18T19:45:49Z

It looks like the PR is failing in the “CircleCI Failure Summary Comment” GitHub Action, not in CircleCI itself.

What I’m seeing:
• CircleCI jobs complete successfully for this commit.
• The GitHub Action fails while trying to find a collection_job and download failure_summary.json.
• In this run, the CircleCI workflow only contains an empty job, so the artifact lookup returns 404 and the script errors (jq: Cannot iterate over null).

I’ve rebased on latest, and the SigLIP2 changes don’t touch CI or CircleCI config, so this seems like an edge case in the failure-summary workflow. not sure, if I missed anything. @zucchini-nlp

vaibhav-research · 2026-01-20T21:50:05Z

hi @zucchini-nlp
just checking in to see if you have some bandwidth to take another look. I had pushed the fixes and addressed the earlier comments.
Let me know if there’s anything else you’d like me to change.
Thanks

vaibhav-research · 2026-01-21T18:41:18Z

looks like there is an open issue for t5 test failures, I will try and rebase again when this is fixed. all other tests are passing.
#43374

github-actions · 2026-01-22T19:20:31Z

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43101&sha=9bac1b

zucchini-nlp

LGTM, thanks a lot for iterating on it! The failing test is flaky and seems unrelated

zucchini-nlp

One tiny comment in docs, to be precise 😄

vaibhav-research · 2026-01-26T15:17:08Z

Thanks a ton @zucchini-nlp and @ArthurZucker I really appreciate the patience and all the helpful nudges along the way 🙏
I’ve pushed the doc tweak and updated the conversion script to use the new Siglip2Tokenizer as suggested.
Whenever you get a chance, would love another look. Thanks again for making this smooth (and for putting up with my many follow-ups 😄)

zucchini-nlp · 2026-01-27T08:29:28Z

Thanks, we can merge imo. The CI is not feeling well since yesterday so let's wait a bit until it's fixed

vaibhav-research · 2026-01-27T14:02:30Z

Thanks, we can merge imo. The CI is not feeling well since yesterday so let's wait a bit until it's fixed

Thanks @zucchini-nlp 🙏 , will wait for CI to recover. I’ll keep an eye on the checks.

ArthurZucker

the attribute error should normally prevent the function to be written

ArthurZucker · 2026-01-28T13:47:55Z

+            backend.normalizer = normalizers.Sequence([normalizers.Lowercase(), backend.normalizer])
+
+    def _unk_id(self) -> int:
+        raise ValueError("_unk_id is not needed for SigLIP2.")


Suggested change

raise ValueError("_unk_id is not needed for SigLIP2.")

raise AttributeError("_unk_id is not needed for SigLIP2.")

Ah shit it does not work one sec

github-actions · 2026-01-28T14:11:47Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, siglip2

vaibhav-research · 2026-01-28T14:34:05Z

thanks @ArthurZucker for fixing the last issue and for merging the PR

ArthurZucker · 2026-01-28T14:36:47Z

for sure, sorry for the delay

ArthurZucker · 2026-02-03T13:00:33Z

+        # Persist for save/load + push_to_hub dynamic tokenizer test
+        if hasattr(self, "init_kwargs") and isinstance(self.init_kwargs, dict):
+            self.init_kwargs.setdefault("tokenizer_class", self.__class__.__name__)
+
+        backend = getattr(self, "_tokenizer", None)
+        if backend is not None and backend.normalizer is not None:
+            backend.normalizer = normalizers.Sequence([normalizers.Lowercase(), backend.normalizer])


we should remove that tho

fancyerii approved these changes Jan 5, 2026

View reviewed changes

zucchini-nlp reviewed Jan 5, 2026

View reviewed changes

vaibhav-research changed the title ~~43054: siglip2 text embedding test and document update~~ 43054: siglip2 text embedding updated to default to lowercase Jan 7, 2026

zucchini-nlp reviewed Jan 8, 2026

View reviewed changes

Comment thread src/transformers/models/siglip/processing_siglip.py Outdated

vaibhav-research changed the title ~~43054: siglip2 text embedding updated to default to lowercase~~ 43054: siglip2 text embedding default to lowercase Jan 9, 2026

vaibhav-research changed the title ~~43054: siglip2 text embedding default to lowercase~~ 43054: Add Siglip2Tokenizer to enforce training-time text preprocessing defaults Jan 9, 2026

ArthurZucker reviewed Jan 9, 2026

View reviewed changes

Comment thread src/transformers/models/siglip/processing_siglip.py Outdated

Comment thread src/transformers/models/siglip/processing_siglip.py Outdated

Comment thread src/transformers/models/siglip2/tokenization_siglip2.py Outdated

zucchini-nlp reviewed Jan 9, 2026

View reviewed changes

vaibhav-research force-pushed the siglip2_text_embedding branch 2 times, most recently from e25cc6c to e1aafd1 Compare January 15, 2026 22:42

zucchini-nlp reviewed Jan 16, 2026

View reviewed changes

Comment thread src/transformers/models/siglip2/modular_siglip2.py Outdated

vaibhav-research force-pushed the siglip2_text_embedding branch from 9993a74 to b167482 Compare January 18, 2026 18:37

vaibhav-research force-pushed the siglip2_text_embedding branch from 0c8f75d to 64e9c82 Compare January 22, 2026 03:07

zucchini-nlp reviewed Jan 22, 2026

View reviewed changes

Comment thread src/transformers/models/siglip2/tokenization_siglip2.py Outdated

vaibhav-research force-pushed the siglip2_text_embedding branch from 0379e11 to d150056 Compare January 22, 2026 15:41

zucchini-nlp approved these changes Jan 23, 2026

View reviewed changes

vaibhav-research added 2 commits January 24, 2026 14:09

added inegration test and removed _unk_id from tokenization

a5b330d

ruff fixes

e648a65

vaibhav-research force-pushed the siglip2_text_embedding branch from 4131a53 to e648a65 Compare January 24, 2026 19:17

updated the doc

4ed6d37

zucchini-nlp approved these changes Jan 26, 2026

View reviewed changes

Comment thread docs/source/en/model_doc/siglip2.md Outdated

Updated doc and conversion script

c913782

vaibhav-research added 5 commits January 26, 2026 10:22

Merge branch 'main' into siglip2_text_embedding

251c6a8

Merge branch 'main' into siglip2_text_embedding

6c1e1dd

Merge branch 'main' into siglip2_text_embedding

bc6cca6

Merge branch 'main' into siglip2_text_embedding

0e9f60c

Merge branch 'main' into siglip2_text_embedding

9beaf58

vaibhav-research added 2 commits January 27, 2026 08:56

Merge branch 'main' into siglip2_text_embedding

9de0edf

Merge branch 'main' into siglip2_text_embedding

c3608a2

vaibhav-research added 5 commits January 27, 2026 10:53

Merge branch 'main' into siglip2_text_embedding

3c1683b

Merge branch 'main' into siglip2_text_embedding

15a1113

Merge branch 'main' into siglip2_text_embedding

050c420

Merge branch 'main' into siglip2_text_embedding

ee811ee

Merge branch 'main' into siglip2_text_embedding

2d9876e

ArthurZucker reviewed Jan 28, 2026

View reviewed changes

ArthurZucker added 2 commits January 28, 2026 15:07

fix modular to skip type hinting when inheriting

34730e1

style

a949320

ArthurZucker merged commit 1be63d9 into huggingface:main Jan 28, 2026
11 of 25 checks passed

ArthurZucker reviewed Feb 3, 2026

View reviewed changes

vaibhav-research changed the title ~~43054: Add Siglip2Tokenizer to enforce training-time text preprocessing defaults~~ Add Siglip2Tokenizer to enforce training-time text preprocessing defaults Apr 6, 2026

	raise ValueError("_unk_id is not needed for SigLIP2.")
	raise AttributeError("_unk_id is not needed for SigLIP2.")

Uh oh!

Conversation

vaibhav-research commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

vaibhav-research commented Jan 4, 2026

Uh oh!

vaibhav-research commented Jan 4, 2026

Uh oh!

fancyerii left a comment

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vaibhav-research commented Jan 5, 2026

Uh oh!

ArthurZucker commented Jan 5, 2026

Uh oh!

vaibhav-research commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vaibhav-research commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vaibhav-research commented Jan 7, 2026

Uh oh!

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vaibhav-research commented Jan 9, 2026

Uh oh!

vaibhav-research commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

zucchini-nlp commented Jan 16, 2026

Uh oh!

vaibhav-research commented Jan 18, 2026

Uh oh!

vaibhav-research commented Jan 20, 2026

Uh oh!

vaibhav-research commented Jan 21, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jan 22, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vaibhav-research commented Jan 26, 2026

Uh oh!

zucchini-nlp commented Jan 27, 2026

Uh oh!

vaibhav-research commented Jan 27, 2026

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

ArthurZucker Jan 28, 2026

vaibhav-research commented Jan 4, 2026 •

edited

Loading

zucchini-nlp left a comment •

edited

Loading

vaibhav-research commented Jan 5, 2026 •

edited

Loading

vaibhav-research commented Jan 5, 2026 •

edited

Loading

vaibhav-research commented Jan 15, 2026 •

edited

Loading