Transformers serve VLM by LysandreJik · Pull Request #39454 · huggingface/transformers

LysandreJik · 2025-07-16T14:43:58Z

Add VLM support for transformers serve.

In case you're interested in playing around with it, Open WebUI is a good tool to work with.

Launching transformers serve with VLM support:

cd <PATH_TO_TRANSFORMERS>
git checkout transformers-serve-vlm
transformers serve --enable_cors

To install Open WebUI:

pip install open-webui
open-webui serve

To set it up to work with the transformers serve backend:

Click on the top right icon
Settings
Connections
Add connection
Set URL to "http://localhost:8000/v1"
Set Key to non-null string
Verify connection
All good!

You should now find a limited (but growing) list of available models:

Enjoy chatting with the model!

HuggingFaceDocBuilderDev · 2025-07-16T14:57:05Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

zucchini-nlp

Great, looking forward to get it shipped! Left a few questions below, to make sure it will work for all VLMs. I will also try to set up an env for playing around

zucchini-nlp · 2025-07-17T07:56:22Z

+            eos_token_id=self.processor.eos_token_id,
+            pad_token_id=self.processor.pad_token_id,


not all processor's have direct access to special tokens. For a general approach we'd do processor.tokenizer.bos_token_ids

Or maybe we could add direct access through ProcessorMixin, lemme see, Makes sense

Thanks, fixed it ina34e597

zucchini-nlp · 2025-07-17T08:01:07Z

+                        file = tempfile.NamedTemporaryFile(suffix=".png", delete=False)
+                        image.save(file.name)
+
+                        parsed_message["content"].append({"type": "image", "url": file.name})


nit: i think we don't need to save temporarily the image, templates can load from any source

Alright, I'll take a look to see how to handle base64 directly, thx!

zucchini-nlp · 2025-07-17T08:03:22Z

-        text = self.tokenizer.apply_chat_template(req["input"], add_generation_prompt=True, tokenize=False)
-        inputs = self.tokenizer(text, return_tensors="pt").to(self.model.device)["input_ids"]
+        text = self.processor.apply_chat_template(req["input"], add_generation_prompt=True, tokenize=False)
+        inputs = self.processor(text, return_tensors="pt").to(self.model.device)["input_ids"]


Has to be keyword argument, because positionally images are always the first arg. In general after Pablo's refactor we've been recommending to use only keyword args in processor. And just a few days ago I removed all BC workarounds we had

processor(text=text, return_tensors="pt")

Thanks Raushan! Actually this can be simplified to do the tokenization within the chat template application directly:

- text = processor.apply_chat_template(req["input"], add_generation_prompt=True, tokenize=False) - inputs = processor(text, return_tensors="pt").to(model.device)["input_ids"] + inputs = processor.apply_chat_template(req["input"], add_generation_prompt=True).to(model.device)

Given that the req["input"] contains both text and images, what do you recommend regarding explicit kwargs?

Yeah, in that case it's perfect. Image-only templates are already settled, tested and won't change in the future. We can pass any keyword kwargs and those will be passed:

To Jinja template in case the template has if/else controlled by users flags, e.g. add_vision_id=True in Qwen

To the processor, after filtering unused kwargs. No warning raised here

zucchini-nlp · 2025-07-17T08:08:54Z

+            elements_are_dicts = any(isinstance(content, str) for content in message["content"])
+
+            if isinstance(message["content"], str):
+                if elements_are_dicts:
+                    # If other elements are dictionaries, then plain strings should be as well
+                    parsed_message["content"].append({"type": "text", "text": message["content"]})
+                else:
+                    # If no other elements are dicts, then the string is the entire content.
+                    parsed_message["content"] = message["content"]


these part is a bit confusing, is the content is single string, how we can have elements_are_dicts within the content? Prob I am missing smth with OpenAI format

Simplified it! Thanks

tjbck · 2025-07-18T13:57:22Z

Looking great! qq from our side (without having done much research): how difficult would it be to simply spawn up the chat completion endpoint/function directly from our backend? It would be great to have pip install open-webui[transformers] option where our users can get started right away with all the batteries included. We'd appreciate any guidance here!

gante

Other than the model loading abstractions, which are out of sync with the (merged) stt serving PR, LGTM 🙌

gante · 2025-07-18T14:14:55Z

+            "Qwen/Qwen2.5-VL-7B-Instruct",
        ]

+        if HF_HUB_OFFLINE:


gante · 2025-07-18T14:22:52Z

@tjbck from my experience, with the exception of changing the OpenAI address manually, integration with open-webui is seamless! Open your app, launch transformers serve --enable-cors, and everything just works 💛

If we can ask for features on your side, it would be nice to be able to specify arbitrary models from the model selection dropdown (as opposed to being limited to the list from /v1/models). Then any model from the HF Hub could be immediately accessed!

Let us know if we can help from our side 🫶

LysandreJik · 2025-07-18T14:34:54Z

Thanks for dropping by @tjbck!

As Joao said above, it should be pretty seamless. You might want to specify some specific extras when installing transformers:

transformers[torch,serving]  # for raw transformers + torch, so for all LLMs basically, and required deps to launch serve
transformers[torch,serving,vision]  # for VLMs as well
transformers[torch,serving,audio]  # for STT (and eventually TTS)

As Joao highlights there is a question around which LLMs/VLMs are shown as transformers supports ~250k LLMs and ~4k VLMs from the Hugging Face Hub, displaying them all in the top-left hand corner would be overwhelming 😀

Let us know in case you run into any issues, we'd be happy to add in features that would make your life simpler.

Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com>

gante

let's goooo 🔥

Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

* Add support for VLMs in Transformers Serve * Raushan comments * Update src/transformers/commands/serving.py Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> * Quick fix * CPU -> Auto * Update src/transformers/commands/serving.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Fixup --------- Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

ArthurZucker approved these changes Jul 16, 2025

View reviewed changes

zucchini-nlp reviewed Jul 17, 2025

View reviewed changes

LysandreJik force-pushed the transformers-serve-vlm branch from 5f3d658 to ebc0c4a Compare July 18, 2025 12:52

LysandreJik added 2 commits July 18, 2025 14:57

Add support for VLMs in Transformers Serve

7d40b7b

Raushan comments

a34e597

LysandreJik force-pushed the transformers-serve-vlm branch from ebc0c4a to e81c243 Compare July 18, 2025 13:17

LysandreJik changed the base branch from main to chat-extra July 18, 2025 13:19

LysandreJik changed the base branch from chat-extra to main July 18, 2025 13:19

LysandreJik force-pushed the transformers-serve-vlm branch from e81c243 to a34e597 Compare July 18, 2025 13:19

gante approved these changes Jul 18, 2025

View reviewed changes

Comment thread src/transformers/commands/serving.py

"Qwen/Qwen2.5-VL-7B-Instruct",

]

if HF_HUB_OFFLINE:

gante Jul 18, 2025

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧠

sergiopaniego reviewed Jul 18, 2025

View reviewed changes

Comment thread src/transformers/commands/serving.py Outdated

LysandreJik and others added 3 commits July 21, 2025 16:25

Update src/transformers/commands/serving.py

c554b17

Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com>

Quick fix

62736e6

CPU -> Auto

725776f

gante approved these changes Jul 23, 2025

View reviewed changes

Comment thread src/transformers/commands/serving.py Outdated

LysandreJik and others added 3 commits July 23, 2025 13:52

Update src/transformers/commands/serving.py

a536c7f

Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

Fixup

e46a4ac

Merge branch 'main' into transformers-serve-vlm

3c559fb

LysandreJik enabled auto-merge (squash) July 23, 2025 13:55

LysandreJik disabled auto-merge July 23, 2025 15:03

LysandreJik merged commit a0e5a7d into main Jul 23, 2025
15 of 17 checks passed

LysandreJik deleted the transformers-serve-vlm branch July 23, 2025 15:03

ebezzam mentioned this pull request Jul 23, 2025

Move openai import #39613

Merged

gante mentioned this pull request Jul 24, 2025

[serve] Add speech-to-text #39631

Draft

		eos_token_id=self.processor.eos_token_id,
		pad_token_id=self.processor.pad_token_id,

Uh oh!

Conversation

LysandreJik commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jul 16, 2025

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tjbck commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gante left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gante commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

LysandreJik commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gante left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

LysandreJik commented Jul 16, 2025 •

edited

Loading

zucchini-nlp Jul 18, 2025 •

edited

Loading

tjbck commented Jul 18, 2025 •

edited

Loading

gante commented Jul 18, 2025 •

edited

Loading

LysandreJik commented Jul 18, 2025 •

edited

Loading