Skip to content

[Frontend] Use a proper chat template for VLM2Vec#9912

Merged
Isotr0py merged 5 commits intomainfrom
vlm2vec-chat-template
Nov 1, 2024
Merged

[Frontend] Use a proper chat template for VLM2Vec#9912
Isotr0py merged 5 commits intomainfrom
vlm2vec-chat-template

Conversation

@DarkLight1337
Copy link
Copy Markdown
Member

@DarkLight1337 DarkLight1337 commented Nov 1, 2024 •
edited
Loading

VLM2Vec is not intended to be a chat model, so the existing chat template from Phi-3.5-vision doesn't work well.

Building on #9759, this PR adds a chat template to convert the incoming messages into a format suitable for VLM2Vec and updates the docs accordingly.

@DarkLight1337 DarkLight1337 requested a review from ywang96 November 1, 2024 10:21
@github-actions
Copy link
Copy Markdown

github-actions bot commented Nov 1, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@mergify mergify bot added documentation Improvements or additions to documentation frontend labels Nov 1, 2024
@DarkLight1337 DarkLight1337 mentioned this pull request Nov 1, 2024
Open
57 tasks
Isotr0py
Isotr0py approved these changes Nov 1, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Isotr0py Isotr0py enabled auto-merge (squash) November 1, 2024 11:58
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 1, 2024
mgoin
mgoin approved these changes Nov 1, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable, thanks for supporting it through template

DarkLight1337
DarkLight1337 commented Nov 1, 2024
View reviewed changes
vllm/entrypoints/chat_utils.py
Comment on lines +500 to +504
wrap_dicts = (chat_template_text_format == "openai"
or (model_config.task == "embedding"
and model_config.is_multimodal_model)
or (model_config.hf_config.model_type
in MODEL_KEEP_MULTI_MODAL_CONTENT))
Copy link
Copy Markdown
Member Author

@DarkLight1337 DarkLight1337 Nov 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should auto-detect this in the future...

@Isotr0py Isotr0py merged commit ba0d892 into main Nov 1, 2024
@Isotr0py Isotr0py deleted the vlm2vec-chat-template branch November 1, 2024 14:09
richardsliu pushed a commit to richardsliu/vllm that referenced this pull request Nov 4, 2024
@DarkLight1337 @richardsliu
[Frontend] Use a proper chat template for VLM2Vec (vllm-project#9912)
3678bf1 
Signed-off-by: Richard Liu <ricliu@google.com>
hissu-hyvarinen pushed a commit to ROCm/vllm that referenced this pull request Nov 6, 2024
@DarkLight1337 @hissu-hyvarinen
[Frontend] Use a proper chat template for VLM2Vec (vllm-project#9912)
73a4880
JC1DA pushed a commit to JC1DA/vllm that referenced this pull request Nov 11, 2024
@DarkLight1337 @JC1DA
[Frontend] Use a proper chat template for VLM2Vec (vllm-project#9912)
60464e8 
Signed-off-by: Loc Huynh <jc1da.3011@gmail.com>
sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024
@DarkLight1337 @sumitd2
[Frontend] Use a proper chat template for VLM2Vec (vllm-project#9912)
f62b1f5 
Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@DarkLight1337
[Frontend] Use a proper chat template for VLM2Vec (vllm-project#9912)
280f745
LeiWang1999 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Mar 26, 2025
@DarkLight1337 @LeiWang1999
[Frontend] Use a proper chat template for VLM2Vec (vllm-project#9912)
b14d2b4 
Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgoin mgoin mgoin approved these changes

@Isotr0py Isotr0py Isotr0py approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation frontend ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DarkLight1337 @mgoin @Isotr0py