Add mm processor args for Qwen3-VL#4196
Conversation
| enable_thinking=request.enable_thinking, | ||
| add_vision_id=request.add_vision_id, | ||
| ) | ||
| mm_processor_kwargs=request.mm_processor_kwargs) |
There was a problem hiding this comment.
should we add this option to "/generate" endpoint? cc @irexyc @RunningLeon
lmdeploy/vl/engine.py
Outdated
| async def preprocess(self, | ||
| messages: List[Dict], | ||
| mm_processor_kwargs: Optional[Dict[str, Any]] = None) -> List[Dict]: | ||
| """Preprocess multimodal data in the messages.""" | ||
| future = asyncio.get_event_loop().run_in_executor(self.executor, self.model.preprocess, messages) | ||
| future = asyncio.get_event_loop().run_in_executor(self.executor, self.model.preprocess, messages, | ||
| mm_processor_kwargs) |
There was a problem hiding this comment.
This changed the number of arguments for self.model.preprocess, will it affect other models like internvl? https://github.com/CUHKSZzxy/lmdeploy/blob/44f0942078bfb5a98da12743f4529f4492636029/lmdeploy/vl/model/internvl.py#L193
There was a problem hiding this comment.
I think it does. pr-test CI failed
There was a problem hiding this comment.
should we add mm_processor_kwargs for all the rest vision models? or any other better solution
There was a problem hiding this comment.
Let's address the internvl and qwen-vl series model first
There was a problem hiding this comment.
Let's address the internvl and qwen-vl series model first
Have resolved this, by first checking if self.model.preprocess accpets mm_processor_kwargs or not. See the updated code.
Motivation
User should be able to configure mm processor parameters for Qwen3-VL, e.g.
min_pixels,max_pixelsetc.Therefore, we add a general mm processor kwargs argument for Qwen3-VL and possibly more VLMs.
https://github.com/QwenLM/Qwen3-VL?tab=readme-ov-file#new-qwen-vl-utils-usage
Usage
Adding
mm_processor_kwargsinto the messageOthers
Remove deprecated env
DEEPEP_MAX_BATCH_SIZE, refers to