v0.12.2

What's Changed

🚀 Features

• support glm5 by @grimoire in #4355
• Qwen/Internlm/Llama Dense/Moe model fp8 quant online by @43758726 in #4324
• Qwen3.5 by @grimoire in #4351
• GLM-4.7-Flash Turbomind support by @lapy in #4362
• Support router replay and ignore quant layer for qwen3.5 by @RunningLeon in #4394
• [Feature] Add TurboMind support for Qwen3.5 models (dense + MoE) by @lapy in #4389
• support repetition ngram logits processor by @grimoire in #4288

💥 Improvements

• Compatible with transformers 5.0 at TurboMind side by @lvhan028 in #4304
• Support fp32 head for qwen and internlm models by @RunningLeon in #4160
• Reduce MLA kv-cache memory by @lzhangzz in #4373
• add recurrent_gated_delta_rule kernel by @grimoire in #4376
• [ascend]adapt for s1-pro dp*tp+ep by @yao-fengchen in #4380
• Support glm4.7 with mtp by @RunningLeon in #4346
• Faster MLA kernels by @lzhangzz in #4391
• Attention kernel self-registration and decoupled dispatching by @lzhangzz in #4396

🐞 Bug fixes

• fix: change debug log from ERROR to DEBUG in RepetitionPenaltyKernel by @murray-macdonald in #4363
• Fix quant config parsing for internvl awq model by @RunningLeon in #4369
• Fix XGrammar bitmask initialization and add null check for gen_config in generate method by @windreamer in #4349
• fix the logic of closing session by @lvhan028 in #4370
• Fix authorization by @lvhan028 in #4338
• Fix some minor issues and provide tests for Pipeline by @windreamer in #4365
• fix dllm mask on set_step by @grimoire in #4278
• fix models for transformers>=5 by @grimoire in #4381
• fix exception when aborting a request by @lvhan028 in #4403
• fix inference crashed on v100 with qwen3.5-0.8b by @lvhan028 in #4420

🌐 Other

• ci(lint): skip flaky deadlink test for python wiki page by @windreamer in #4357
• fix fa3 install by @irexyc in #4361
• fix lint by @windreamer in #4375
• upgrade triton and torch by @grimoire in #4379
• Add speculative decoding test by @littlegy in #4377
• ci: integrate clang-format lint into pre-commit hooks by @windreamer in #4390
• Update dockerfile by removing cu11 and changing cu12.4 to cu12.6 by @lvhan028 in #4398
• manually build dev image instead of publishing it every version by @lvhan028 in #4409
• bump version to v0.12.2 by @lvhan028 in #4378

New Contributors

• @murray-macdonald made their first contribution in #4363
• @lapy made their first contribution in #4362

Full Changelog: v0.12.1...v0.12.2

v0.12.1

What's Changed

🚀 Features

• support glm-4.7-flash by @RunningLeon in #4320
• [ascend]suppot ep by @yao-fengchen in #3696

💥 Improvements

• fix rotary embedding for transformers v5 by @grimoire in #4303
• Improve metrics log by @CUHKSZzxy in #4297
• Support ignore layers in quant config for qwen3 models by @RunningLeon in #4293
• add custom noaux kernel by @grimoire in #4345
• fix qwen3vl with transformers5 by @grimoire in #4348

🐞 Bug fixes

• fix tool call parser's streaming cursor by @lvhan028 in #4333
• Fix data race for guided decoding in TP mode by @lzhangzz in #4341
• fa3 check by @grimoire in #4340
• Fix time series preprocess by @CUHKSZzxy in #4339
• Negative KV sequence length error in Attention op by @jinminxi104 in #4316
• fix qwen3-vl-moe long context by @grimoire in #4342
• fix: move quantized norm to CPU instead of stale q_linear reference in smooth_quant by @Mr-Neutr0n in #4352
• update noaux-kernel check by @grimoire in #4358

🌐 Other

• change INPUT_CUDA_VERSION to 12.6.2 by @lvhan028 in #4322
• add Qwen3-8B accuracy evaluation in llm_compressor.md by @43758726 in #4319
• [ci] refactor ete testcase by @zhulinJulia24 in #4274
• Set alias interns1_1 for interns1_pro by @lvhan028 in #4334
• build(docker): skip FA2 when use cu13 by @windreamer in #4356
• bump version to v0.12.1 by @lvhan028 in #4350

New Contributors

• @Mr-Neutr0n made their first contribution in #4352

Full Changelog: v0.12.0...v0.12.1

v0.12.0

What's Changed

🚀 Features

• Add Gloo communication to turbomind by @irexyc in #3362
• [Feat] Support llm-compressor AWQ models in TurboMind by @43758726 in #4290
• Router replay for gpt oss by @RunningLeon in #4298
• Support llm-compressor symmetric quantized model inference in TurboMind by @43758726 in #4305
• Support Intern-S1-Pro by @CUHKSZzxy in #4318

💥 Improvements

• Configurable max CTAs and NVLS usage for CUDA IPC communicator by @lzhangzz in #4227
• Improve aborting all sessions by @lvhan028 in #4215
• Moe Reduce kernel by @grimoire in #4228
• Refactor attn by @grimoire in #4238
• Optimize exception raising and error process by @grimoire in #4236
• [AsyncEngine Refactor 1/N] define MultimodalProcessor to handle multimodal data processing by @lvhan028 in #4250
• [AsyncEngine Refactor 2/N] Remove deprecates from chat template by @lvhan028 in #4252
• Configurable uvicorn timeout by @CUHKSZzxy in #4255
• Adapt to dlsime v0.0.2 by @JimyMa in #4242
• [Fix] fix quant calibration dataset by @43758726 in #4256
• lmdeploy suppport parrllel embedding by @Tsundoku958 in #4192
• Refactor turbomind engine by @lzhangzz in #4223
• Refactor Engine & ModelAgent interact by @grimoire in #4265
• Support sleep and destroy deepep buffer by @RunningLeon in #4246
• add yarn truncate by @grimoire in #4301
• [AsyncEngine Refactor 3/N] Introduce Session and SessionManager by @lvhan028 in #4253
• Add warning about NCCL 2.27 memory leaks by @lzhangzz in #4313

🐞 Bug fixes

• Fix fope cos/sin coef device type by @CUHKSZzxy in #4240
• Fix include_stop_str_in_output with output_logits Exception by @windreamer in #4244
• fix logit softcapping is None by @grimoire in #4247
• Fix performance regression for prefix caching by @lzhangzz in #4270
• convert float16 weight to bfloat16 for FP8 models by @lvhan028 in #4276
• [ascend] fix dp multinode rank_table mapping by @tangzhiyi11 in #4268
• [Fix] move calibrate load dataset location by @43758726 in #4289
• fix ignore-eos by @grimoire in #4282
• fix MPEngine poll by @grimoire in #4287
• Fix prefix caching by @lzhangzz in #4292
• Fix gemma chat template by @lvhan028 in #4280
• Fix scheduler metrics by @lzhangzz in #4294
• Fix NVLS init for mixed DP+TP by @lzhangzz in #4296
• [side-effect] The tool message dump is incomplete by @lvhan028 in #4299
• Fix mla with spec tokens by @RunningLeon in #4302
• fix stop long context by @grimoire in #4309
• fix crash on client disconnect (Ctrl+C) by @lvhan028 in #4308
• Ensure the pipe benchmark uses kwargs when calling pipe.stream_infer by @lvhan028 in #4312
• fix get_ppl for long context by @lvhan028 in #4314
• fix sleep engine for dp=1 by @RunningLeon in #4315

🌐 Other

• [ci] fix fail testcase and add generate testcase in pr test by @zhulinJulia24 in #4231
• Pin nvshmem version by @CUHKSZzxy in #4257
• fix: Pin timm version to avoid failed tests by @windreamer in #4258
• docs: add generated openapi spec documentation by @windreamer in #4251
• fix: get rid of buggy timm-1.0.23 by @windreamer in #4260
• [ascend] fix paged prefill by @tangzhiyi11 in #4254
• Fix ascend/maca/camb runtime_requirements by @jinminxi104 in #4262
• docs: refine the documents by @windreamer in #4259
• docs: add cli docs by @windreamer in #4264
• Drop support for Python 3.9 as it has reached end-of-life by @lvhan028 in #4281
• bump version to v0.12.0 by @lvhan028 in #4300

New Contributors

• @43758726 made their first contribution in #4256

Full Changelog: v0.11.1...v0.12.0

v0.11.1

What's Changed

🚀 Features

• [ascend] support dptp by @tangzhiyi11 in #4218
• Support Deepseek v32 by @grimoire in #4026

💥 Improvements

• Improve metrics by @CUHKSZzxy in #4178
• reserve blocks for dummy inputs by @grimoire in #4157
• Add vision id for Qwen3-VL by @CUHKSZzxy in #4183
• [Enhance]: Return routed experts when request canceled by @RunningLeon in #4197
• Add mm processor args for Qwen3-VL by @CUHKSZzxy in #4196
• support chat_template_kwargs in v1/chat/completions by @lvhan028 in #4201
• Refactor scheduler and engine.py by @grimoire in #4163
• update dp timeout by @grimoire in #4204
• Improve Qwen3-VL by @CUHKSZzxy in #4207

🐞 Bug fixes

• [Fix]: Split routed experts with query lens by @RunningLeon in #4180
• [Maca] fix ray and memory sync by @wanfengcxz in #4164
• Build block trie in prefill and add hit rate by @RunningLeon in #4184
• fix fope by @CUHKSZzxy in #4191
• fix hf modules read/write conflicts by multi processors by @lvhan028 in #4188
• Some Minor fix by @windreamer in #4185
• fix insecure deserialization when calling torch.load() by @lvhan028 in #4202
• Fix processor args by @CUHKSZzxy in #4200
• remove get_model_config to avoid pickle hf_config error in rpc calling by @lvhan028 in #4217
• Fix quant scale-fmt by @grimoire in #4212
• Fix requests of mix return_logprobs by @RunningLeon in #4222
• fix fillkv quant8 by @grimoire in #4229
• fix scale-fmt by @grimoire in #4230

📚 Documentations

• [Docs]: Add guide for VLMEvalKit by @CUHKSZzxy in #4156

🌐 Other

• Add FA3 by @CUHKSZzxy in #4166
• Add distributed test cases by @littlegy in #4161
• Add generate test by @littlegy in #4181
• [ci] add mllm eval by @zhulinJulia24 in #4194
• [ascend] refactor code by @yao-fengchen in #4176
• install serve.txt when building the docker image by @lvhan028 in #4219
• bump version to v0.11.1 by @lvhan028 in #4221

Full Changelog: v0.11.0...v0.11.1

v0.11.0

What's Changed

🚀 Features

• add endpoint /abort_request by @lvhan028 in #4092
• Qwen3 next by @grimoire in #4039
• Support Qwen3-VL by @CUHKSZzxy in #4093
• Support sync weights with flattened bucket tensor by @RunningLeon in #4109
• Support group router for moe models by @RunningLeon in #4120
• [Feature]: return routed experts to reuse by @RunningLeon in #4090
• support context parallel by @irexyc in #3951
• fope by @grimoire in #4043
• [Feature]: Support speculative decoding by @RunningLeon in #3945
• Moe bf16 ep by @grimoire in #4144

💥 Improvements

• Enlarge gc threshold by @grimoire in #4076
• remove num_tokens from EngineOutput by @lvhan028 in #4088
• revert masking vocab_size by @lvhan028 in #4089
• feat: add json_object support in response_format by @windreamer in #4080
• support image_data input to /generate endpoint by @irexyc in #4086
• [Fix] all RayEngineWorker actors created at node 0 in RL training by @CyCle1024 in #4107
• Optimize sleep level=1 for turbomind backend by @irexyc in #4074
• [Feat] enable ascend update_params by @CyCle1024 in #4111
• Enhance request checker by @lvhan028 in #4104
• Refactor dp tp by @grimoire in #4004
• fix kernel numerical error by @grimoire in #4133
• free ray put by @grimoire in #4137
• Reduce experts cache when resize by @RunningLeon in #4138
• support interleave text and image in messages by @lvhan028 in #4141
• optimize rms norm by @grimoire in #4153
• fix evict policy by @Tsundoku958 in #4127

🐞 Bug fixes

• fix type hint by @grimoire in #4078
• Fix inputs split by @RunningLeon in #4083
• add missing update_model_meta by @jinminxi104 in #4099
• Fix update_params for pytorch backend when loading vl model by @irexyc in #4101
• workaround for issue "TypeError argument 'tokens': 'NoneType' object cannot be converted to 'PyString" by @lvhan028 in #4103
• fix bug: schedule ratio support prefix-caching by @Tsundoku958 in #4100
• remove prefill free ratio threshold by @grimoire in #4110
• fix key error: api_server node might be removed by @lvhan028 in #4112
• Incorrectly judging the request as a bad request by @lvhan028 in #4121
• fix dist config keys by @grimoire in #4125
• proxy server miss media_type in streaming mode by @lvhan028 in #4130
• Fix logprobs to_tensor by @RunningLeon in #4132
• Fix cli help by @RunningLeon in #4139
• fix and optimize fill_kv_cache_quant by @grimoire in #4140
• fix: fix package deprecation introduced by CUDA 13 by @windreamer in #4117
• yield empty list for token_ids when it runs out of tokens by @lvhan028 in #4148
• Fix interns1 routed experts outputs by @RunningLeon in #4149
• fix qwen3-30-a3b lcb-code score by @yao-fengchen in #4142
• Fix ep deployment issues by @CUHKSZzxy in #4084
• Fix dllm to not use fa3 decoding by @RunningLeon in #4159
• fix: handle non-tuple decoder outputs during Qwen-2.5 quantization by @chengyuma in #4158
• fix cu11 docker build by @CUHKSZzxy in #4165
• Fix model config by @CUHKSZzxy in #4170
• fix lora by @grimoire in #4172
• fix cmake logic detect sm70, sm75 by @tuilakhanh in #4175

📚 Documentations

• Update model evalution guide by @lvhan028 in #4094
• [Docs]: Add guide for update weights by @RunningLeon in #4151

🌐 Other

• add dockerfile to build dev image by @lvhan028 in #4091
• add ascend_a3 Dockerfile by @yao-fengchen in #4097
• [ci] refactor longtext benchmark by @zhulinJulia24 in #4087
• enable metrics by default by @lvhan028 in #4108
• Replace pynvml with nvidia-ml-py in requirements by @myhloli in #4118
• [ci] add free disk before build test whl package and add session_len args in benchmark script by @zhulinJulia24 in #4136
• Add prefixcache functionality and performance testing by @littlegy in #4119
• [ci] modify pipeline.close and add more case into pr_test by @zhulinJulia24 in #4150
• bump version to v0.11.0 by @lvhan028 in #4155

New Contributors

• @myhloli made their first contribution in #4118
• @tuilakhanh made their first contribution in #4175

Full Changelog: v0.10.2...v0.11.0

v0.10.2

What's Changed

🚀 Features

• add /generate api by @irexyc in #4019
• Guided decoding with xgrammar for TurboMind by @windreamer in #3965
• Reimplement guided decoding with xgrammar for PyTorch Engine by @windreamer in #4028

💥 Improvements

• [ascend] support aclgraph by @yao-fengchen in #4063
• Leverage incremental output between the inference and async engines to improve performance by @lvhan028 in #4054
• Optimize multinomial sampling by @grimoire in #4056

🐞 Bug fixes

• zmqrpc localhost only by @grimoire in #4017
• fix bug: dp+tp warmup by @Tsundoku958 in #3991
• fix dllm long-context by @grimoire in #4012
• Fix GPT-OSS streaming tool call parsing by @QwertyJack in #4023
• move releasing resource from async_engine to inference engine by @lvhan028 in #4041
• fix: fix tokenizer parsing bug for guided decoding by @windreamer in #4044
• Fix message content field handling for tool calls and multimodal input by @QwertyJack in #4029
• fix builder for kimi-k2 by @CUHKSZzxy in #4069
• Skip unnecessary sampling and fix the random offset by @grimoire in #4068
• fix duplicated stop_token_string when ignore_special_tokens is False by @irexyc in #4077

🌐 Other

• Drop CUDA 11.8 build support, upgrade CI/CD to CUDA 12.6/12.8 by @windreamer in #4013
• remove profile_generation.py and its testcases by @lvhan028 in #4027
• [ci] refactor eval into api eval and add h800 eval workflow by @zhulinJulia24 in #4008
• Add Docker image for NVIDIA Jetson by @windreamer in #3834
• [ci] refactor api evaluate test into llm judger evaluation by @littlegy in #4046
• Check color logger by @grimoire in #4060
• Update API testing with HLE and LCB datasets by @littlegy in #4061
• update ascend requirements by @yao-fengchen in #4066
• bump version to v0.10.2 by @lvhan028 in #4062

Full Changelog: v0.10.1...v0.10.2

v0.10.1

What's Changed

🚀 Features

• Add ROCm support: installation guide and FlashAttention compatibility for AMD GPUs by @Vivicai1005 in #3925
• support gpt-oss basic output by @irexyc in #3956
• Add FP8*(B)F16 GEMM by @lzhangzz in #3960
• Support GLM-4.5 by @CUHKSZzxy in #3863
• [Refactor]: Remove tokenizer when building engine by @RunningLeon in #3978
• Support InternVL3.5-Flash by @CUHKSZzxy in #3952
• support gpt-oss function/reasoning in /v1/chat/completions by @irexyc in #3962
• support returning stop_str in output by @lvhan028 in #3984
• Support SDAR by @grimoire in #3922

💥 Improvements

• specify installation on GeForce RTX 50 series by @lvhan028 in #3947
• cherry pick PR-3708 to return token_id by @lvhan028 in #3976
• Optimize AsyncEngine generation method by @shell-nlp in #3982
• Use blocking sync when TP engine is idling by @lzhangzz in #3974
• add openai_harmony to requirements by @irexyc in #4006

🐞 Bug fixes

• fix bugs with triton3.4.0 by @grimoire in #3946
• fix longrope by @grimoire in #3968
• Fix tm rl usage in xtuner by @irexyc in #3912
• Disable prefix caching when serving a VLM model by @lvhan028 in #3990
• remove NCCL_LAUNCH_MODE by @irexyc in #3994
• return the last token's logprobs, logits and last_hidden_states if include_stop_str_in_output is requested by @lvhan028 in #4000
• [Fix] device args in chat cli when using pytorch engine by @CyCle1024 in #3999
• fix internvl by @CUHKSZzxy in #3997
• fix not-returned iterator in SequenceManager::Erase by @irexyc in #4001
• fix cudagraph without warmup by @grimoire in #4005
• fix internvl flash long context acc by @CUHKSZzxy in #4003

🌐 Other

• [ci] update daily testcase by @zhulinJulia24 in #3944
• [maca] change kv layout from pagedattn to flashattn by @yuchiwang in #3958
• remove cudnn by @irexyc in #3969
• build(pypi): add cuda 12.8 support for wheels by @windreamer in #3948
• [CI] add ascend test by @littlegy in #3959
• update serve requirement by @RunningLeon in #3986
• [ci] add h800 function test workflow by @zhulinJulia24 in #3985
• bump version to v0.10.1 by @lvhan028 in #3989

New Contributors

• @Vivicai1005 made their first contribution in #3925
• @shell-nlp made their first contribution in #3982
• @littlegy made their first contribution in #3959

Full Changelog: v0.10.0...v0.10.1

v0.10.0

What's Changed

🚀 Features

• support offloading weights & kv_cache for turbomind by @irexyc in #3798
• Add PPU backend support by @guozixu2001 in #3807
• Add turbomind metrics by @lvhan028 in #3811
• PytorchEngine support gpt-oss bf16 by @grimoire in #3820
• support sleep/wakeup for pt engine by @irexyc in #3687
• [ascend] run intern-s1 on A3 by @yao-fengchen in #3831
• Initial gpt-oss support for turbomind by @lzhangzz in #3839
• Support GLM-4-0414 and GLM-4.1V by @CUHKSZzxy in #3846
• support internvl3.5 by @lvhan028 in #3886
• Update turbomind communication library by @lzhangzz in #3736
• MXFP4 support for turbomind GEMM library by @lzhangzz in #3927
• Dispatch MXFP4 weight conversion for sm70 & sm75 by @lzhangzz in #3937

💥 Improvements

• fix: turbomind backend config in cli serve by @PeymanRM in #3784
• remove deprecated codes by @lvhan028 in #3759
• Refactor FP8 MoE GEMM by @lzhangzz in #3795
• Fix build rope params by @grimoire in #3760
• Optimize rmsnorm with head_dim=128 by @grimoire in #3814
• Simplify GEMM interface by @lzhangzz in #3818
• Optimize create_model_inputs and schedule_decoding by @grimoire in #3766
• add remote logs;optimize forward lock by @grimoire in #3737
• support deepgemm new api by @grimoire in #3827
• remove serving with gradio by @lvhan028 in #3829
• Deprecate interactive mode from api_server by @lvhan028 in #3830
• build(docker): Try to optimize docker by @windreamer in #3779
• Make a common chat.py to replace each engine's by @lvhan028 in #3836
• Ray mp engine backend by @grimoire in #3790
• [Feat] support using external ray pg with bundles by @CyCle1024 in #3850
• Remove unused code in PT Engine by @grimoire in #3858
• support logprobs by @grimoire in #3852
• optimize prefill preprocess by @grimoire in #3869
• fix flash-attn bc by @grimoire in #3873
• Graph warmup by @grimoire in #3851
• Improve turbomind's prefix cache by @lvhan028 in #3835
• Support OpenAI compatible parameter max_completion_tokens by @Huarong in #3876
• [ascend] add env to set rt visable by ray and disable warmup by @tangzhiyi11 in #3894
• support cache_max_entry_count >= 1 for Turbomind backend by @lh9171338 in #3913
• adjust default values by @lvhan028 in #3921
• [refactor][chat_template][1/N] adopt tokenizer's apply_chat_template by @lvhan028 in #3845
• use FA 2.8.3 which is compatible with torch 2.8.0 by @lvhan028 in #3936
• refactor ascend Dockerfile by @yao-fengchen in #3926

🐞 Bug fixes

• fix gemma3 by @grimoire in #3772
• fix head_dim=None by @grimoire in #3793
• fix user-specified max_session_len by @grimoire in #3785
• remove 'lmdeploy convert' from CLI by @lvhan028 in #3813
• Fix EP with large batch size by @grimoire in #3808
• fix internvl disable_vision_encoder by @grimoire in #3800
• Align response behavior across both engines by @lvhan028 in #3821
• fix: set text_config.tie_word_embedding = False in qwen2vl by @zenosai in #3824
• Fix v1 comp protocol by @CUHKSZzxy in #3828
• [dlinfer] fix get_backend err by @yao-fengchen in #3847
• Update internvl.py to fix #3528 by @zodiacg in #3837
• fix partial rotary factor by @CUHKSZzxy in #3861
• fix: duplicated token usage in /chat/completions stream mode by @Huarong in #3859
• fix chatting with VLM model via CLI by @lvhan028 in #3862
• fix inference on windows platform by @irexyc in #3865
• fix prebuild on cuda12.8 by @lvhan028 in #3857
• Fix uninitialized members in cuBLAS wrapper by @lzhangzz in #3874
• fix flashmla build for cuda12.4 by @CUHKSZzxy in #3872
• [Fix] ray mp engine on ascend platform by @CyCle1024 in #3877
• fix bug: leaves empty by @Tsundoku958 in #3868
• Fix side effect brought by gpt-oss support by @lvhan028 in #3880
• fix pytorch metrics in mp engine by @CUHKSZzxy in #3882
• Fix stream assert error when wakeup 30+ times by @CyCle1024 in #3883
• fix batched prefill by @grimoire in #3887
• fix side effect brought by #3821 by @lvhan028 in #3888
• check_env in multiprocess by @grimoire in #3879
• fix cli serve --help by @RunningLeon in #3895
• 1. [PD Disaggregation] Some Bug Fix (adapte p2p_initialize, metrics, uniexecutor with pd disagg) by @JimyMa in #3893
• Resolve a crash in the sleep endpoint by casting the level parameter from string to int by @irexyc in #3897
• Fix nccl for docker cu11 by @RunningLeon in #3896
• disable check_env in multiprocess on dlinfer devices by @tangzhiyi11 in #3914
• [dlinfer] fix nn layout typo and scale t by @yuchiwang in #3915
• fix chat and warmup of lora adapter by @grimoire in #3911
• build(acsend): try to fix acsend CI docker build by @windreamer in #3906
• fix internvl3 hf by @CUHKSZzxy in #3932
• build(docker): fix ascend tag name by @windreamer in #3939
• put eot_token to stop_words by @lvhan028 in #3941

📚 Documentations

• update proxy docs by @CUHKSZzxy in #3796
• add missing docs by @CUHKSZzxy in #3871
• fix docs by @CUHKSZzxy in #3885
• update news and citation by @lvhan028 in #3889

🌐 Other

• add prometheus client by @CUHKSZzxy in #3792
• fix: add dummy_prefill guard for PD connection operations by @FirwoodLin in #3803
• minor fix about the log level and logs by @lvhan028 in #3758
• assert PytorchEngineConfig block size by @Tsundoku958 in #3826
• [ci] change restful api into openai and add more testcase by @zhulinJulia24 in #3866
• remove ppu backend by @yao-fengchen in #3904
• [ci] remove flash attn installation in ete test workflow by @zhulinJulia24 in #3908
• dlinfer backend support ray by @yao-fengchen in #3903
• style(types): fix return type annotation for get_all_requests by @xiaoajie738 in #3919
• upgrade torch to 2.8.0 and triton 3.4.0 by @lvhan028 in #3930
• Dlinfer readme by @jinminxi104 in #3938
• bump version to v0.10.0 by @lvhan028 in #3933

New Contributors

• @PeymanRM made their first contribution in #3784
• @FirwoodLin made their first contribution in #3803
• @zenosai made their first contribution in https://github.com/In...

v0.9.2.post1

What's Changed

• Fix interns1 LLM mapping for turbomind engine by @lvhan028 in #3848
• bump version to v0.9.2.post1 by @lvhan028 in #3849

Full Changelog: v0.9.2...v0.9.2.post1

v0.9.2

What's Changed

🚀 Features

• [Feature] metrics support by @CUHKSZzxy in #3534
• Relax FP8 TP requirement by @lzhangzz in #3697
• FA3 by @zhaochaoxing in #3623
• support qwen2/2.5-vl in turbomind by @irexyc in #3744
• feat: add pytorch_engine_qwen2_5vl_sm120 by @kolmogorov-quyet in #3750
• Internvl pt by @RunningLeon in #3765
• Improve internvl for turbomind engine by @lvhan028 in #3769

💥 Improvements

• Refactor linear by @grimoire in #3653
• remove python3.8 support and add python3.13 support by @lvhan028 in #3638
• refactor vl inputs split by @grimoire in #3699
• [Fix]: Replace mutable default with default_factory for scheduler_stats by @ConvolutedDog in #3730
• Fix the logic of calculating max_new_tokens and determining finish_reason by @lvhan028 in #3727
• Override HF config.json via CLI by @CUHKSZzxy in #3722
• feat(build): Integrate and build turbomind backend directly in setup.py by @windreamer in #3726
• Generate the benchmark output filename with given arguments by @lvhan028 in #3740
• Make loading llm without vlm as an option by @grimoire in #3745

🐞 Bug fixes

• add ray to ascend requirements by @sigma-plus in #3713
• fix accessing undefined attribute seq_aux of deepseek-r1-0528 by @lvhan028 in #3728
• [Fix]: Avoid quantize qk norm for qwen3 dense models by @taishan1994 in #3733
• fix py313 env creation failed when building lmdeploy-builder image by @lvhan028 in #3739
• [Fix]: kernel meta retrieval for SM7X does not work by @xiaoajie738 in #3746
• limit max_session_len by @grimoire in #3751
• fix internvl norm by @grimoire in #3756
• support qwen3 moe yarn and vlm hf_overrides by @grimoire in #3757
• [PD Disaggregation] fix double unshelf by @JimyMa in #3762
• fix(build): fix version parse regex to support post-release versions by @windreamer in #3764
• adapt transformers>=v4.52.0 to loading qwen2.5-vl with turbomind by @irexyc in #3771
• fix chat template with tool call by @RunningLeon in #3773
• fix vl nothink mode by @RunningLeon in #3776

📚 Documentations

• update reward model docs by @CUHKSZzxy in #3721

🌐 Other

• update twomicrobatch by @SHshenhao in #3651
• [CI]: Upgrade to py310 for ut by @RunningLeon in #3718
• [ci] update dailytest environment and scripts by @zhulinJulia24 in #3716
• Preliminary Blackwell (sm_120a, RTX 50 series) support by @lzhangzz in #3701
• [ci] add fp8 evaluation workflow by @zhulinJulia24 in #3729
• Add VRAM bandwidth utilization stat to attention test by @lzhangzz in #3731
• doc: fix dead links to MindX DL to recover CI. by @windreamer in #3741
• fix free cache in MPEngine branch by @JimyMa in #3670
• fix: make RelWithDebInfo default cmake build type by @windreamer in #3774
• bump version to v0.9.2 by @lvhan028 in #3770

New Contributors

• @sigma-plus made their first contribution in #3713
• @ConvolutedDog made their first contribution in #3730
• @windreamer made their first contribution in #3726
• @taishan1994 made their first contribution in #3733
• @xiaoajie738 made their first contribution in #3746
• @kolmogorov-quyet made their first contribution in #3750

Full Changelog: v0.9.1...v0.9.2