From 87995e3f5dcc5a6fcc2f7c4fe3dce7687620a13f Mon Sep 17 00:00:00 2001 From: Grzegorz Karch Date: Thu, 16 Apr 2026 15:29:43 +0200 Subject: [PATCH 1/8] updated docs Signed-off-by: Grzegorz Karch --- examples/puzzletron/README.md | 15 +++++--- modelopt/torch/puzzletron/anymodel/README.md | 36 ++++++++++++++++++++ 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index 5bd7c650648..0296bc26cfe 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -275,12 +275,17 @@ For a quick smoke test, add `--limit 10`. Now let's evaluate how much speedup we get with the compressed model in terms of throughput and latency. - Install [vLLM from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source). -- Rearrange the model safetensors to be used for vLLM. -```bash -cd path/to/model -mv subblocks_safetensors/* . -sed -i 's+subblocks_safetensors/++g' model.safetensors.index.json +- Modify the model's `config.json` file to support vLLM, see [AnyModel Guide](../../modelopt/torch/puzzletron/anymodel/README.md#deploy-compressed-model-in-vllm) for details. + + +```json +{ + ... + "architectures": ["AnyModel"], + "base_architecture": "LlamaForCausalLM", + ... +} ``` - Benchmark latency diff --git a/modelopt/torch/puzzletron/anymodel/README.md b/modelopt/torch/puzzletron/anymodel/README.md index 1c8c68b60bf..546e1a7560c 100644 --- a/modelopt/torch/puzzletron/anymodel/README.md +++ b/modelopt/torch/puzzletron/anymodel/README.md @@ -98,6 +98,42 @@ Update pruning YAML files (`ffn_pruning.yaml`, `expert_pruning.yaml`, etc.): See [test_puzzletron.py](../../../../tests/gpu/torch/puzzletron/test_puzzletron.py) for a complete example that runs both convert and compression steps. For container setup and dependencies needed to run this test, see the [Puzzletron README environment section](../../../../examples/puzzletron/README.md#environment). +## Deploy compressed model in vLLM + +To deploy a compressed model in vLLM, you need to add the following to the model's `config.json` file (here we use Llama as an example): + +```json +{ + ... + "architectures": ["AnyModel"], + "base_architecture": "LlamaForCausalLM", + ... +} +``` + +For new architectures that are not supported by vLLM, you additionally need to add the following to the `config.json` file (using Llama3 as an example): + +```json +{ + ... + "anymodel_arch_info": { + "decoder_layer_module": ".", + "decoder_layer_class": "", + "base_model_module": ".", + "layers_path": "", + "init_prefix": "model", + "Layer_hf_config": "", + } + ... +} +``` + +With these changes it is now possible to load the compressed model in vLLM for inference: + +```bash +vllm serve +``` + --- ## Advanced Topics From 95d06aa85c18ca2fa8e4f3f490c8301821b328cf Mon Sep 17 00:00:00 2001 From: Grzegorz Karch Date: Mon, 20 Apr 2026 12:33:00 +0200 Subject: [PATCH 2/8] Moved vLLM deployment instructions from anymodel folder to examples/puzzletron Signed-off-by: Grzegorz Karch --- examples/puzzletron/README.md | 37 ++++++++++++++++---- modelopt/torch/puzzletron/anymodel/README.md | 36 ------------------- 2 files changed, 30 insertions(+), 43 deletions(-) diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index 0296bc26cfe..0868152b903 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -270,14 +270,10 @@ For a quick smoke test, add `--limit 10`. > **Alternative:** For server-based evaluation via an OpenAI-compatible endpoint, > see [evaluation/nemo_evaluator_instructions.md](./evaluation/nemo_evaluator_instructions.md). -## Inference Performance Benchmarking - -Now let's evaluate how much speedup we get with the compressed model in terms of throughput and latency. - -- Install [vLLM from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source). - -- Modify the model's `config.json` file to support vLLM, see [AnyModel Guide](../../modelopt/torch/puzzletron/anymodel/README.md#deploy-compressed-model-in-vllm) for details. +## Deploy compressed model in vLLM +To deploy a compressed model in vLLM, install [vLLM from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source). +Then, add the following to the model's `config.json` file (here we use Llama as an example): ```json { @@ -288,6 +284,33 @@ Now let's evaluate how much speedup we get with the compressed model in terms of } ``` +For new architectures that are not supported by vLLM, you additionally need to add the following to the `config.json` file (using Llama3 as an example): + +```json +{ + ... + "anymodel_arch_info": { + "decoder_layer_module": ".", + "decoder_layer_class": "", + "base_model_module": ".", + "layers_path": "", + "init_prefix": "model", + "Layer_hf_config": "", + } + ... +} +``` + +With these changes it is now possible to load the compressed model in vLLM for inference: + +```bash +vllm serve +``` + +### Inference Performance Benchmarking + +Now let's evaluate how much speedup we get with the compressed model in terms of throughput and latency. + - Benchmark latency ```bash diff --git a/modelopt/torch/puzzletron/anymodel/README.md b/modelopt/torch/puzzletron/anymodel/README.md index 546e1a7560c..1c8c68b60bf 100644 --- a/modelopt/torch/puzzletron/anymodel/README.md +++ b/modelopt/torch/puzzletron/anymodel/README.md @@ -98,42 +98,6 @@ Update pruning YAML files (`ffn_pruning.yaml`, `expert_pruning.yaml`, etc.): See [test_puzzletron.py](../../../../tests/gpu/torch/puzzletron/test_puzzletron.py) for a complete example that runs both convert and compression steps. For container setup and dependencies needed to run this test, see the [Puzzletron README environment section](../../../../examples/puzzletron/README.md#environment). -## Deploy compressed model in vLLM - -To deploy a compressed model in vLLM, you need to add the following to the model's `config.json` file (here we use Llama as an example): - -```json -{ - ... - "architectures": ["AnyModel"], - "base_architecture": "LlamaForCausalLM", - ... -} -``` - -For new architectures that are not supported by vLLM, you additionally need to add the following to the `config.json` file (using Llama3 as an example): - -```json -{ - ... - "anymodel_arch_info": { - "decoder_layer_module": ".", - "decoder_layer_class": "", - "base_model_module": ".", - "layers_path": "", - "init_prefix": "model", - "Layer_hf_config": "", - } - ... -} -``` - -With these changes it is now possible to load the compressed model in vLLM for inference: - -```bash -vllm serve -``` - --- ## Advanced Topics From 80f7a9aa3c36d1457d5df15f748460e3add3f50c Mon Sep 17 00:00:00 2001 From: Grzegorz Karch Date: Wed, 22 Apr 2026 09:23:21 +0200 Subject: [PATCH 3/8] updated vllm installation instruction Signed-off-by: Grzegorz Karch --- examples/puzzletron/README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index 0868152b903..95756670f83 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -272,7 +272,14 @@ For a quick smoke test, add `--limit 10`. ## Deploy compressed model in vLLM -To deploy a compressed model in vLLM, install [vLLM from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source). +To deploy a compressed model in vLLM, install vLLM fork with AnyModel enabled: + +```bash +git clone -b feature/add_anymodel_to_vllm https://github.com/askliar/vllm.git +cd vllm +VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto +``` + Then, add the following to the model's `config.json` file (here we use Llama as an example): ```json From 955eef6580041146fcf3e8c3099a0bfde51683c9 Mon Sep 17 00:00:00 2001 From: Grzegorz Karch Date: Wed, 22 Apr 2026 10:29:08 +0200 Subject: [PATCH 4/8] removed trailing comma in the json snippet Signed-off-by: Grzegorz Karch --- examples/puzzletron/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index 95756670f83..7490fd78f0c 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -302,7 +302,7 @@ For new architectures that are not supported by vLLM, you additionally need to a "base_model_module": ".", "layers_path": "", "init_prefix": "model", - "Layer_hf_config": "", + "Layer_hf_config": "" } ... } From cf35d205f8c3f259863a2aa48c4f59da76ffb0d2 Mon Sep 17 00:00:00 2001 From: Grzegorz Karch Date: Wed, 22 Apr 2026 10:38:31 +0200 Subject: [PATCH 5/8] update installation instruction Signed-off-by: Grzegorz Karch --- examples/puzzletron/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index 7490fd78f0c..d39c5da5147 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -280,6 +280,10 @@ cd vllm VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto ``` +See [vLLM documentation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for more details on installation. + +**NOTE:** This a temporary workaround pending official vLLM integration. You can track merge status [here](https://github.com/vllm-project/vllm/pull/36512). + Then, add the following to the model's `config.json` file (here we use Llama as an example): ```json From 3a84fd02152203bdb93cfa6ff5bab24566b19dfe Mon Sep 17 00:00:00 2001 From: Grzegorz Karch Date: Wed, 22 Apr 2026 10:52:42 +0200 Subject: [PATCH 6/8] solution for https://github.com/NVIDIA/Model-Optimizer/pull/1317#discussion_r3122632514 Signed-off-by: Grzegorz Karch --- examples/puzzletron/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index d39c5da5147..2a68aaab3a2 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -275,8 +275,9 @@ For a quick smoke test, add `--limit 10`. To deploy a compressed model in vLLM, install vLLM fork with AnyModel enabled: ```bash -git clone -b feature/add_anymodel_to_vllm https://github.com/askliar/vllm.git +git clone https://github.com/askliar/vllm.git cd vllm +git checkout 6a5ea42e55b85444da1a511a007f2d627845536f VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto ``` From 7b1d15e0ff1e9be7252aeb16454f32dfb91b3050 Mon Sep 17 00:00:00 2001 From: Grzegorz Karch Date: Wed, 22 Apr 2026 11:07:30 +0200 Subject: [PATCH 7/8] solution for https://github.com/NVIDIA/Model-Optimizer/pull/1317#discussion_r3122632527 Signed-off-by: Grzegorz Karch --- examples/puzzletron/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index 2a68aaab3a2..915fdb3ddc7 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -283,7 +283,7 @@ VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto See [vLLM documentation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for more details on installation. -**NOTE:** This a temporary workaround pending official vLLM integration. You can track merge status [here](https://github.com/vllm-project/vllm/pull/36512). +**NOTE:** This is a temporary workaround pending official vLLM integration. You can track merge status [here](https://github.com/vllm-project/vllm/pull/36512). Then, add the following to the model's `config.json` file (here we use Llama as an example): From 01ed4ac8b3d7dd234e473e774aec89b545c742b9 Mon Sep 17 00:00:00 2001 From: Grzegorz Karch Date: Wed, 22 Apr 2026 17:50:27 +0200 Subject: [PATCH 8/8] using branch name for fork repo Signed-off-by: Grzegorz Karch --- examples/puzzletron/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index 915fdb3ddc7..89183073399 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -277,7 +277,7 @@ To deploy a compressed model in vLLM, install vLLM fork with AnyModel enabled: ```bash git clone https://github.com/askliar/vllm.git cd vllm -git checkout 6a5ea42e55b85444da1a511a007f2d627845536f +git checkout feature/add_anymodel_to_vllm VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto ```