Support model initialization without loading pretrained weights

docs/source/Instruction/Command-line-parameters.md

@@ -54,6 +54,11 @@
 - max_memory: device_map设置为'auto'或者'sequential'时，会根据max_memory进行模型权重的device分配，例如：`--max_memory '{0: "20GB", 1: "20GB"}'`。默认为None。该参数会透传入transformers的`from_pretrained`接口。
 - local_repo_path: 部分模型在加载时依赖于github repo，例如[deepseek-vl2](https://github.com/deepseek-ai/DeepSeek-VL2)。为了避免`git clone`时遇到网络问题，可以直接使用本地repo。该参数需要传入本地repo的路径, 默认为`None`。
 - init_strategy: 加载模型时，初始化模型中所有未初始化的参数（自定义模型架构时）。可选为'zero', 'uniform', 'normal', 'xavier_uniform', 'xavier_normal', 'kaiming_uniform', 'kaiming_normal', 'orthogonal'。默认为None。
+- load_model: 是否实例化模型。默认为True。设置为False时，只准备config和processor（tokenizer/image_processor），返回的model为None，且**不会下载权重文件**（`.bin`/`.safetensors`）。该参数通常用于调试tokenize过程和对话模板。
+- return_dummy_model: 是否只根据`config.json`构建模型（即`cls(config)`），跳过`from_pretrained`。默认为False。设置为True时，**模型架构完整但参数为随机初始化，且不会下载权重文件**。该参数需要`load_model`为True（否则拿不到模型对象）。建议结合`--device_map meta`使用，在meta设备上建图从而不占用真实内存。
+  - 提示：`load_model`和`return_dummy_model`只要有一个「生效」就不会加载权重，即唯一会加载权重的组合是`load_model=true`且`return_dummy_model=false`（默认值）。
+  - 注意：meta设备上的参数无法直接进行前向推理，如需推理请先调用`model.to_empty(device='cpu')`。
+  - 注意：设置`return_dummy_model=true`时，`new_special_tokens`不会触发`resize_token_embeddings`，只会修改config中的`vocab_size`。
 
 
 ### 数据参数
@@ -803,6 +808,8 @@ App参数继承于[部署参数](#部署参数), [Web-UI参数](#Web-UI参数)
   - 提示：你可以通过`--split_dataset_ratio`或者`--val_dataset`指定验证集内容。
 - template_mode: 用于支持对`swift rlhf`训练的`cached_dataset`功能。该参数只在`--to_cached_dataset true`时生效。可选项包括: 'train'、'rlhf'和'kto'。其中`swift pt/sft`使用'train'，`swift rlhf --rlhf_type kto`使用'kto'，其他rlhf算法使用'rlhf'。注意：当前'gkd', 'ppo', 'grpo'算法不支持`cached_dataset`功能。默认为'train'。
 - to_ollama: 产生ollama所需的Modelfile文件。默认为False。
+- to_model_summary: 打印模型架构、参数量统计以及tokenizer/template信息，默认为False。结合`--return_dummy_model true`可以在**不加载任何预训练权重**的情况下查看完整模型架构；结合`--load_model false`则只查看tokenizer和template。若指定了`--output_dir`，还会输出`model_summary.json`和`model_architecture.txt`。例子参考[这里](https://github.com/modelscope/ms-swift/tree/main/examples/export/model_summary)。
+  - 注意：该参数与`--merge_lora`、`--quant_method`不兼容，因为二者都需要加载预训练权重。
 - 🔥to_mcore: HF格式权重转成Megatron格式。默认为False。
 - to_hf: Megatron格式权重转成HF格式。默认为False。
 - mcore_model: mcore格式模型路径。默认为None。

docs/source_en/Instruction/Command-line-parameters.md

@@ -54,6 +54,11 @@ The command-line arguments will be introduced in four categories: basic argument
 - max_memory: When `device_map` is set to `'auto'` or `'sequential'`, model weights are allocated across devices according to `max_memory`, e.g., `--max_memory '{0: "20GB", 1: "20GB"}'`. Default is `None`. Passed through to the `from_pretrained` interface in Transformers.
 - local_repo_path: Some models depend on GitHub repositories during loading, e.g., [deepseek-vl2](https://github.com/deepseek-ai/DeepSeek-VL2). To avoid network issues during `git clone`, you can use a local repository. This parameter takes the path to the local repo. Default is `None`.
 - init_strategy: Strategy for initializing uninitialized parameters when loading a model (especially useful for custom architectures). Options: `'zero'`, `'uniform'`, `'normal'`, `'xavier_uniform'`, `'xavier_normal'`, `'kaiming_uniform'`, `'kaiming_normal'`, `'orthogonal'`. Default is `None`.
+- load_model: Whether to instantiate the model. Default is `True`. When set to `False`, only the config and the processor (tokenizer/image_processor) are prepared, the returned model is `None`, and the **weight files are not downloaded** (`.bin`/`.safetensors`). Useful for debugging tokenization and chat templates.
+- return_dummy_model: Whether to build the model from `config.json` only (i.e. `cls(config)`), skipping `from_pretrained`. Default is `False`. When set to `True`, **the architecture is complete but the parameters are randomly initialized, and the weight files are not downloaded**. Requires `load_model` to be `True` (otherwise no model object is returned). It is recommended to combine it with `--device_map meta` so the model is built on the meta device and no real memory is allocated.
+  - Tip: weights are skipped as soon as either flag "takes effect"; the only combination that loads weights is `load_model=true` and `return_dummy_model=false` (the defaults).
+  - Note: parameters on the meta device cannot be used for a forward pass directly; call `model.to_empty(device='cpu')` first.
+  - Note: when `return_dummy_model=true`, `new_special_tokens` does not trigger `resize_token_embeddings`; only `vocab_size` in the config is updated.
 
 ### Data Arguments
 - 🔥dataset: A list of dataset IDs or paths. Default is `[]`. Each dataset should be specified in the format: `'dataset_id_or_path:subset#sample_size'`, where subset and sample size are optional. Local datasets support formats such as jsonl, csv, json, and folders. **Open-source datasets from the hub can be used offline by `git clone`-ing them locally and passing the local folder path**. For custom dataset formats, refer to the [Custom Dataset Documentation](../Customization/Custom-dataset.md). You can use multiple datasets by passing `--dataset <dataset1> <dataset2>`.
@@ -824,6 +829,8 @@ Export Arguments include the [basic arguments](#base-arguments) and [merge argum
   - Note: You can specify the validation set content through `--split_dataset_ratio` or `--val_dataset`.
 - template_mode: Used to support the `cached_dataset` feature for `swift rlhf` training. This parameter only takes effect when `--to_cached_dataset true` is set. Available options include: 'train', 'rlhf', and 'kto'. Among them, `swift pt/sft` uses 'train', `swift rlhf --rlhf_type kto` uses 'kto', and other rlhf algorithms use 'rlhf'. Note: Currently, 'gkd', 'ppo', and 'grpo' algorithms do not support the `cached_dataset` feature. Default is 'train'.
 - to_ollama: Generate the Modelfile required by Ollama. Default is False.
+- to_model_summary: Print the model architecture, parameter statistics, and the tokenizer/template information. Default is False. Combine it with `--return_dummy_model true` to inspect the full architecture **without loading any pretrained weights**, or with `--load_model false` to inspect only the tokenizer and template. If `--output_dir` is specified, `model_summary.json` and `model_architecture.txt` are also written. See the example [here](https://github.com/modelscope/ms-swift/tree/main/examples/export/model_summary).
+  - Note: this argument is incompatible with `--merge_lora` and `--quant_method`, because both of them require the pretrained weights to be loaded.
 - 🔥to_mcore: Convert weights from HF format to Megatron format. Default is False.
 - to_hf: Convert weights from Megatron format to HF format. Default is False.
 - mcore_model: Path to the mcore format model. Default is None.

examples/export/model_summary/dummy_model.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+# 查看 Qwen3 的完整模型架构，但不加载任何预训练权重。
+#
+# 原理：
+#   --return_dummy_model true  通过 cls(config) 按 config.json 搭建完整结构，
+#                              跳过 from_pretrained，因此不读权重；
+#                              同时 safetensors / bin 权重文件也不会被下载。
+#   --device_map meta          在 meta 设备上建图，不分配真实内存（8B 模型也是 0 内存）。
+#   --load_model true          必须为 true，否则拿不到模型对象。
+#
+# 注意：meta 设备上的参数不能直接前向推理，如需前向请先 model.to_empty(device='cpu')。
+
+set -e
+
+MODEL=${MODEL:-Qwen/Qwen3-8B}
+OUTPUT_DIR=${OUTPUT_DIR:-./output/qwen3-model-summary}
+
+# CPU 环境显式关闭 GPU，避免 device_map 自动探测
+export CUDA_VISIBLE_DEVICES=""
+
+swift export \
+    --model "$MODEL" \
+    --to_model_summary true \
+    --load_model true \
+    --return_dummy_model true \
+    --device_map meta \
+    --torch_dtype bfloat16 \
+    --output_dir "$OUTPUT_DIR" \
+    --exist_ok true

examples/export/model_summary/no_model.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+# 只查看 tokenizer 与 template，不实例化模型（model 为 None）。
+#
+# 原理：
+#   --load_model false  跳过 get_model()，只准备 config + processor(tokenizer)，
+#                       权重文件同样不会被下载。
+#
+# 适用场景：调试对话模板、检查 tokenize 结果、确认 max_length / special tokens。
+# 对 Qwen3 这类纯文本模型（template.use_model = False），这个模式已经够用，
+# 不需要 dummy model。
+
+set -e
+
+MODEL=${MODEL:-Qwen/Qwen3-8B}
+OUTPUT_DIR=${OUTPUT_DIR:-./output/qwen3-tokenizer-summary}
+
+export CUDA_VISIBLE_DEVICES=""
+
+swift export \
+    --model "$MODEL" \
+    --to_model_summary true \
+    --load_model false \
+    --output_dir "$OUTPUT_DIR" \
+    --exist_ok true

swift/arguments/base_args/model_args.py

@@ -64,6 +64,13 @@ class ModelArguments:
         init_strategy (Optional[str]): The strategy to initialize all uninitialized parameters when loading a model
             (especially for custom architectures). Options include 'zero', 'uniform', 'normal', 'xavier_uniform',
             'xavier_normal', 'kaiming_uniform', 'kaiming_normal', 'orthogonal'. Defaults to None.
+        load_model (bool): Whether to instantiate the model. If False, only the config and the processor
+            (tokenizer/image_processor) are prepared, and the model weight files (.bin/.safetensors) are not even
+            downloaded. Useful for tokenization/template debugging. Defaults to True.
+        return_dummy_model (bool): Whether to build the model from `config.json` only (i.e. `cls(config)`), skipping
+            `from_pretrained`. The architecture is complete but the parameters are randomly initialized, and the
+            weight files are not downloaded. Requires `load_model` to be True. Combine with `--device_map meta` to
+            build the model on the meta device so that no real memory is allocated. Defaults to False.
     """
     model: Optional[str] = None  # model id or model path
     model_type: Optional[str] = field(
@@ -91,6 +98,10 @@ class ModelArguments:
     local_repo_path: Optional[str] = None
     init_strategy: Literal['zero', 'uniform', 'normal', 'xavier_uniform', 'xavier_normal', 'kaiming_uniform',
                            'kaiming_normal', 'orthogonal'] = None
+    # False: only build the config & processor, the model will be None.
+    load_model: bool = True
+    # True: build the model via `cls(config)` instead of `from_pretrained`, i.e. no weights are loaded.
+    return_dummy_model: bool = False
 
     def _init_device_map(self):
         """Prepare device map args"""
@@ -190,7 +201,7 @@ def _init_rope_scaling(self):
 
     def _init_model_info(self) -> torch.dtype:
         model_kwargs = self.get_model_kwargs()
-        if self.tuner_backend == 'unsloth':
+        if self.tuner_backend == 'unsloth' and self.load_model:
             model_kwargs['download_model'] = True
         self.model_info, self.model_meta = get_model_info_meta(**model_kwargs)
         self.task_type = self.model_info.task_type
@@ -216,11 +227,25 @@ def _init_new_special_tokens(self):
                 new_special_tokens.append(token)
         self.new_special_tokens = new_special_tokens
 
+    def _init_load_model(self):
+        """Validate `load_model` / `return_dummy_model` and log the effective weight-loading behavior."""
+        if self.return_dummy_model and not self.load_model:
+            raise ValueError('`--return_dummy_model true` requires `--load_model true`, because a dummy model still '
+                             'needs to be instantiated. If you only need the tokenizer/template, just set '
+                             '`--load_model false`.')
+        if not self.load_model:
+            logger.info('Setting args.load_model: False. The model will not be instantiated (model is None), '
+                        'and the model weight files will not be downloaded.')
+        elif self.return_dummy_model:
+            logger.info('Setting args.return_dummy_model: True. The model will be built from `config.json` with '
+                        'randomly initialized parameters, and the model weight files will not be downloaded.')
+
     def __post_init__(self):
         if self.model is None:
             raise ValueError(f'Please set --model <model_id_or_path>`, model: {self.model}')
         self._init_new_special_tokens()
         self.model_suffix = get_model_name(self.model)
+        self._init_load_model()
         self._init_device_map()
         self._init_max_memory()
         self._init_torch_dtype()
@@ -246,4 +271,6 @@ def get_model_kwargs(self):
             'num_labels': self.num_labels,
             'problem_type': self.problem_type,
             'init_strategy': self.init_strategy,
+            'load_model': self.load_model,
+            'return_dummy_model': self.return_dummy_model,
         }

swift/arguments/export_args.py

@@ -28,6 +28,10 @@ class ExportArguments(MergeArguments, BaseArguments):
             to False. Note: You can specify the validation set content through
             `--split_dataset_ratio` or `--val_dataset`.
         to_ollama (bool): Whether to generate the `Modelfile` required by Ollama. Defaults to False.
+        to_model_summary (bool): Whether to print the model architecture / parameter statistics and the
+            tokenizer & template information. Combine with `--return_dummy_model true` to inspect the full
+            architecture without loading any pretrained weights, or with `--load_model false` to only inspect
+            the tokenizer/template. Defaults to False.
         to_mcore (bool): Whether to convert Hugging Face format weights to Megatron-Core format. Defaults to False.
         to_hf (bool): Whether to convert Megatron-Core format weights to Hugging Face format. Defaults to False.
         mcore_model (Optional[str]): The path to the Megatron-Core format model. Defaults to None.
@@ -63,6 +67,9 @@ class ExportArguments(MergeArguments, BaseArguments):
     # ollama
     to_ollama: bool = False
 
+    # model summary (architecture / tokenizer / template inspection)
+    to_model_summary: bool = False
+
     # megatron
     to_mcore: bool = False
     to_hf: bool = False
@@ -87,6 +94,16 @@ def load_args_from_ckpt(self) -> None:
             return
         super().load_args_from_ckpt()
 
+    def _init_model_summary(self):
+        if not self.to_model_summary:
+            return
+        if self.merge_lora:
+            raise ValueError('`--to_model_summary true` does not support `--merge_lora true`, because merging LoRA '
+                             'requires the pretrained weights to be loaded.')
+        if self.quant_method is not None:
+            raise ValueError('`--to_model_summary true` does not support `--quant_method`, because quantization '
+                             'requires the pretrained weights to be loaded.')
+
     def _init_output_dir(self):
         if self.output_dir is None:
             ckpt_dir = self.ckpt_dir or f'./{self.model_suffix}'
@@ -107,6 +124,8 @@ def _init_output_dir(self):
                 suffix = 'hf'
             elif self.to_cached_dataset:
                 suffix = 'cached_dataset'
+            elif self.to_model_summary:
+                suffix = 'model_summary'
             else:
                 return
 
@@ -137,6 +156,7 @@ def __post_init__(self):
                 set_default_ddp_config()
                 init_process_group(backend=self.ddp_backend, timeout=self.ddp_timeout)
 
+        self._init_model_summary()
         BaseArguments.__post_init__(self)
         self._init_output_dir()
         self.test_convert_dtype = HfConfigFactory.to_torch_dtype(self.test_convert_dtype)

swift/model/patcher.py

@@ -8,7 +8,7 @@
 import torch.nn.functional as F
 import transformers
 from accelerate.utils import find_device
-from contextlib import contextmanager
+from contextlib import contextmanager, nullcontext
 from functools import wraps
 from packaging import version
 from peft import PeftModel
@@ -388,10 +388,20 @@ def _new_from_pretrained(cls, *args, **kwargs):
         if hasattr(cls, '_tp_plan'):  # fix tp_plan
             cls._tp_plan = cls._tp_plan or {}
         if return_dummy_model:
+            # `cls(config)` does not accept `device_map`, so honor `device_map='meta'` via a device context manager,
+            # otherwise the dummy model would be materialized on CPU and consume real memory.
+            # Note: only 'meta' is handled here. Other values (including 'cpu') must not open a device context,
+            # because callers such as `Template._get_model` already wrap this call in `torch.device('meta')`
+            # and an inner context would silently override it.
+            device_map = kwargs.get('device_map')
+            device_context = torch.device('meta') if device_map == 'meta' else nullcontext()
             origin_torch_dtype = torch.get_default_dtype()
             torch.set_default_dtype(kwargs['config'].torch_dtype)
-            model = cls(copy.deepcopy(kwargs['config']))
-            torch.set_default_dtype(origin_torch_dtype)
+            try:
+                with device_context:
+                    model = cls(copy.deepcopy(kwargs['config']))
+            finally:
+                torch.set_default_dtype(origin_torch_dtype)
         else:
             model = from_pretrained(cls, *args, **kwargs)
         return model

swift/pipelines/export/__init__.py

@@ -2,5 +2,6 @@
 from .cached_dataset import export_cached_dataset
 from .export import SwiftExport, export_main
 from .merge_lora import merge_lora
+from .model_summary import export_model_summary
 from .ollama import export_to_ollama
 from .quant import quantize_model

swift/pipelines/export/export.py

@@ -7,6 +7,7 @@
 from swift.utils import get_logger
 from .cached_dataset import export_cached_dataset
 from .merge_lora import merge_lora
+from .model_summary import export_model_summary
 from .ollama import export_to_ollama
 from .quant import quantize_model
 
@@ -31,6 +32,8 @@ def run(self):
             quantize_model(args)
         elif args.to_ollama:
             export_to_ollama(args)
+        elif args.to_model_summary:
+            export_model_summary(args)
         elif args.to_cached_dataset:
             export_cached_dataset(args)
         elif args.to_hf or args.mcore_adapter and args.to_mcore: