diff --git a/docs/source/Instruction/Command-line-parameters.md b/docs/source/Instruction/Command-line-parameters.md index cba43833e4..0d35d9fa76 100644 --- a/docs/source/Instruction/Command-line-parameters.md +++ b/docs/source/Instruction/Command-line-parameters.md @@ -54,6 +54,11 @@ - max_memory: device_map设置为'auto'或者'sequential'时，会根据max_memory进行模型权重的device分配，例如：`--max_memory '{0: "20GB", 1: "20GB"}'`。默认为None。该参数会透传入transformers的`from_pretrained`接口。 - local_repo_path: 部分模型在加载时依赖于github repo，例如[deepseek-vl2](https://github.com/deepseek-ai/DeepSeek-VL2)。为了避免`git clone`时遇到网络问题，可以直接使用本地repo。该参数需要传入本地repo的路径, 默认为`None`。 - init_strategy: 加载模型时，初始化模型中所有未初始化的参数（自定义模型架构时）。可选为'zero', 'uniform', 'normal', 'xavier_uniform', 'xavier_normal', 'kaiming_uniform', 'kaiming_normal', 'orthogonal'。默认为None。 +- load_model: 是否实例化模型。默认为True。设置为False时，只准备config和processor（tokenizer/image_processor），返回的model为None，且**不会下载权重文件**（`.bin`/`.safetensors`）。该参数通常用于调试tokenize过程和对话模板。 +- return_dummy_model: 是否只根据`config.json`构建模型（即`cls(config)`），跳过`from_pretrained`。默认为False。设置为True时，**模型架构完整但参数为随机初始化，且不会下载权重文件**。该参数需要`load_model`为True（否则拿不到模型对象）。建议结合`--device_map meta`使用，在meta设备上建图从而不占用真实内存。 + - 提示：`load_model`和`return_dummy_model`只要有一个「生效」就不会加载权重，即唯一会加载权重的组合是`load_model=true`且`return_dummy_model=false`（默认值）。 + - 注意：meta设备上的参数无法直接进行前向推理，如需推理请先调用`model.to_empty(device='cpu')`。 + - 注意：设置`return_dummy_model=true`时，`new_special_tokens`不会触发`resize_token_embeddings`，只会修改config中的`vocab_size`。 ### 数据参数 @@ -803,6 +808,8 @@ App参数继承于[部署参数](#部署参数), [Web-UI参数](#Web-UI参数) - 提示：你可以通过`--split_dataset_ratio`或者`--val_dataset`指定验证集内容。 - template_mode: 用于支持对`swift rlhf`训练的`cached_dataset`功能。该参数只在`--to_cached_dataset true`时生效。可选项包括: 'train'、'rlhf'和'kto'。其中`swift pt/sft`使用'train'，`swift rlhf --rlhf_type kto`使用'kto'，其他rlhf算法使用'rlhf'。注意：当前'gkd', 'ppo', 'grpo'算法不支持`cached_dataset`功能。默认为'train'。 - to_ollama: 产生ollama所需的Modelfile文件。默认为False。 +- to_model_summary: 打印模型架构、参数量统计以及tokenizer/template信息，默认为False。结合`--return_dummy_model true`可以在**不加载任何预训练权重**的情况下查看完整模型架构；结合`--load_model false`则只查看tokenizer和template。若指定了`--output_dir`，还会输出`model_summary.json`和`model_architecture.txt`。例子参考[这里](https://github.com/modelscope/ms-swift/tree/main/examples/export/model_summary)。 + - 注意：该参数与`--merge_lora`、`--quant_method`不兼容，因为二者都需要加载预训练权重。 - 🔥to_mcore: HF格式权重转成Megatron格式。默认为False。 - to_hf: Megatron格式权重转成HF格式。默认为False。 - mcore_model: mcore格式模型路径。默认为None。 diff --git a/docs/source_en/Instruction/Command-line-parameters.md b/docs/source_en/Instruction/Command-line-parameters.md index 9302c55f7b..4d2a179f6e 100644 --- a/docs/source_en/Instruction/Command-line-parameters.md +++ b/docs/source_en/Instruction/Command-line-parameters.md @@ -54,6 +54,11 @@ The command-line arguments will be introduced in four categories: basic argument - max_memory: When `device_map` is set to `'auto'` or `'sequential'`, model weights are allocated across devices according to `max_memory`, e.g., `--max_memory '{0: "20GB", 1: "20GB"}'`. Default is `None`. Passed through to the `from_pretrained` interface in Transformers. - local_repo_path: Some models depend on GitHub repositories during loading, e.g., [deepseek-vl2](https://github.com/deepseek-ai/DeepSeek-VL2). To avoid network issues during `git clone`, you can use a local repository. This parameter takes the path to the local repo. Default is `None`. - init_strategy: Strategy for initializing uninitialized parameters when loading a model (especially useful for custom architectures). Options: `'zero'`, `'uniform'`, `'normal'`, `'xavier_uniform'`, `'xavier_normal'`, `'kaiming_uniform'`, `'kaiming_normal'`, `'orthogonal'`. Default is `None`. +- load_model: Whether to instantiate the model. Default is `True`. When set to `False`, only the config and the processor (tokenizer/image_processor) are prepared, the returned model is `None`, and the **weight files are not downloaded** (`.bin`/`.safetensors`). Useful for debugging tokenization and chat templates. +- return_dummy_model: Whether to build the model from `config.json` only (i.e. `cls(config)`), skipping `from_pretrained`. Default is `False`. When set to `True`, **the architecture is complete but the parameters are randomly initialized, and the weight files are not downloaded**. Requires `load_model` to be `True` (otherwise no model object is returned). It is recommended to combine it with `--device_map meta` so the model is built on the meta device and no real memory is allocated. + - Tip: weights are skipped as soon as either flag "takes effect"; the only combination that loads weights is `load_model=true` and `return_dummy_model=false` (the defaults). + - Note: parameters on the meta device cannot be used for a forward pass directly; call `model.to_empty(device='cpu')` first. + - Note: when `return_dummy_model=true`, `new_special_tokens` does not trigger `resize_token_embeddings`; only `vocab_size` in the config is updated. ### Data Arguments - 🔥dataset: A list of dataset IDs or paths. Default is `[]`. Each dataset should be specified in the format: `'dataset_id_or_path:subset#sample_size'`, where subset and sample size are optional. Local datasets support formats such as jsonl, csv, json, and folders. **Open-source datasets from the hub can be used offline by `git clone`-ing them locally and passing the local folder path**. For custom dataset formats, refer to the [Custom Dataset Documentation](../Customization/Custom-dataset.md). You can use multiple datasets by passing `--dataset `. @@ -824,6 +829,8 @@ Export Arguments include the [basic arguments](#base-arguments) and [merge argum - Note: You can specify the validation set content through `--split_dataset_ratio` or `--val_dataset`. - template_mode: Used to support the `cached_dataset` feature for `swift rlhf` training. This parameter only takes effect when `--to_cached_dataset true` is set. Available options include: 'train', 'rlhf', and 'kto'. Among them, `swift pt/sft` uses 'train', `swift rlhf --rlhf_type kto` uses 'kto', and other rlhf algorithms use 'rlhf'. Note: Currently, 'gkd', 'ppo', and 'grpo' algorithms do not support the `cached_dataset` feature. Default is 'train'. - to_ollama: Generate the Modelfile required by Ollama. Default is False. +- to_model_summary: Print the model architecture, parameter statistics, and the tokenizer/template information. Default is False. Combine it with `--return_dummy_model true` to inspect the full architecture **without loading any pretrained weights**, or with `--load_model false` to inspect only the tokenizer and template. If `--output_dir` is specified, `model_summary.json` and `model_architecture.txt` are also written. See the example [here](https://github.com/modelscope/ms-swift/tree/main/examples/export/model_summary). + - Note: this argument is incompatible with `--merge_lora` and `--quant_method`, because both of them require the pretrained weights to be loaded. - 🔥to_mcore: Convert weights from HF format to Megatron format. Default is False. - to_hf: Convert weights from Megatron format to HF format. Default is False. - mcore_model: Path to the mcore format model. Default is None. diff --git a/examples/export/model_summary/dummy_model.sh b/examples/export/model_summary/dummy_model.sh new file mode 100755 index 0000000000..e28a364da3 --- /dev/null +++ b/examples/export/model_summary/dummy_model.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# 查看 Qwen3 的完整模型架构，但不加载任何预训练权重。 +# +# 原理： +# --return_dummy_model true 通过 cls(config) 按 config.json 搭建完整结构， +# 跳过 from_pretrained，因此不读权重； +# 同时 safetensors / bin 权重文件也不会被下载。 +# --device_map meta 在 meta 设备上建图，不分配真实内存（8B 模型也是 0 内存）。 +# --load_model true 必须为 true，否则拿不到模型对象。 +# +# 注意：meta 设备上的参数不能直接前向推理，如需前向请先 model.to_empty(device='cpu')。 + +set -e + +MODEL=${MODEL:-Qwen/Qwen3-8B} +OUTPUT_DIR=${OUTPUT_DIR:-./output/qwen3-model-summary} + +# CPU 环境显式关闭 GPU，避免 device_map 自动探测 +export CUDA_VISIBLE_DEVICES="" + +swift export \ + --model "$MODEL" \ + --to_model_summary true \ + --load_model true \ + --return_dummy_model true \ + --device_map meta \ + --torch_dtype bfloat16 \ + --output_dir "$OUTPUT_DIR" \ + --exist_ok true diff --git a/examples/export/model_summary/no_model.sh b/examples/export/model_summary/no_model.sh new file mode 100755 index 0000000000..2161c19b1c --- /dev/null +++ b/examples/export/model_summary/no_model.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# 只查看 tokenizer 与 template，不实例化模型（model 为 None）。 +# +# 原理： +# --load_model false 跳过 get_model()，只准备 config + processor(tokenizer)， +# 权重文件同样不会被下载。 +# +# 适用场景：调试对话模板、检查 tokenize 结果、确认 max_length / special tokens。 +# 对 Qwen3 这类纯文本模型（template.use_model = False），这个模式已经够用， +# 不需要 dummy model。 + +set -e + +MODEL=${MODEL:-Qwen/Qwen3-8B} +OUTPUT_DIR=${OUTPUT_DIR:-./output/qwen3-tokenizer-summary} + +export CUDA_VISIBLE_DEVICES="" + +swift export \ + --model "$MODEL" \ + --to_model_summary true \ + --load_model false \ + --output_dir "$OUTPUT_DIR" \ + --exist_ok true diff --git a/swift/arguments/base_args/model_args.py b/swift/arguments/base_args/model_args.py index 4caeed8802..465e8d52d3 100644 --- a/swift/arguments/base_args/model_args.py +++ b/swift/arguments/base_args/model_args.py @@ -64,6 +64,13 @@ class ModelArguments: init_strategy (Optional[str]): The strategy to initialize all uninitialized parameters when loading a model (especially for custom architectures). Options include 'zero', 'uniform', 'normal', 'xavier_uniform', 'xavier_normal', 'kaiming_uniform', 'kaiming_normal', 'orthogonal'. Defaults to None. + load_model (bool): Whether to instantiate the model. If False, only the config and the processor + (tokenizer/image_processor) are prepared, and the model weight files (.bin/.safetensors) are not even + downloaded. Useful for tokenization/template debugging. Defaults to True. + return_dummy_model (bool): Whether to build the model from `config.json` only (i.e. `cls(config)`), skipping + `from_pretrained`. The architecture is complete but the parameters are randomly initialized, and the + weight files are not downloaded. Requires `load_model` to be True. Combine with `--device_map meta` to + build the model on the meta device so that no real memory is allocated. Defaults to False. """ model: Optional[str] = None # model id or model path model_type: Optional[str] = field( @@ -91,6 +98,10 @@ class ModelArguments: local_repo_path: Optional[str] = None init_strategy: Literal['zero', 'uniform', 'normal', 'xavier_uniform', 'xavier_normal', 'kaiming_uniform', 'kaiming_normal', 'orthogonal'] = None + # False: only build the config & processor, the model will be None. + load_model: bool = True + # True: build the model via `cls(config)` instead of `from_pretrained`, i.e. no weights are loaded. + return_dummy_model: bool = False def _init_device_map(self): """Prepare device map args""" @@ -190,7 +201,7 @@ def _init_rope_scaling(self): def _init_model_info(self) -> torch.dtype: model_kwargs = self.get_model_kwargs() - if self.tuner_backend == 'unsloth': + if self.tuner_backend == 'unsloth' and self.load_model: model_kwargs['download_model'] = True self.model_info, self.model_meta = get_model_info_meta(**model_kwargs) self.task_type = self.model_info.task_type @@ -216,11 +227,25 @@ def _init_new_special_tokens(self): new_special_tokens.append(token) self.new_special_tokens = new_special_tokens + def _init_load_model(self): + """Validate `load_model` / `return_dummy_model` and log the effective weight-loading behavior.""" + if self.return_dummy_model and not self.load_model: + raise ValueError('`--return_dummy_model true` requires `--load_model true`, because a dummy model still ' + 'needs to be instantiated. If you only need the tokenizer/template, just set ' + '`--load_model false`.') + if not self.load_model: + logger.info('Setting args.load_model: False. The model will not be instantiated (model is None), ' + 'and the model weight files will not be downloaded.') + elif self.return_dummy_model: + logger.info('Setting args.return_dummy_model: True. The model will be built from `config.json` with ' + 'randomly initialized parameters, and the model weight files will not be downloaded.') + def __post_init__(self): if self.model is None: raise ValueError(f'Please set --model `, model: {self.model}') self._init_new_special_tokens() self.model_suffix = get_model_name(self.model) + self._init_load_model() self._init_device_map() self._init_max_memory() self._init_torch_dtype() @@ -246,4 +271,6 @@ def get_model_kwargs(self): 'num_labels': self.num_labels, 'problem_type': self.problem_type, 'init_strategy': self.init_strategy, + 'load_model': self.load_model, + 'return_dummy_model': self.return_dummy_model, } diff --git a/swift/arguments/export_args.py b/swift/arguments/export_args.py index cf08d8c72a..e873486920 100644 --- a/swift/arguments/export_args.py +++ b/swift/arguments/export_args.py @@ -28,6 +28,10 @@ class ExportArguments(MergeArguments, BaseArguments): to False. Note: You can specify the validation set content through `--split_dataset_ratio` or `--val_dataset`. to_ollama (bool): Whether to generate the `Modelfile` required by Ollama. Defaults to False. + to_model_summary (bool): Whether to print the model architecture / parameter statistics and the + tokenizer & template information. Combine with `--return_dummy_model true` to inspect the full + architecture without loading any pretrained weights, or with `--load_model false` to only inspect + the tokenizer/template. Defaults to False. to_mcore (bool): Whether to convert Hugging Face format weights to Megatron-Core format. Defaults to False. to_hf (bool): Whether to convert Megatron-Core format weights to Hugging Face format. Defaults to False. mcore_model (Optional[str]): The path to the Megatron-Core format model. Defaults to None. @@ -63,6 +67,9 @@ class ExportArguments(MergeArguments, BaseArguments): # ollama to_ollama: bool = False + # model summary (architecture / tokenizer / template inspection) + to_model_summary: bool = False + # megatron to_mcore: bool = False to_hf: bool = False @@ -87,6 +94,16 @@ def load_args_from_ckpt(self) -> None: return super().load_args_from_ckpt() + def _init_model_summary(self): + if not self.to_model_summary: + return + if self.merge_lora: + raise ValueError('`--to_model_summary true` does not support `--merge_lora true`, because merging LoRA ' + 'requires the pretrained weights to be loaded.') + if self.quant_method is not None: + raise ValueError('`--to_model_summary true` does not support `--quant_method`, because quantization ' + 'requires the pretrained weights to be loaded.') + def _init_output_dir(self): if self.output_dir is None: ckpt_dir = self.ckpt_dir or f'./{self.model_suffix}' @@ -107,6 +124,8 @@ def _init_output_dir(self): suffix = 'hf' elif self.to_cached_dataset: suffix = 'cached_dataset' + elif self.to_model_summary: + suffix = 'model_summary' else: return @@ -137,6 +156,7 @@ def __post_init__(self): set_default_ddp_config() init_process_group(backend=self.ddp_backend, timeout=self.ddp_timeout) + self._init_model_summary() BaseArguments.__post_init__(self) self._init_output_dir() self.test_convert_dtype = HfConfigFactory.to_torch_dtype(self.test_convert_dtype) diff --git a/swift/model/patcher.py b/swift/model/patcher.py index f52b72c850..c2414b8f5a 100644 --- a/swift/model/patcher.py +++ b/swift/model/patcher.py @@ -8,7 +8,7 @@ import torch.nn.functional as F import transformers from accelerate.utils import find_device -from contextlib import contextmanager +from contextlib import contextmanager, nullcontext from functools import wraps from packaging import version from peft import PeftModel @@ -388,10 +388,20 @@ def _new_from_pretrained(cls, *args, **kwargs): if hasattr(cls, '_tp_plan'): # fix tp_plan cls._tp_plan = cls._tp_plan or {} if return_dummy_model: + # `cls(config)` does not accept `device_map`, so honor `device_map='meta'` via a device context manager, + # otherwise the dummy model would be materialized on CPU and consume real memory. + # Note: only 'meta' is handled here. Other values (including 'cpu') must not open a device context, + # because callers such as `Template._get_model` already wrap this call in `torch.device('meta')` + # and an inner context would silently override it. + device_map = kwargs.get('device_map') + device_context = torch.device('meta') if device_map == 'meta' else nullcontext() origin_torch_dtype = torch.get_default_dtype() torch.set_default_dtype(kwargs['config'].torch_dtype) - model = cls(copy.deepcopy(kwargs['config'])) - torch.set_default_dtype(origin_torch_dtype) + try: + with device_context: + model = cls(copy.deepcopy(kwargs['config'])) + finally: + torch.set_default_dtype(origin_torch_dtype) else: model = from_pretrained(cls, *args, **kwargs) return model diff --git a/swift/pipelines/export/__init__.py b/swift/pipelines/export/__init__.py index d74170d7c0..10795d9ea7 100644 --- a/swift/pipelines/export/__init__.py +++ b/swift/pipelines/export/__init__.py @@ -2,5 +2,6 @@ from .cached_dataset import export_cached_dataset from .export import SwiftExport, export_main from .merge_lora import merge_lora +from .model_summary import export_model_summary from .ollama import export_to_ollama from .quant import quantize_model diff --git a/swift/pipelines/export/export.py b/swift/pipelines/export/export.py index 2f202bbc41..9b5f5555eb 100644 --- a/swift/pipelines/export/export.py +++ b/swift/pipelines/export/export.py @@ -7,6 +7,7 @@ from swift.utils import get_logger from .cached_dataset import export_cached_dataset from .merge_lora import merge_lora +from .model_summary import export_model_summary from .ollama import export_to_ollama from .quant import quantize_model @@ -31,6 +32,8 @@ def run(self): quantize_model(args) elif args.to_ollama: export_to_ollama(args) + elif args.to_model_summary: + export_model_summary(args) elif args.to_cached_dataset: export_cached_dataset(args) elif args.to_hf or args.mcore_adapter and args.to_mcore: diff --git a/swift/pipelines/export/model_summary.py b/swift/pipelines/export/model_summary.py new file mode 100644 index 0000000000..49f6655161 --- /dev/null +++ b/swift/pipelines/export/model_summary.py @@ -0,0 +1,102 @@ +# Copyright (c) ModelScope Contributors. All rights reserved. +import json +import os +from typing import List, Optional, Union + +from swift.arguments import ExportArguments +from swift.utils import get_logger + +logger = get_logger() + + +def _human_readable(num: int) -> str: + for unit in ['', 'K', 'M', 'B']: + if abs(num) < 1000: + return f'{num:.2f}{unit}' if unit else str(num) + num /= 1000 + return f'{num:.2f}T' + + +def _count_parameters(model): + """Count parameters without touching the tensor data (safe on the meta device).""" + total, by_dtype = 0, {} + for _, param in model.named_parameters(): + numel = param.numel() + total += numel + key = str(param.dtype).replace('torch.', '') + by_dtype[key] = by_dtype.get(key, 0) + numel + return total, by_dtype + + +def export_model_summary(args: ExportArguments) -> None: + """Print the model architecture / parameter statistics without loading any pretrained weights. + + This is driven by `--load_model` and `--return_dummy_model`: + - `--return_dummy_model true`: the full architecture is built from `config.json`, no weights are read. + - `--load_model false`: only the tokenizer/template are prepared, so no architecture is printed. + """ + model, processor = args.get_model_processor() + template = args.get_template(processor) + if model is not None and template.use_model: + template.model = model + + logger.info(f'model_type: {args.model_type}') + logger.info(f'model_dir: {args.model_dir}') + logger.info(f'task_type: {args.task_type}') + logger.info(f'torch_dtype: {args.torch_dtype}') + logger.info(f'template: {template.template_meta.template_type}') + logger.info(f'max_length: {template.max_length}') + logger.info(f'tokenizer: {type(template.tokenizer).__name__}, vocab_size: {len(template.tokenizer)}') + + summary = { + 'model': args.model, + 'model_type': args.model_type, + 'model_dir': args.model_dir, + 'task_type': args.task_type, + 'torch_dtype': str(args.torch_dtype), + 'template': template.template_meta.template_type, + 'max_length': template.max_length, + 'tokenizer_class': type(template.tokenizer).__name__, + 'vocab_size': len(template.tokenizer), + 'load_model': args.load_model, + 'return_dummy_model': args.return_dummy_model, + } + + if model is None: + logger.info('The model was not instantiated because `--load_model false` was set. ' + 'Set `--return_dummy_model true` if you want to inspect the architecture ' + 'without loading the weights.') + else: + n_params, by_dtype = _count_parameters(model) + devices = sorted({str(p.device) for p in model.parameters()}) + logger.info(f'model architecture:\n{model}') + logger.info(f'model_class: {type(model).__name__}') + logger.info(f'num_parameters: {n_params} ({_human_readable(n_params)})') + logger.info(f'parameters_by_dtype: {by_dtype}') + logger.info(f'parameter devices: {devices}') + if 'meta' in devices: + logger.info('The parameters live on the meta device, so no real memory is allocated. ' + 'They cannot be used for a forward pass; call `model.to_empty(device="cpu")` first.') + summary.update({ + 'model_class': type(model).__name__, + 'num_parameters': n_params, + 'num_parameters_readable': _human_readable(n_params), + 'parameters_by_dtype': by_dtype, + 'parameter_devices': devices, + }) + + if args.output_dir: + os.makedirs(args.output_dir, exist_ok=True) + summary_path = os.path.join(args.output_dir, 'model_summary.json') + with open(summary_path, 'w', encoding='utf-8') as f: + json.dump(summary, f, ensure_ascii=False, indent=2) + logger.info(f'The model summary has been saved to: `{summary_path}`') + if model is not None: + arch_path = os.path.join(args.output_dir, 'model_architecture.txt') + with open(arch_path, 'w', encoding='utf-8') as f: + f.write(str(model)) + logger.info(f'The model architecture has been saved to: `{arch_path}`') + + +def model_summary_main(args: Optional[Union[List[str], ExportArguments]] = None): + return export_model_summary(args)