diff --git a/.claude/scheduled_tasks.lock b/.claude/scheduled_tasks.lock new file mode 100644 index 00000000000..653c69bd0e0 --- /dev/null +++ b/.claude/scheduled_tasks.lock @@ -0,0 +1 @@ +{"sessionId":"c73837a8-dfc8-4c65-87dd-ba6efe62db78","pid":5301,"procStart":"863627582","acquiredAt":1780509271557} \ No newline at end of file diff --git a/examples/speculative_decoding/pipeline/eagle3/eagle3_new_model_triage_guide.md b/examples/speculative_decoding/pipeline/eagle3/eagle3_new_model_triage_guide.md index bd44ac35af7..cb7c73f6f7e 100644 --- a/examples/speculative_decoding/pipeline/eagle3/eagle3_new_model_triage_guide.md +++ b/examples/speculative_decoding/pipeline/eagle3/eagle3_new_model_triage_guide.md @@ -114,10 +114,10 @@ When adding a new patch: ## Step 6: Document results -Update `examples/speculative_decoding/pipeline/eagle3/eagle3_triage_chart.md`: -1. Update the model row in the **Model Test Matrix** (status + per-task results) -2. Add a **Per-Model Test Results** entry with experiment IDs, errors, and fixes -3. Add new failure patterns to the **Observed Failure Catalog** +Record the outcome of each run (status per task, experiment IDs, errors, and the fix +applied) in the team's internal triage tracker, and capture any new failure pattern so the +next model benefits. Once a model is verified end-to-end, add its launcher YAML example in a +dedicated PR. ## Known Model-Specific Issues diff --git a/examples/speculative_decoding/pipeline/eagle3/eagle3_triage_chart.md b/examples/speculative_decoding/pipeline/eagle3/eagle3_triage_chart.md deleted file mode 100644 index 19060e5d8b6..00000000000 --- a/examples/speculative_decoding/pipeline/eagle3/eagle3_triage_chart.md +++ /dev/null @@ -1,207 +0,0 @@ -# EAGLE3 Automation Triage Chart - -This document catalogs failure modes observed when running the EAGLE3 pipeline across -different model architectures. Updated as new models are tested. - -## Model Test Matrix - -| # | Model | Type | Params | Status | task_0 | task_1 | task_2 | task_3 | Notes | -|---|-------|------|--------|--------|--------|--------|--------|--------|-------| -| 1 | Qwen3-8B | Dense | 8B | Existing | - | - | - | - | Reference model | -| 2 | Kimi-K2.5 | MoE | 1T/32B | Existing | - | - | - | - | GB200 required | -| 3 | Qwen3.5-9B | Dense (VLM) | 9B | Not run | - | - | - | - | Text-only path | -| 4 | Qwen3.5-27B | Dense (VLM) | 27B | Not run | - | - | - | - | Text-only path | -| 5 | Qwen3.5-35B-A3B | MoE (VLM) | 35B/3B | **Blocked** | TIMEOUT | FAIL | FAIL | FAIL | Data synth too slow; infra issues | -| 6 | MiniMax-M2.5 | MoE | 230B/10B | **Blocked** | TIMEOUT | FAIL | FAIL | FAIL | trust_remote_code needed | -| 7 | Ministral-3-8B | Dense (VLM) | 8B | **WIP** | SKIP | PASS | PASS | FAIL | `use_cache=null` in export; see below | -| 8 | Ministral-3-14B | Dense (VLM) | 14B | **Blocked** | FAIL | FAIL | FAIL | FAIL | vLLM engine init fails (NoneType) | -| 9 | DeepSeek-V3.2 | MoE (MLA) | 685B/37B | **Blocked** | no log | FAIL | FAIL | FAIL | No task_0 log; infra issues | -| 10 | gpt-oss-20b | Dense | 20B | **Blocked** | FAIL | FAIL | FAIL | FAIL | Tokenizer `HarmonyError` | -| 11 | Step-3.5-Flash | MoE (SWA) | 197B/11B | **Blocked** | TIMEOUT | FAIL | FAIL | FAIL | Data synth hit time limit | -| 12 | GLM-5 | MoE (DSA) | 744B/40B | Not run | - | - | - | - | 2 nodes, gated | - -Legend: PASS / FAIL-{code} / SKIP / Pending - -## Triage Decision Tree - -```mermaid -flowchart TD - START([EAGLE3 Pipeline Failed]) --> WHICH_STEP{Which step failed?} - - WHICH_STEP -->|task_0: Data Synthesis| T0_CHECK{Server started?} - WHICH_STEP -->|task_1: Hidden States| T1_CHECK{TRT-LLM launched?} - WHICH_STEP -->|task_2: Training| T2_CHECK{Dependencies installed?} - WHICH_STEP -->|task_3: Benchmark| T3_CHECK{Engine started?} - - %% task_0 branch - T0_CHECK -->|No - hangs at health check| T0_OOM{CUDA OOM in log?} - T0_CHECK -->|Yes - server up but query fails| T0_QUERY[Check query.py errors:
bad prompt format,
connection timeout,
empty response] - T0_OOM -->|Yes| T0_FIX_OOM[Reduce max_num_tokens
or increase TP] - T0_OOM -->|No| T0_ARCH{Architecture error?} - T0_ARCH -->|Yes - RuntimeError / KeyError| T0_FIX_ARCH[Model not supported by
this TRT-LLM version.
Try newer container.] - T0_ARCH -->|No - trust_remote_code| T0_FIX_TRC[Add --trust_remote_code
before -- separator] - - %% task_1 branch - T1_CHECK -->|No - launch fails| T1_TRTLLM{Error type?} - T1_CHECK -->|Yes - but no .pt output| T1_DATA[Check --input-data path
and data format] - T1_TRTLLM -->|OOM| T1_FIX_OOM[Increase TP or nodes] - T1_TRTLLM -->|NCCL error| T1_FIX_NCCL[Network issue.
Retry or reduce EP.] - T1_TRTLLM -->|Unsupported model| T1_FIX_MODEL[Model arch not in TRT-LLM.
Check release notes.] - - %% task_2 branch - T2_CHECK -->|No - pip install fails| T2_FIX_DEPS[Network issue in container.
Check proxy/mirror.] - T2_CHECK -->|Yes| T2_TRAIN{Training starts?} - T2_TRAIN -->|No - ImportError| T2_FIX_IMPORT[modelopt not installed
or wrong version] - T2_TRAIN -->|No - FileNotFoundError| T2_FIX_DATA[task_1 output missing.
Re-run task_1.] - T2_TRAIN -->|Yes but crashes| T2_CRASH{Error type?} - T2_CRASH -->|OOM| T2_FIX_OOM[Reduce train_bs
or training_seq_len] - T2_CRASH -->|NaN loss| T2_FIX_NAN[Reduce lr.
Check data quality.] - T2_CRASH -->|KeyError / arch error| T2_FIX_EAGLE[Model type not recognized
by EAGLE3 training code.
Needs code change in modelopt.] - T2_TRAIN -->|Yes - export fails| T2_FIX_EXPORT[Check /scratchspace/eagle3
has model.safetensors] - - %% task_3 branch - T3_CHECK -->|No - engine crash| T3_ENGINE{Engine type?} - T3_CHECK -->|Yes - but AR below threshold| T3_AR[AR too low: need more
epochs, data, or
larger draft head] - T3_CHECK -->|Yes - but wrong output| T3_FORMAT[Check draft model
config.json compatibility
with engine version] - T3_ENGINE -->|vLLM - ImportError eagle| T3_FIX_VLLM[vLLM version too old.
Use latest container.] - T3_ENGINE -->|TRT-LLM - build error| T3_FIX_TRTLLM[Draft model format
incompatible. Check export.] - T3_ENGINE -->|OOM| T3_FIX_OOM[Target + draft too large.
Increase TP.] -``` - -## Observed Failure Catalog - -This section is updated as models are tested. Each entry records the model, step, -error, root cause, and resolution. - -### Architecture-Level Failures - -| Category | Affected Models | Step | Error | Root Cause | Resolution | -|----------|----------------|------|-------|------------|------------| -| VLM text-only | Qwen3.5-*, Ministral-3-* | task_0 | TBD | VLM models may load vision encoder unnecessarily | TBD — may need --language-model-only | -| VLM detection miss | Ministral-3-* | task_2 | `ValueError: Unrecognized config for AutoModelForCausalLM` | `load_vlm_or_llm` only checks `"vl"` in model_type; `"mistral3"` missed | Check `text_config`/`llm_config` attrs — fixed in repo | -| Missing HF shard | Ministral-3-8B | task_2 | `FileNotFoundError: model-00001-of-00004.safetensors` | Incomplete HF shards + Mistral native `consolidated.safetensors` | Fallback to consolidated with key aliases — fixed in repo | -| Exported config validation | All (via FakeBaseModel) | task_3 | `StrictDataclassFieldValidationError: use_cache` | Template placeholder `None` not filled; strict `huggingface_hub` rejects | Set `use_cache: True` in template — fixed in repo | -| MLA attention | DeepSeek-V3.2 | task_2 | TBD | EAGLE3 decoder type may not support MLA | TBD — verify eagle_decoder_type | -| Custom model code | MiniMax-M2.5 | task_0 | TBD | Non-standard architecture needs trust_remote_code | Add --trust_remote_code | -| Sliding window attn | Step-3.5-Flash | task_1 | TBD | SWA may not be supported in TRT-LLM hidden state extraction | TBD | -| Large MoE (>1 node) | DeepSeek-V3.2, GLM-5 | task_0/1 | TBD | Multi-node EP coordination | TBD — verify NCCL config | -| Gated models | DeepSeek-V3.2, GLM-5 | task_0 | FileNotFoundError | Model not mirrored to /hf-local | Request HF local mirror | - -### Per-Model Test Results - -#### Model: Ministral-3-8B-Instruct-2512-BF16 - -- **Date tested:** 2026-05-26 -- **Config:** `tools/launcher/examples/Mistral/Ministral-3-8B/eagle3_quick_check.yaml` -- **Experiments:** `cicd_1779312692` (dump), `cicd_1779829129` (train+bench), `cicd_1779901409` (retry w/ fixes) -- **task_0 (data synth):** SKIP — used vLLM dump path instead (`dump_offline_data_vllm.sh`) -- **task_1 (hidden states):** PASS — 330/330 conversations via vLLM dump (`cicd_1779312692`) -- **task_2 (training + export):** PASS — required 2 runtime patches (see issues below). `train_loss=31.93`, epoch 1, 278s total. Export to `/scratchspace/export` succeeded. -- **task_3 (benchmark):** FAIL — `StrictDataclassFieldValidationError: use_cache expected bool, got None` -- **AR:** Not measured (benchmark didn't complete) -- **New failure patterns?** Yes — 3 issues: - - 1. **VLM detection miss** — `model_type="mistral3"` is a VLM (`Mistral3ForConditionalGeneration`) but `load_vlm_or_llm` only checks `"vl" in model_type`. Fix: also check `text_config`/`llm_config` attrs. Applied in `modelopt/torch/speculative/utils.py` + runtime patch. - - 2. **Missing HF shard** — Checkpoint has shards 2-4 + `consolidated.safetensors` but shard 1 is absent. `FakeBaseModel._load_weights` fails. Fix: fallback to `consolidated.safetensors` with Mistral native key aliases (`tok_embeddings.weight`, `output.weight`). Applied in `modelopt/torch/speculative/plugins/modeling_fakebase.py` + runtime patch. - - 3. **`use_cache=null` in exported config** — Export template placeholder stays `None` when `FakeBaseConfig` doesn't define `use_cache`. Newer `huggingface_hub` strict validation rejects it. Fix: set `"use_cache": True` in export template (draft models always use cache). Applied in `modelopt/torch/export/plugins/hf_spec_configs.py` + post-export fixup in pipeline. - - 4. **(Potential) vLLM Pixtral resolution** — vLLM resolves base model as `PixtralForConditionalGeneration` (VLM). May cause further issues loading the EAGLE3 draft. Needs investigation. - -- **Repo fixes (branch `yeyu/speculative-lora-cotrain`):** - - `modelopt/torch/speculative/utils.py` — VLM detection via `text_config`/`llm_config` - - `modelopt/torch/speculative/plugins/modeling_fakebase.py` — consolidated.safetensors fallback - - `modelopt/torch/export/plugins/hf_spec_configs.py` — `use_cache: True` in templates -- **Pipeline fixes (`common/eagle3/train_eagle.sh`):** runtime patches matching above (applied only if the container ships an older modelopt) - ---- - -#### Model: gpt-oss-20b - -- **Date tested:** 2026-04-15 -- **Config:** `tools/launcher/examples/OpenAI/GPT-OSS-20B/eagle3_quick_check.yaml` -- **Experiment:** `cicd_1776272530` -- **task_0 (data synth):** FAIL — `openai_harmony.HarmonyError: error downloading or loading vocab file`. vLLM server starts loading model but tokenizer fails. Likely a gated/proprietary tokenizer issue. -- **task_1 (hidden states):** FAIL — `dump_offline_data_vllm.sh: No such file or directory` (script didn't exist at time of run) -- **task_2 (training):** FAIL — `service_utils.sh: No such file or directory` (infra issue at time of run) -- **task_3 (benchmark):** FAIL — `Error retrieving file list: Repo id must be in the form 'repo_name'` — no exported model -- **Blocker:** Tokenizer loading. Needs special tokenizer setup or newer vLLM with OpenAI model support. - -#### Model: Qwen3.5-35B-A3B - -- **Date tested:** 2026-04-15 -- **Config:** `tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/eagle3_quick_check.yaml` -- **Experiment:** `cicd_1776272531` -- **task_0 (data synth):** TIMEOUT — Server started successfully, data synthesis was running (5%/3295 at 38min), cancelled at time limit. `TCPTransport closed` errors during generation. -- **task_1:** FAIL — script not found (infra issue) -- **task_2:** FAIL — infra issue -- **task_3:** FAIL — no exported model -- **Blocker:** Data synthesis too slow. Needs longer wall time or reduced dataset size. Server itself works. - -#### Model: Step-3.5-Flash - -- **Date tested:** 2026-04-15 -- **Config:** `tools/launcher/examples/StepFun/Step-3.5-Flash/eagle3_quick_check.yaml` -- **Experiment:** `cicd_1776272532` -- **task_0 (data synth):** TIMEOUT — `CANCELLED AT 2026-04-15 DUE TO TIME LIMIT` -- **task_1:** FAIL — script not found (infra issue) -- **task_2:** FAIL — infra issue -- **task_3:** FAIL — no exported model -- **Blocker:** Data synthesis hit time limit. Needs investigation of whether server started successfully. - -#### Model: MiniMax-M2.5 - -- **Date tested:** 2026-04-15 -- **Config:** `tools/launcher/examples/MiniMax/MiniMax-M2.5/eagle3_quick_check.yaml` -- **Experiment:** `cicd_1776272524` -- **task_0 (data synth):** TIMEOUT — `CANCELLED DUE TO TIME LIMIT` -- **task_1:** FAIL — script not found (infra issue) -- **task_2:** FAIL — infra issue -- **task_3:** FAIL — `trust_remote_code=True` required for custom model code -- **Blocker:** Time limit on data synth + `trust_remote_code` needed for benchmark. - -#### Model: Ministral-3-14B - -- **Date tested:** 2026-04-15 -- **Config:** `tools/launcher/examples/Mistral/Ministral-3-14B/eagle3_quick_check.yaml` -- **Experiment:** `cicd_1776272522` -- **task_0 (data synth):** FAIL — `TypeError: 'NoneType' object is not iterable` on all TP workers during engine core init. vLLM cannot load this model architecture. -- **task_1:** FAIL — script not found (infra issue) -- **task_2:** FAIL — infra issue -- **task_3:** FAIL — `KeyError: 'ministral3'` — transformers in vLLM container doesn't recognize `ministral3` model type -- **Blocker:** vLLM engine fails to initialize. Same `mistral3` model type issue as 8B variant. Needs newer vLLM + transformers. - -#### Model: DeepSeek-V3.2 - -- **Date tested:** 2026-04-15 -- **Config:** `tools/launcher/examples/DeepSeek/DeepSeek-V3.2/eagle3_quick_check.yaml` -- **Experiment:** `cicd_1776275945` -- **task_0 (data synth):** No log file — job may not have started (gated model?) -- **task_1 (hidden states):** FAIL — script not found (infra issue) -- **task_2:** FAIL — infra issue -- **task_3:** FAIL — `Error retrieving file list` — no exported model -- **Blocker:** Model may not be mirrored to `/hf-local`. Needs 2 nodes for MLA architecture. - ---- - -*Use the following template for additional models:* - -```markdown -#### Model: -- **Date tested:** YYYY-MM-DD -- **Config:** tools/launcher/examples///eagle3_quick_check.yaml -- **task_0:** PASS/FAIL — -- **task_1:** PASS/FAIL — -- **task_2:** PASS/FAIL — -- **task_3:** PASS/FAIL — -- **AR:** (threshold: >= 2.1) -- **New failure pattern?** Yes/No — -``` - -## Revision History - -| Date | Author | Change | -|------|--------|--------| -| 2026-04-02 | Ye Yu | Initial chart with 12 models, triage decision tree | -| 2026-05-27 | Claude Code | Updated with results from initial batch (`cicd_1776272*`) and Ministral-3-8B deep dive. Added per-model test results for 7 models. Added 4 new failure catalog entries. | diff --git a/tools/launcher/examples/DeepSeek/DeepSeek-V3.2/eagle3_quick_check.yaml b/tools/launcher/examples/DeepSeek/DeepSeek-V3.2/eagle3_quick_check.yaml deleted file mode 100644 index 57fb3d09393..00000000000 --- a/tools/launcher/examples/DeepSeek/DeepSeek-V3.2/eagle3_quick_check.yaml +++ /dev/null @@ -1,111 +0,0 @@ -# EAGLE3 quick-fail pipeline for deepseek-ai/DeepSeek-V3.2. -# -# DeepSeek-V3.2 is a 685B MoE (37B active, MLA attention). -# BF16 weights: ~1370 GB — requires 2 GB200 nodes (8 × 192 GB = 1536 GB). -# -# Special requirements: -# - Gated model — must be downloaded/mirrored before use -# - MLA attention — verify eagle_decoder_type compatibility -# - trust_remote_code required -# -# MoE-specific notes: -# - TP=4 per node, EP across nodes if needed -# - Draft head intermediate_size may need tuning in eagle_config.json -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/DeepSeek/DeepSeek-V3.2/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/DeepSeek/DeepSeek-V3.2/eagle3_quick_check.yaml --yes - -job_name: DeepSeek-V3.2_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/deepseek-ai/DeepSeek-V3.2 - - # Step 1: Data synthesis via vLLM server (2 nodes, TP=8) - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 8 - - --trust-remote-code - - --distributed-executor-backend ray - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 2 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states (2 nodes) - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 2 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=4 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (2 nodes, TP=4, EP=2) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 2 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - - TP: 4 - - EP: 2 - slurm_config: - _factory_: "slurm_factory" - nodes: 2 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/DeepSeek/DeepSeek-V3.2/hf_offline_eagle3.yaml b/tools/launcher/examples/DeepSeek/DeepSeek-V3.2/hf_offline_eagle3.yaml deleted file mode 100644 index ee1059da913..00000000000 --- a/tools/launcher/examples/DeepSeek/DeepSeek-V3.2/hf_offline_eagle3.yaml +++ /dev/null @@ -1,112 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for deepseek-ai/DeepSeek-V3.2. -# -# DeepSeek-V3.2 is a 685B MoE (37B active, MLA attention). -# BF16 weights ~1370 GB — requires 2 GB200 nodes (8 × 192 GB = 1536 GB). -# -# Special requirements: -# - Gated model — must be downloaded/mirrored before use -# - MLA attention — verify eagle_decoder_type compatibility in eagle_config.json -# - trust_remote_code required -# -# MoE-specific notes: -# - TP=4 per node, EP=2 across 2 nodes for benchmark -# - task_2: consider increasing intermediate_size in eagle_config.json -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/DeepSeek/DeepSeek-V3.2/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/DeepSeek/DeepSeek-V3.2/hf_offline_eagle3.yaml --yes - -job_name: DeepSeek-V3.2_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/deepseek-ai/DeepSeek-V3.2 - - # Step 1: Data synthesis via vLLM server (2 nodes, TP=8) - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 8 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 2 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states (2 nodes, TP=8) - task_1: - script: common/eagle3/dump_offline_data.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - - --tp 8 - - --moe-ep 1 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 2 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (1 node) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (2 nodes, TP=4, EP=2) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 2 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 2 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/EAGLE3_TRIAGE.md b/tools/launcher/examples/EAGLE3_TRIAGE.md deleted file mode 100644 index 4f94011a17a..00000000000 --- a/tools/launcher/examples/EAGLE3_TRIAGE.md +++ /dev/null @@ -1,272 +0,0 @@ -# EAGLE3 Automation Triage Chart - -This document tracks failure modes discovered when running the 4-step EAGLE3 offline -pipeline against 10 new models. Updated as models are tested. -Claude can update the status table, diagram, and issue catalog when new results arrive. - ---- - -## Pipeline Overview - -```text -Model checkpoint (HuggingFace) - │ - ▼ -┌──────────────────┐ -│ Task 0: Query │ vLLM server generates prompt/response pairs -│ (data synthesis)│ Script: common/vllm/query.sh -└────────┬─────────┘ - │ (afterany — downstream tasks run even if this times out) - ▼ -┌──────────────────┐ -│ Task 1: Dump │ Target model runs forward pass, saves hidden states -│ (hidden states) │ Script: common/eagle3/dump_offline_data.sh (TRT-LLM) -└────────┬─────────┘ or dump_offline_data_hf.sh (HF device_map=auto) - or dump_offline_data_vllm.sh (vLLM native extractor) - │ - ▼ -┌──────────────────┐ -│ Task 2: Train │ Draft head trained on hidden states (Accelerate + FSDP) -│ (EAGLE3 head) │ Script: common/eagle3/train_eagle.sh -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ -│ Task 3: Bench │ Speculative decoding benchmark via vLLM -│ (benchmark) │ Script: common/specdec_bench/quick_check.sh -└──────────────────┘ -``` - ---- - -## Triage Decision Tree - -```mermaid -flowchart TD - START([EAGLE3 Pipeline Failed]) --> WHICH_STEP{Which step failed?} - - WHICH_STEP -->|task_0: Data synthesis| T0_CHECK{Server started?} - WHICH_STEP -->|task_1: Hidden states| T1_CHECK{Script found?} - WHICH_STEP -->|task_2: Training| T2_CHECK{Dependencies installed?} - WHICH_STEP -->|task_3: Benchmark| T3_CHECK{Engine started?} - - %% ── task_0 ────────────────────────────────────────────────── - T0_CHECK -->|No - hangs at health check| T0_OOM{CUDA OOM in log?} - T0_CHECK -->|Yes - server up, query fails| T0_QUERY[Check query.py errors:\nbad prompt format,\nconnection timeout,\nempty response] - T0_OOM -->|Yes| T0_FIX_OOM[⚠ OOM\nReduce max_num_tokens\nor increase TP] - T0_OOM -->|No| T0_ARCH{Error type?} - T0_ARCH -->|vocab / tokenizer error| T0_TOKENIZER[⚠ TOKENIZER\nMissing tokenizer cache.\ne.g. GPT-OSS-20B needs\nTIKTOKEN_RS_CACHE_DIR pre-populated] - T0_ARCH -->|Architecture / RuntimeError| T0_FIX_ARCH[⚠ VLLM_SUPPORT\nModel arch not supported\nin this vLLM version.\nTry newer container.] - T0_ARCH -->|trust_remote_code| T0_FIX_TRC[⚠ TRUST_REMOTE_CODE\nAdd --trust-remote-code\nbefore -- separator in args] - T0_CHECK -->|Cancelled - time limit| T0_TIMEOUT[⚠ TIMEOUT\nJob wall-clock limit too short.\nNote: afterany deps ensure\ntask_1 still runs.\nFix: increase time limit\nor reduce dataset size.] - - %% ── task_1 ────────────────────────────────────────────────── - T1_CHECK -->|No - script not found| T1_SCRIPT[⚠ MISSING_SCRIPT\nVerify script path. Three backends:\n• dump_offline_data_vllm.sh (vLLM native extractor)\n• dump_offline_data_hf.sh (HF device_map=auto)\n• dump_offline_data.sh (TRT-LLM, --tp/--moe-ep)] - T1_CHECK -->|Yes| T1_RUN{Runs OK?} - T1_RUN -->|No - OOM| T1_OOM[⚠ OOM\nIncrease TP, add EP,\nor switch to _hf script.] - T1_RUN -->|No - NCCL error| T1_NCCL[⚠ NCCL\nNetwork/multi-node issue.\nRetry or reduce EP.] - T1_RUN -->|No - arch unsupported| T1_ARCH[⚠ ARCH\nModel not supported by TRT-LLM.\nSwitch to dump_offline_data_hf.sh.] - T1_RUN -->|Yes - no .pt output| T1_DATA[Check --input-data path\nand data format from task_0] - - %% ── task_2 ────────────────────────────────────────────────── - T2_CHECK -->|No - pip install fails| T2_FIX_DEPS[Network issue in container.\nCheck proxy/mirror.] - T2_CHECK -->|Yes| T2_TRAIN{Training starts?} - T2_TRAIN -->|No - ImportError| T2_FIX_IMPORT[modelopt not installed\nor wrong version] - T2_TRAIN -->|No - FileNotFoundError| T2_FIX_DATA[task_1 output missing.\nRe-run task_1.] - T2_TRAIN -->|Yes but crashes| T2_CRASH{Error type?} - T2_CRASH -->|OOM| T2_FIX_OOM[⚠ OOM\nReduce train_bs\nor training_seq_len] - T2_CRASH -->|NaN loss| T2_FIX_NAN[Reduce lr.\nCheck data quality.] - T2_CRASH -->|KeyError / arch| T2_FIX_EAGLE[⚠ ARCH\nModel type not recognized\nby EAGLE3 training code.\nNeeds code change in modelopt.\nCheck eagle_decoder_type in config.] - T2_TRAIN -->|Yes - export fails| T2_FIX_EXPORT[Check /scratchspace/eagle3\nhas model.safetensors] - - %% ── task_3 ────────────────────────────────────────────────── - T3_CHECK -->|No - export dir missing| T3_EXPORT[⚠ CASCADE\nTask 2 failed or timed out.\nResolve task_2 first.] - T3_CHECK -->|No - engine crash| T3_ENGINE{Engine type?} - T3_CHECK -->|Yes - AR below threshold| T3_AR[AR too low:\nneed more epochs, data,\nor larger draft head] - T3_CHECK -->|Yes - wrong output| T3_FORMAT[Check draft model\nconfig.json vs engine version] - T3_ENGINE -->|vLLM - trust_remote_code| T3_TRUST[⚠ TRUST_REMOTE_CODE\nAdd --trust-remote-code\nto quick_check.sh invocation] - T3_ENGINE -->|vLLM - spec decode unsupported| T3_VLLM[⚠ VLLM_SPECDEC\nvLLM version too old.\nUse latest container.] - T3_ENGINE -->|NVFP4 - unsupported| T3_NVFP4[⚠ NVFP4\nRequires vllm-openai:v0.15.0+\nand Blackwell GPU.] - T3_ENGINE -->|OOM| T3_FIX_OOM[Target + draft too large.\nIncrease TP.] -``` - ---- - -## Model Test Matrix - -Tests run on OCI-HSG cluster (GB200 nodes, 4 × 192 GB HBM3e per node). - -| # | Model | Type | Size | task_0 | task_1 | task_2 | task_3 | Notes | -|---|-------|------|------|--------|--------|--------|--------|-------| -| 1 | Ministral-3-8B | Dense | 8B | 🔁 RERUNNING (--num-shards 3) | 🔁 RERUNNING (vLLM native extractor) | 🔲 | 🔲 | Issues 6+7 fixed; re-run in progress | -| 2 | Ministral-3-14B | Dense | 14B | ⏱ TIMEOUT | 🔁 NEEDS RERUN (_vllm) | 🔲 | 🔲 | — | -| 3 | GPT-OSS-20B | Dense | 20B | ❌ TOKENIZER | 🔁 NEEDS RERUN (_vllm) | 🔲 | 🔲 | Fix: populate TIKTOKEN_RS_CACHE_DIR first | -| 4 | MiniMax-M2.5 | MoE | 230B/10B | ⏱ TIMEOUT | 🔁 NEEDS RERUN (_vllm) | 🔲 | ❌ TRUST_REMOTE_CODE | trust_remote_code needed at bench | -| 5 | Qwen3.5-35B-A3B | MoE | 35B/3B | ⏱ TIMEOUT | 🔁 NEEDS RERUN (_vllm) | 🔲 | 🔲 | — | -| 6 | Step-3.5-Flash | MoE/SWA | 197B/11B | ⏱ TIMEOUT | 🔁 NEEDS RERUN (_vllm) | 🔲 | 🔲 | SWA: use _vllm or_hf script | -| 7 | DeepSeek-V3.2 | MoE/MLA | 685B/37B | 🔍 (tarball only) | 🔁 NEEDS RERUN (_vllm, 2-node) | 🔲 | 🔲 | 2-node; previous t1 OOM-killed | -| 8 | Kimi-K2.5 | MoE/MLA | 1T/32B | 🔲 | 🔲 | 🔲 | 🔲 | MLA attention: verify eagle_decoder_type | -| 9 | GLM-5 | MoE/DSA | 744B/40B | 🔲 | 🔲 | 🔲 | 🔲 | Gated, 2-node | -| 10 | Kimi-K2.5-NVFP4 | NVFP4 | ~591GB | 🔲 | 🔲 | 🔲 | 🔲 | Blackwell required; t1/t2 use BF16 base | - -**Legend:** ✅ Pass · ❌ Fail · ⏱ Timeout · 🔍 Inconclusive · 🔲 Not yet tested · 🔁 Rerun needed - ---- - -## Known Issues - -### Issue 1: Missing `dump_offline_data_vllm.sh` (Task 1 — universal) — FIXED ✅ - -**Symptom:** `/usr/bin/bash: .../dump_offline_data_vllm.sh: No such file or directory` - -**Affected:** All 7 models tested (root cause of universal task_1 failure in first round). - -**Root cause:** Quick-fail pipeline configs referenced `dump_offline_data_vllm.sh`, which had -not yet been created. Only two scripts existed: `dump_offline_data.sh` (TRT-LLM) and -`dump_offline_data_hf.sh` (HF `device_map="auto"`). - -**Fix applied:** `dump_offline_data_vllm.sh` and its backing script -`compute_hidden_states_vllm.py` were added. The vLLM script drives vLLM's built-in -`extract_hidden_states` speculative method (via the `ExampleHiddenStatesConnector` KV -connector) and saves output in the same `.pt` format as the HF variant. No third-party -data-generation dependency is required. Both files are now in: -- `tools/launcher/common/eagle3/dump_offline_data_vllm.sh` -- `examples/speculative_decoding/collect_hidden_states/compute_hidden_states_vllm.py` - -Three backends now available for task_1: - -| Backend | Script | When to use | -|---------|--------|-------------| -| TRT-LLM | `dump_offline_data.sh` | Pure-text models with TRT-LLM support; needs `--tp`/`--moe-ep` | -| HF | `dump_offline_data_hf.sh` | VLMs, custom-code models, SWA; `device_map="auto"` | -| vLLM | `dump_offline_data_vllm.sh` | Broad coverage via vLLM model implementations; uses vLLM's native extractor | - ---- - -### Issue 2: Training-step HuggingFace Hub upload bug — FIXED ✅ - -**Was:** `HFValidationError: Repo id must be in the form 'repo_name': '/scratchspace/eagle3'` - -**Fix applied:** The training step (`common/eagle3/train_eagle.sh`) trains and then exports the -HF checkpoint to a local path only — no HF Hub upload — and sources `error_handler` from -`service_utils.sh`. - ---- - -### Issue 3: Task 0 time limit (most models) — PARTIALLY ADDRESSED ⚠ - -**Symptom:** `STEP CANCELLED AT ... DUE TO TIME LIMIT` - -**Affected:** Ministral-3-8B (3277/3295 samples — nearly complete), Ministral-3-14B, -MiniMax-M2.5, Qwen3.5-35B-A3B, Step-3.5-Flash. - -**Status:** `afterany` Slurm dependencies were added so downstream tasks (task_1, 2, 3) -run even when task_0 times out. The data synthesis timeout itself is not yet resolved. - -**Fix options:** -- Increase Slurm `--time` limit for task_0. -- Add `--max-samples N` to limit dataset size for quick-fail validation. - ---- - -### Issue 4: GPT-OSS-20B tokenizer cache missing (Task 0) — OPEN - -**Symptom:** `openai_harmony.HarmonyError: error downloading or loading vocab file` - -**Affected:** GPT-OSS-20B only. vLLM started (model loaded) but vocab download failed. - -**Root cause:** GPT-OSS-20B uses the `openai_harmony` tokenizer backed by tiktoken, which -requires `TIKTOKEN_RS_CACHE_DIR` to point to a pre-populated local cache. The cluster did -not have this directory populated. - -**Fix:** Ensure `TIKTOKEN_RS_CACHE_DIR` is set to a valid pre-populated tiktoken cache -path before submitting task_0. - ---- - -### Issue 5: MiniMax-M2.5 missing `trust_remote_code` at benchmark (Task 3) — OPEN - -**Symptom:** - -```text -ValueError: The repository ... contains custom code... Please pass trust_remote_code=True -``` - -**Affected:** MiniMax-M2.5 task_3. - -**Root cause:** `quick_check.sh` does not forward `--trust-remote-code` to vLLM for models -that require it. - -**Fix:** Pass `--trust-remote-code` in the `quick_check.sh` vLLM invocation when -`trust_remote_code` is set in the pipeline environment. - ---- - -### Issue 6: `speculators` dependency removed — superseded by vLLM's native extractor ✅ - -**History:** The vLLM dump path originally used `VllmHiddenStatesGenerator` from the -`speculators` library. This was brittle: `speculators==0.5.0` removed that class -(`ImportError: cannot import name 'VllmHiddenStatesGenerator'`), forcing a -`pip install "speculators<0.5.0"` pin plus several runtime source-patches of the -installed library for vLLM/pydantic compatibility. - -**Resolution:** `dump_offline_data_vllm.sh` / `compute_hidden_states_vllm.py` now use -vLLM's built-in `extract_hidden_states` speculative method (via the -`ExampleHiddenStatesConnector` KV connector). The `speculators` dependency and all -runtime source-patches were removed, so the version pin and these compatibility -patches no longer apply. - ---- - -### Issue 7: `query.py` auto-downgrades shards → empty data on timeout (Task 0) — FIXED ✅ - -**Symptom:** task_0 times out; `/scratchspace/data/` is empty despite partial generation. - -**Affected:** Models with datasets ≤ 33,000 samples where `num_shards * 100 > dataset_size`. - -**Root cause:** `query.py` auto-downgrades `--num-shards` to `min(16, dataset_size//100)` -when the default of 1000 is too large relative to dataset size. For 3295 samples this -becomes 1 shard, meaning all data is processed in one batch and nothing is saved until -the entire map completes. A timeout yields zero data. - -**Fix applied:** Pass `--num-shards 3` explicitly in task_0 args. Since `3*100=300 < 3295`, -the auto-downgrade is bypassed. Data is saved incrementally across 3 shard files (~1100 -samples each). Partial data survives a timeout. - ---- - -### Issue 8: DeepSeek-V3.2 task_1 OOM (Task 1) — OPEN - -**Symptom:** `pyxis: child terminated with signal 15` (SIGTERM, likely OOM-triggered) - -**Affected:** DeepSeek-V3.2 only (685B MoE, 2-node job). - -**Root cause:** Task_1 was also blocked by Issue 1 (missing vllm script); the SIGTERM may -indicate OOM during the brief moment before the script-not-found failure propagated. Needs -further investigation with `dump_offline_data_hf.sh`. - ---- - -## How to Update This Document - -When a new model completes testing: - -1. **Status table**: Update the row — fill in ✅/❌/⏱/🔍 and brief notes. -2. **Decision tree**: If a new failure mode appears that has no matching leaf, add a new - branch under the appropriate step node. -3. **Issue catalog**: Add a new numbered section with symptom, affected models, root cause, - fix, and status (OPEN / FIXED / PARTIALLY ADDRESSED). -4. Mark resolved issues as **FIXED ✅** and update the status in the table. - -Per-model results template: - -```markdown -#### Model: -- **Date tested:** YYYY-MM-DD -- **task_0:** PASS/FAIL/TIMEOUT — -- **task_1:** PASS/FAIL — -- **task_2:** PASS/FAIL — -- **task_3:** PASS/FAIL — -- **AR speedup:** (target ≥ 2.1×) -- **New failure pattern:** Yes/No — -``` diff --git a/tools/launcher/examples/GLM/GLM-5/eagle3_quick_check.yaml b/tools/launcher/examples/GLM/GLM-5/eagle3_quick_check.yaml deleted file mode 100644 index 3b769117db8..00000000000 --- a/tools/launcher/examples/GLM/GLM-5/eagle3_quick_check.yaml +++ /dev/null @@ -1,112 +0,0 @@ -# EAGLE3 quick-fail pipeline for zai-org/GLM-5. -# -# GLM-5 is a 744B MoE (40B active, 256 experts, top-8). DeepSeek Sparse Attention. -# BF16 weights: ~1488 GB — requires 2 GB200 nodes (8 × 192 GB = 1536 GB). -# -# Special requirements: -# - Gated model — must be downloaded/mirrored before use -# - trust_remote_code required -# - 200K context window supported but use 8192 for quick-fail -# - DeepSeek Sparse Attention — verify TRT-LLM support -# -# MoE-specific notes: -# - TP=4 per node, EP=2 across 2 nodes -# - Draft head intermediate_size may need tuning in eagle_config.json -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/GLM/GLM-5/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/GLM/GLM-5/eagle3_quick_check.yaml --yes - -job_name: GLM-5_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/zai-org/GLM-5 - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=4 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (TP=4, EP=2) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 2 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - - TP: 4 - - EP: 2 - slurm_config: - _factory_: "slurm_factory" - nodes: 2 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/GLM/GLM-5/hf_offline_eagle3.yaml b/tools/launcher/examples/GLM/GLM-5/hf_offline_eagle3.yaml deleted file mode 100644 index 4a709c7d061..00000000000 --- a/tools/launcher/examples/GLM/GLM-5/hf_offline_eagle3.yaml +++ /dev/null @@ -1,113 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for zai-org/GLM-5. -# -# GLM-5 is a 744B MoE (40B active, 256 experts, top-8, DeepSeek Sparse Attention). -# BF16 weights ~1488 GB — requires 2 GB200 nodes for benchmark (8 × 192 GB = 1536 GB). -# -# Special requirements: -# - Gated model — must be downloaded/mirrored before use -# - trust_remote_code required -# - 200K context window supported; use 8192 for typical EAGLE3 training -# - DeepSeek Sparse Attention — verify vLLM support before running -# -# MoE-specific notes: -# - task_2: consider increasing intermediate_size in eagle_config.json -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/GLM/GLM-5/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/GLM/GLM-5/hf_offline_eagle3.yaml --yes - -job_name: GLM-5_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/zai-org/GLM-5 - - # Step 1: Data synthesis via vLLM server (1 node, TP=4) - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states (1 node, TP=4) - task_1: - script: common/eagle3/dump_offline_data.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - - --tp 4 - - --moe-ep 1 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (1 node) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (2 nodes, TP=4, EP=2) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 2 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 2 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/MiniMax/MiniMax-M2.5/eagle3_quick_check.yaml b/tools/launcher/examples/MiniMax/MiniMax-M2.5/eagle3_quick_check.yaml deleted file mode 100644 index 24ab04c7c06..00000000000 --- a/tools/launcher/examples/MiniMax/MiniMax-M2.5/eagle3_quick_check.yaml +++ /dev/null @@ -1,107 +0,0 @@ -# EAGLE3 quick-fail pipeline for MiniMaxAI/MiniMax-M2.5. -# -# MiniMax-M2.5 is a 230B MoE (10B active, 256 experts, top-8). -# BF16 weights: ~460 GB — fits on a single GB200 node (4 × 192 GB = 768 GB). -# -# Special requirements: -# - trust_remote_code is required (custom model code, model type: minimax_m2) -# -# MoE-specific notes: -# - task_2: check eagle_config.json and consider increasing intermediate_size -# since the draft head uses a dense layer by default. -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/MiniMax/MiniMax-M2.5/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/MiniMax/MiniMax-M2.5/eagle3_quick_check.yaml --yes - -job_name: MiniMax-M2.5_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/MiniMaxAI/MiniMax-M2.5 - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/MiniMax/MiniMax-M2.5/hf_offline_eagle3.yaml b/tools/launcher/examples/MiniMax/MiniMax-M2.5/hf_offline_eagle3.yaml deleted file mode 100644 index 399807829c3..00000000000 --- a/tools/launcher/examples/MiniMax/MiniMax-M2.5/hf_offline_eagle3.yaml +++ /dev/null @@ -1,111 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for MiniMaxAI/MiniMax-M2.5. -# -# MiniMax-M2.5 is a 230B MoE (10B active, 256 experts, top-8). -# BF16 weights ~460 GB — fits on a single GB200 node (4 × 192 GB = 768 GB). -# -# Special requirements: -# - trust_remote_code is required (custom model type: minimax_m2) -# -# MoE-specific notes: -# - task_2: consider increasing intermediate_size in eagle_config.json -# since the draft head uses a dense layer by default. -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/MiniMax/MiniMax-M2.5/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/MiniMax/MiniMax-M2.5/hf_offline_eagle3.yaml --yes - -job_name: MiniMax-M2.5_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/MiniMaxAI/MiniMax-M2.5 - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - # Uses HF backend (device_map=auto) — MiniMax-M2.5 requires trust_remote_code and - # uses a custom architecture not supported by TRT-LLM hidden state extraction. - task_1: - script: common/eagle3/dump_offline_data_hf.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Mistral/Ministral-3-14B/eagle3_quick_check.yaml b/tools/launcher/examples/Mistral/Ministral-3-14B/eagle3_quick_check.yaml deleted file mode 100644 index 9392d2281fb..00000000000 --- a/tools/launcher/examples/Mistral/Ministral-3-14B/eagle3_quick_check.yaml +++ /dev/null @@ -1,104 +0,0 @@ -# EAGLE3 quick-fail pipeline for mistralai/Ministral-3-14B-Instruct-2512-BF16. -# -# Ministral-3-14B is a 14B dense model (model type: mistral3, multimodal but -# EAGLE3 uses text-only path). -# BF16 weights: ~28 GB — fits easily on a single GB200 node. -# -# Special requirements: -# - trust_remote_code may be needed for tokenizer -# - Model type is mistral3 (Mistral3ForConditionalGeneration) -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Mistral/Ministral-3-14B/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Mistral/Ministral-3-14B/eagle3_quick_check.yaml --yes - -job_name: Ministral-3-14B_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/mistralai/Ministral-3-14B-Instruct-2512-BF16 - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Mistral/Ministral-3-14B/hf_offline_eagle3.yaml b/tools/launcher/examples/Mistral/Ministral-3-14B/hf_offline_eagle3.yaml deleted file mode 100644 index 1328657cc17..00000000000 --- a/tools/launcher/examples/Mistral/Ministral-3-14B/hf_offline_eagle3.yaml +++ /dev/null @@ -1,107 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for mistralai/Ministral-3-14B-Instruct-2512-BF16. -# -# Ministral-3-14B is a 14B dense model (model type: mistral3, multimodal but -# EAGLE3 uses the text-only path). -# BF16 weights ~28 GB — fits easily on a single GB200 node. -# -# Special requirements: -# - trust_remote_code may be needed for the tokenizer -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Mistral/Ministral-3-14B/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Mistral/Ministral-3-14B/hf_offline_eagle3.yaml --yes - -job_name: Ministral-3-14B_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/mistralai/Ministral-3-14B-Instruct-2512-BF16 - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - # Uses HF backend (device_map=auto) — mistral3/Mistral3ForConditionalGeneration is not - # supported by TRT-LLM hidden state extraction. - task_1: - script: common/eagle3/dump_offline_data_hf.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Mistral/Ministral-3-8B/eagle3_quick_check.yaml b/tools/launcher/examples/Mistral/Ministral-3-8B/eagle3_quick_check.yaml deleted file mode 100644 index 30253bcf0d7..00000000000 --- a/tools/launcher/examples/Mistral/Ministral-3-8B/eagle3_quick_check.yaml +++ /dev/null @@ -1,106 +0,0 @@ -# EAGLE3 quick-fail pipeline for mistralai/Ministral-3-8B-Instruct-2512-BF16. -# -# Ministral-3-8B is an 8B dense model (model type: mistral3, multimodal but -# EAGLE3 uses text-only path). -# BF16 weights: ~16 GB — fits easily on a single GB200 node. -# -# Special requirements: -# - trust_remote_code may be needed for tokenizer -# - Model type is mistral3 (Mistral3ForConditionalGeneration) -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Mistral/Ministral-3-8B/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Mistral/Ministral-3-8B/eagle3_quick_check.yaml --yes - -job_name: Ministral-3-8B_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/mistralai/Ministral-3-8B-Instruct-2512-BF16 - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - - --num-shards 10000 - - --shard-id-step 10001 - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Mistral/Ministral-3-8B/hf_offline_eagle3.yaml b/tools/launcher/examples/Mistral/Ministral-3-8B/hf_offline_eagle3.yaml deleted file mode 100644 index 0aabf9dfe3d..00000000000 --- a/tools/launcher/examples/Mistral/Ministral-3-8B/hf_offline_eagle3.yaml +++ /dev/null @@ -1,107 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for mistralai/Ministral-3-8B-Instruct-2512-BF16. -# -# Ministral-3-8B is an 8B dense model (model type: mistral3). -# BF16 weights ~16 GB — fits easily on a single GB200 node. -# -# Special requirements: -# - trust_remote_code may be needed for the tokenizer -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Mistral/Ministral-3-8B/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Mistral/Ministral-3-8B/hf_offline_eagle3.yaml --yes - -job_name: Ministral-3-8B_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/mistralai/Ministral-3-8B-Instruct-2512-BF16 - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - # Uses HF backend (device_map=auto) — mistral3/PixtralForConditionalGeneration is not - # supported by TRT-LLM hidden state extraction. - task_1: - script: common/eagle3/dump_offline_data_hf.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - model.trust_remote_code=true - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/MoonshotAI/Kimi-K2.5/eagle3_quick_check.yaml b/tools/launcher/examples/MoonshotAI/Kimi-K2.5/eagle3_quick_check.yaml deleted file mode 100644 index 63894e540bb..00000000000 --- a/tools/launcher/examples/MoonshotAI/Kimi-K2.5/eagle3_quick_check.yaml +++ /dev/null @@ -1,135 +0,0 @@ -# EAGLE3 quick-fail pipeline for moonshotai/Kimi-K2.5. -# -# Kimi-K2.5 is a 1T-parameter MoE VLM (32B active, 384 experts, 8 selected, MLA attention). -# EAGLE3 uses only the text path — the vision encoder is not invoked. -# BF16 weights ~595 GB — fits on a single GB200 node (4 × 192 GB = 768 GB). -# -# Special requirements: -# - trust_remote_code required for the Kimi tokenizer -# - MLA attention — verify eagle_decoder_type compatibility in eagle_config.json -# -# MoE-specific notes: -# - task_1: --moe-ep 1 since TP=4 spans all 4 GPUs; increase --moe-ep and -# reduce --tp if you have more GPUs and want expert parallelism. -# - task_2: review eagle_config.json and consider increasing intermediate_size -# for MoE models (the draft head uses a dense layer by default, which may be -# undersized relative to K2.5's expert hidden dim of 2048). -# - task_2: --eagle_decoder_type may need adjustment if K2.5's MLA attention -# is not covered by the default llama decoder type. -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/MoonshotAI/Kimi-K2.5/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/MoonshotAI/Kimi-K2.5/eagle3_quick_check.yaml --yes - -job_name: Kimi-K2.5_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/moonshotai/Kimi-K2.5 - - # Step 1: Data synthesis via TRT-LLM server - # Args before "--" go to trtllm-serve; args after "--" go to tools/query.py. - # OCI-HSG GB200: 4 GPUs × 192 GB = 768 GB per node — fits K2.5 BF16 (~595 GB) with TP=4. - # --trust_remote_code is required for the Kimi tokenizer. - task_0: - script: common/tensorrt_llm/query.sh - args: - - --model <> - - --tp_size 4 - - --ep_size 1 - - --max_num_tokens 32000 - - --port 8000 - - --host 0.0.0.0 - - --trust_remote_code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 4 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc2 - - # Step 2: Dump hidden states from target model (TP=4, moe-ep=1) - task_1: - script: common/eagle3/dump_offline_data.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - - --tp 4 - - --moe-ep 1 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 4 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc2 - - # Step 3: Train EAGLE3 draft head (offline, single task) - # For K2.5 (MoE): check eagle_config.json and consider increasing - # intermediate_size since K2.5's per-expert hidden dim is 2048. - # Also verify --eagle_decoder_type is compatible with MLA attention. - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc2 - - # Step 4: Benchmark speculative decoding (VLLM backend) - # - # For K2.5 with VLLM, additional flags are recommended but may require - # extending quick_check.sh or using run.sh with a custom config: - # --mm-encoder-tp-mode data (data-parallel vision encoder, improves throughput) - # --tool-call-parser kimi_k2 (required for tool-call responses) - # --reasoning-parser kimi_k2 (required for thinking-mode responses) - # - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - - TP: 4 - - EP: 1 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/MoonshotAI/Kimi-K2.5/hf_offline_eagle3.yaml b/tools/launcher/examples/MoonshotAI/Kimi-K2.5/hf_offline_eagle3.yaml deleted file mode 100644 index 33f3cb52790..00000000000 --- a/tools/launcher/examples/MoonshotAI/Kimi-K2.5/hf_offline_eagle3.yaml +++ /dev/null @@ -1,114 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for moonshotai/Kimi-K2.5. -# -# Kimi-K2.5 is a 1T MoE VLM (32B active, 384 experts, top-8, MLA attention). -# EAGLE3 uses only the text path — the vision encoder is not invoked. -# BF16 weights ~595 GB — fits on a single GB200 node (4 × 192 GB = 768 GB). -# -# Special requirements: -# - trust_remote_code required for the Kimi tokenizer -# - MLA attention — verify eagle_decoder_type compatibility in eagle_config.json -# -# MoE-specific notes: -# - task_1: --moe-ep 1 since TP=4 spans all 4 GPUs -# - task_2: consider increasing intermediate_size in eagle_config.json -# (K2.5's per-expert hidden dim is 2048; the draft head uses a dense layer) -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/MoonshotAI/Kimi-K2.5/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/MoonshotAI/Kimi-K2.5/hf_offline_eagle3.yaml --yes - -job_name: Kimi-K2.5_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/moonshotai/Kimi-K2.5 - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model (TP=4, moe-ep=1) - task_1: - script: common/eagle3/dump_offline_data.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - - --tp 4 - - --moe-ep 1 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/NVIDIA/Kimi-K2.5-NVFP4/eagle3_quick_check.yaml b/tools/launcher/examples/NVIDIA/Kimi-K2.5-NVFP4/eagle3_quick_check.yaml deleted file mode 100644 index 683794cf3ad..00000000000 --- a/tools/launcher/examples/NVIDIA/Kimi-K2.5-NVFP4/eagle3_quick_check.yaml +++ /dev/null @@ -1,149 +0,0 @@ -# EAGLE3 quick-fail pipeline for nvidia/Kimi-K2.5-NVFP4. -# -# nvidia/Kimi-K2.5-NVFP4 is the NVIDIA-quantized NVFP4 variant of moonshotai/Kimi-K2.5. -# Same architecture: 1T-parameter MoE VLM (32B active, 384 experts, 8 selected, MLA attention). -# Only the weights and activations of linear ops within MoE transformer blocks are quantized; -# embeddings and other tensors remain in higher precision. -# EAGLE3 uses only the text path — the vision encoder is not invoked. -# -# Hardware requirements: -# NVFP4 checkpoint is ~591 GB (vs ~595 GB BF16) — same single-node requirement. -# OCI-HSG uses GB200 (Blackwell) nodes with 4 GPUs × 192 GB HBM3e = 768 GB per node. -# NVFP4 inference requires Blackwell GPUs — OCI-HSG GB200 satisfies this. -# A single 4×GB200 node with TP=4 is sufficient. -# -# Why different models for task_0 vs tasks 1–2: -# task_0 (data synthesis) uses vLLM + NVFP4 directly, since vLLM v0.15.0 supports -# KimiK25ForConditionalGeneration with NVFP4 on Blackwell. -# tasks 1–2 (hidden-state extraction and training) use the BF16 moonshotai/Kimi-K2.5 -# checkpoint via TRT-LLM, which does not support KimiK25ForConditionalGeneration -# in v1.3.0. The draft head trained on BF16 hidden states transfers directly to -# NVFP4 — NVFP4 is near-lossless, so hidden state distributions are equivalent. -# -# MoE-specific notes: -# - task_1: --moe-ep 1 since TP=4 spans all 4 GPUs; increase --moe-ep and -# reduce --tp if you have more GPUs and want expert parallelism. -# - task_2: review eagle_config.json and consider increasing intermediate_size -# for MoE models (the draft head uses a dense layer by default, which may be -# undersized relative to K2.5's expert hidden dim of 2048). -# - task_2: --eagle_decoder_type may need adjustment if K2.5's MLA attention -# is not covered by the default llama decoder type. -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/NVIDIA/Kimi-K2.5-NVFP4/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/NVIDIA/Kimi-K2.5-NVFP4/eagle3_quick_check.yaml --yes - -job_name: Kimi-K2.5-NVFP4_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/nvidia/Kimi-K2.5-NVFP4 - hf_model_bf16: /hf-local/moonshotai/Kimi-K2.5 - - # Step 1: Data synthesis via vLLM server using NVFP4 checkpoint - # Uses NVFP4 directly via vLLM v0.15.0, which supports KimiK25ForConditionalGeneration. - # vLLM is single-process (ntasks_per_node: 1); --tensor-parallel-size handles GPU distribution. - # --trust-remote-code is required for the Kimi tokenizer. - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --max-num-seqs 32 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:v0.15.0 - - # Step 2: Dump hidden states using BF16 checkpoint (TP=4, moe-ep=1) - # Uses BF16 moonshotai/Kimi-K2.5: same TRT-LLM compatibility reason as task_0. - task_1: - script: common/eagle3/dump_offline_data.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - - --tp 4 - - --moe-ep 1 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 4 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc2 - - # Step 3: Train EAGLE3 draft head on BF16 hidden states - # HF_MODEL_CKPT (via model.model_name_or_path) points to BF16 for tokenizer/config - # loading; hidden states come from task_1 (also BF16). - # Draft head transfers to NVFP4 at benchmark time. - # For K2.5 (MoE): check eagle_config.json and consider increasing - # intermediate_size since K2.5's per-expert hidden dim is 2048. - # Also verify --eagle_decoder_type is compatible with MLA attention. - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc2 - - # Step 4: Benchmark speculative decoding against NVFP4 target (VLLM backend) - # - # vllm/vllm-openai:v0.15.0 is required for NVFP4 support on Blackwell GPUs. - # Additional flags required for full K2.5 functionality but may require - # extending quick_check.sh or using run.sh with a custom config: - # --tool-call-parser kimi_k2 (required for tool-call responses) - # --reasoning-parser kimi_k2 (required for thinking-mode responses) - # - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - - TP: 4 - - EP: 1 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:v0.15.0 diff --git a/tools/launcher/examples/NVIDIA/Kimi-K2.5-NVFP4/hf_offline_eagle3.yaml b/tools/launcher/examples/NVIDIA/Kimi-K2.5-NVFP4/hf_offline_eagle3.yaml deleted file mode 100644 index 16ae740c44c..00000000000 --- a/tools/launcher/examples/NVIDIA/Kimi-K2.5-NVFP4/hf_offline_eagle3.yaml +++ /dev/null @@ -1,121 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for nvidia/Kimi-K2.5-NVFP4. -# -# nvidia/Kimi-K2.5-NVFP4 is the NVIDIA-quantized NVFP4 variant of moonshotai/Kimi-K2.5. -# Same architecture: 1T MoE VLM (32B active, 384 experts, top-8, MLA attention). -# EAGLE3 uses only the text path — the vision encoder is not invoked. -# NVFP4 weights ~591 GB — single GB200 node (4 × 192 GB = 768 GB). -# -# Special requirements: -# - Blackwell GPU required (NVFP4 inference only on B100/B200/GB200) -# - trust_remote_code required for the Kimi tokenizer -# - task_0 uses the NVFP4 checkpoint directly (vLLM v0.15.0+ required) -# - tasks 1–2 use the BF16 moonshotai/Kimi-K2.5 checkpoint for hidden-state -# extraction and training; draft head transfers to NVFP4 at benchmark time -# (NVFP4 is near-lossless so hidden state distributions are equivalent) -# -# MoE-specific notes: -# - task_1: --moe-ep 1 since TP=4 spans all 4 GPUs -# - task_2: consider increasing intermediate_size in eagle_config.json -# (K2.5's per-expert hidden dim is 2048; the draft head uses a dense layer) -# - task_2: verify eagle_decoder_type compatibility with MLA attention -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server (NVFP4) to generate prompt samples -# task_1: Dump hidden states — run BF16 target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head on BF16 hidden states -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM (NVFP4) -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/NVIDIA/Kimi-K2.5-NVFP4/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/NVIDIA/Kimi-K2.5-NVFP4/hf_offline_eagle3.yaml --yes - -job_name: Kimi-K2.5-NVFP4_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/nvidia/Kimi-K2.5-NVFP4 - hf_model_bf16: /hf-local/moonshotai/Kimi-K2.5 - - # Step 1: Data synthesis via vLLM server using NVFP4 checkpoint - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:v0.15.0 - - # Step 2: Dump hidden states using BF16 checkpoint (TP=4, moe-ep=1) - task_1: - script: common/eagle3/dump_offline_data.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - - --tp 4 - - --moe-ep 1 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head on BF16 hidden states - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding against NVFP4 target (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:v0.15.0 diff --git a/tools/launcher/examples/OpenAI/GPT-OSS-20B/eagle3_quick_check.yaml b/tools/launcher/examples/OpenAI/GPT-OSS-20B/eagle3_quick_check.yaml deleted file mode 100644 index f6870249be9..00000000000 --- a/tools/launcher/examples/OpenAI/GPT-OSS-20B/eagle3_quick_check.yaml +++ /dev/null @@ -1,105 +0,0 @@ -# EAGLE3 quick-fail pipeline for openai/gpt-oss-20b. -# -# GPT-OSS-20B is a 20B dense model. -# BF16 weights: ~40 GB — fits easily on a single GB200 node (4 × 192 GB). -# -# Special requirements: -# - TIKTOKEN_RS_CACHE_DIR required for the GPT tokenizer -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/OpenAI/GPT-OSS-20B/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/OpenAI/GPT-OSS-20B/eagle3_quick_check.yaml --yes - -job_name: gpt-oss-20b_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/openai/gpt-oss-20b - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - - TIKTOKEN_RS_CACHE_DIR: /hf-local/tiktoken_cache - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - - TIKTOKEN_RS_CACHE_DIR: /hf-local/tiktoken_cache - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/OpenAI/GPT-OSS-20B/hf_offline_eagle3.yaml b/tools/launcher/examples/OpenAI/GPT-OSS-20B/hf_offline_eagle3.yaml deleted file mode 100644 index f5d99e37aa2..00000000000 --- a/tools/launcher/examples/OpenAI/GPT-OSS-20B/hf_offline_eagle3.yaml +++ /dev/null @@ -1,111 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for openai/gpt-oss-20b. -# -# GPT-OSS-20B is a 20B dense model. -# BF16 weights ~40 GB — fits easily on a single GB200 node (4 × 192 GB = 768 GB). -# -# Special requirements: -# - trust_remote_code required -# - TIKTOKEN_RS_CACHE_DIR must point to a pre-populated tiktoken cache directory -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/OpenAI/GPT-OSS-20B/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/OpenAI/GPT-OSS-20B/hf_offline_eagle3.yaml --yes - -job_name: GPT-OSS-20B_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/openai/gpt-oss-20b - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - - TIKTOKEN_RS_CACHE_DIR: /hf-local/tiktoken_cache - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - # Uses HF backend (device_map=auto) — GPT-OSS-20B uses a custom OpenAI tokenizer - # (openai_harmony) not supported by TRT-LLM hidden state extraction. - task_1: - script: common/eagle3/dump_offline_data_hf.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - - TIKTOKEN_RS_CACHE_DIR: /hf-local/tiktoken_cache - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --trust-remote-code - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - - TIKTOKEN_RS_CACHE_DIR: /hf-local/tiktoken_cache - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Qwen/Qwen3.5-27B/eagle3_quick_check.yaml b/tools/launcher/examples/Qwen/Qwen3.5-27B/eagle3_quick_check.yaml deleted file mode 100644 index b2745525d7b..00000000000 --- a/tools/launcher/examples/Qwen/Qwen3.5-27B/eagle3_quick_check.yaml +++ /dev/null @@ -1,98 +0,0 @@ -# EAGLE3 quick-fail pipeline for Qwen/Qwen3.5-27B. -# -# Qwen3.5-27B is a 27B dense VLM (text + vision). EAGLE3 uses only the text path. -# BF16 weights: ~54 GB — fits on a single GB200 node (4 × 192 GB). -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Qwen/Qwen3.5-27B/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Qwen/Qwen3.5-27B/eagle3_quick_check.yaml --yes - -job_name: Qwen3.5-27B_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/Qwen/Qwen3.5-27B - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=4 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Qwen/Qwen3.5-27B/hf_offline_eagle3.yaml b/tools/launcher/examples/Qwen/Qwen3.5-27B/hf_offline_eagle3.yaml deleted file mode 100644 index eab5005691e..00000000000 --- a/tools/launcher/examples/Qwen/Qwen3.5-27B/hf_offline_eagle3.yaml +++ /dev/null @@ -1,102 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for Qwen/Qwen3.5-27B. -# -# Qwen3.5-27B is a 27B dense VLM (text + vision). EAGLE3 uses only the text path. -# BF16 weights ~54 GB — fits on a single GB200 node (4 × 192 GB = 768 GB). -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Qwen/Qwen3.5-27B/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Qwen/Qwen3.5-27B/hf_offline_eagle3.yaml --yes - -job_name: Qwen3.5-27B_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/Qwen/Qwen3.5-27B - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - # Uses HF backend (device_map=auto) — Qwen3.5-27B is a VLM; HF backend handles - # the text-only path without requiring TRT-LLM support. - task_1: - script: common/eagle3/dump_offline_data_hf.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/eagle3_quick_check.yaml b/tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/eagle3_quick_check.yaml deleted file mode 100644 index fb3d00b0cae..00000000000 --- a/tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/eagle3_quick_check.yaml +++ /dev/null @@ -1,104 +0,0 @@ -# EAGLE3 quick-fail pipeline for Qwen/Qwen3.5-35B-A3B. -# -# Qwen3.5-35B-A3B is a 35B MoE VLM (3B active, 256 experts, top-8, shared expert). -# EAGLE3 uses only the text path. -# BF16 weights: ~70 GB — fits on a single GB200 node (4 × 192 GB). -# -# MoE-specific notes: -# - task_2: check eagle_config.json and consider increasing intermediate_size -# since the draft head uses a dense layer by default. -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Qwen/Qwen3.5-35B-A3B/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/eagle3_quick_check.yaml --yes - -job_name: Qwen3.5-35B-A3B_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/Qwen/Qwen3.5-35B-A3B - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/hf_offline_eagle3.yaml b/tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/hf_offline_eagle3.yaml deleted file mode 100644 index bf2e35632b4..00000000000 --- a/tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/hf_offline_eagle3.yaml +++ /dev/null @@ -1,107 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for Qwen/Qwen3.5-35B-A3B. -# -# Qwen3.5-35B-A3B is a 35B MoE (3B active, 256 experts, top-8, shared expert). -# EAGLE3 uses only the text path. -# BF16 weights ~70 GB — fits on a single GB200 node (4 × 192 GB). -# -# MoE-specific notes: -# - task_2: consider increasing intermediate_size in eagle_config.json -# since the draft head uses a dense layer by default. -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Qwen/Qwen3.5-35B-A3B/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Qwen/Qwen3.5-35B-A3B/hf_offline_eagle3.yaml --yes - -job_name: Qwen3.5-35B-A3B_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/Qwen/Qwen3.5-35B-A3B - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - # Uses HF backend (device_map=auto) — confirmed working in sandbox testing. - task_1: - script: common/eagle3/dump_offline_data_hf.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Qwen/Qwen3.5-9B/eagle3_quick_check.yaml b/tools/launcher/examples/Qwen/Qwen3.5-9B/eagle3_quick_check.yaml deleted file mode 100644 index 7d77fdf299b..00000000000 --- a/tools/launcher/examples/Qwen/Qwen3.5-9B/eagle3_quick_check.yaml +++ /dev/null @@ -1,98 +0,0 @@ -# EAGLE3 quick-fail pipeline for Qwen/Qwen3.5-9B. -# -# Qwen3.5-9B is a 9B dense VLM (text + vision). EAGLE3 uses only the text path. -# BF16 weights: ~18 GB — fits easily on a single GB200 node (4 × 192 GB). -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Qwen/Qwen3.5-9B/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Qwen/Qwen3.5-9B/eagle3_quick_check.yaml --yes - -job_name: Qwen3.5-9B_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/Qwen/Qwen3.5-9B - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/Qwen/Qwen3.5-9B/hf_offline_eagle3.yaml b/tools/launcher/examples/Qwen/Qwen3.5-9B/hf_offline_eagle3.yaml deleted file mode 100644 index a725b54ba0d..00000000000 --- a/tools/launcher/examples/Qwen/Qwen3.5-9B/hf_offline_eagle3.yaml +++ /dev/null @@ -1,102 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for Qwen/Qwen3.5-9B. -# -# Qwen3.5-9B is a 9B dense VLM (text + vision). EAGLE3 uses only the text path. -# BF16 weights ~18 GB — fits easily on a single GB200 node (4 × 192 GB = 768 GB). -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/Qwen/Qwen3.5-9B/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/Qwen/Qwen3.5-9B/hf_offline_eagle3.yaml --yes - -job_name: Qwen3.5-9B_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/Qwen/Qwen3.5-9B - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - # Uses HF backend (device_map=auto) — Qwen3.5-9B is a VLM; HF backend handles - # the text-only path without requiring TRT-LLM support. - task_1: - script: common/eagle3/dump_offline_data_hf.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/StepFun/Step-3.5-Flash/eagle3_quick_check.yaml b/tools/launcher/examples/StepFun/Step-3.5-Flash/eagle3_quick_check.yaml deleted file mode 100644 index c98b75621d0..00000000000 --- a/tools/launcher/examples/StepFun/Step-3.5-Flash/eagle3_quick_check.yaml +++ /dev/null @@ -1,108 +0,0 @@ -# EAGLE3 quick-fail pipeline for stepfun-ai/Step-3.5-Flash. -# -# Step-3.5-Flash is a 197B MoE (11B active, 288 experts + 1 shared, top-8). -# Has built-in MTP (multi-token prediction) head and sliding window attention. -# BF16 weights: ~394 GB — fits on a single GB200 node (4 × 192 GB = 768 GB). -# -# Special requirements: -# - trust_remote_code required -# - 256K context window supported but use 8192 for quick-fail -# - Sliding window attention (SWA) — verify TRT-LLM support -# -# MoE-specific notes: -# - Draft head intermediate_size may need tuning in eagle_config.json -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/StepFun/Step-3.5-Flash/eagle3_quick_check.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/StepFun/Step-3.5-Flash/eagle3_quick_check.yaml --yes - -job_name: Step-3.5-Flash_EAGLE3_quick_fail_check -pipeline: - allow_to_fail: true - skip: false - note: - - global_vars: - hf_model: /hf-local/stepfun-ai/Step-3.5-Flash - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - task_1: - script: common/eagle3/dump_offline_data_vllm.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 3: Train EAGLE3 draft head (offline, single task) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.num_epochs=1 - - training.lr=3e-4 - - training.save_steps=500000 - - training.train_bs=8 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest diff --git a/tools/launcher/examples/StepFun/Step-3.5-Flash/hf_offline_eagle3.yaml b/tools/launcher/examples/StepFun/Step-3.5-Flash/hf_offline_eagle3.yaml deleted file mode 100644 index 75c639ea7ce..00000000000 --- a/tools/launcher/examples/StepFun/Step-3.5-Flash/hf_offline_eagle3.yaml +++ /dev/null @@ -1,113 +0,0 @@ -# EAGLE3 offline speculative decoding pipeline for stepfun-ai/Step-3.5-Flash. -# -# Step-3.5-Flash is a 197B MoE (11B active, 288 experts + 1 shared, top-8). -# Has built-in MTP head and sliding window attention (SWA). -# BF16 weights ~394 GB — fits on a single GB200 node (4 × 192 GB = 768 GB). -# -# Special requirements: -# - trust_remote_code required -# - 256K context window supported; use 8192 for typical EAGLE3 training -# - Sliding window attention (SWA) — verify vLLM support before running -# -# MoE-specific notes: -# - task_2: consider increasing intermediate_size in eagle_config.json -# since the draft head uses a dense layer by default. -# -# 4-step pipeline: -# task_0: Data synthesis — query vLLM server to generate prompt samples -# task_1: Dump hidden states — run target model to capture hidden states -# task_2: Offline training — train the EAGLE3 draft head -# task_3: Benchmark — evaluate speculative decoding speedup via VLLM -# -# All tasks share /scratchspace to pass artifacts between steps. -# -# Usage: -# uv run launch.py --yaml examples/StepFun/Step-3.5-Flash/hf_offline_eagle3.yaml --yes -# uv run slurm.py --yaml modules/Model-Optimizer/tools/launcher/examples/StepFun/Step-3.5-Flash/hf_offline_eagle3.yaml --yes - -job_name: Step-3.5-Flash_EAGLE3_offline -pipeline: - allow_to_fail: false - skip: false - note: - - global_vars: - hf_model: /hf-local/stepfun-ai/Step-3.5-Flash - - # Step 1: Data synthesis via vLLM server - task_0: - script: common/vllm/query.sh - args: - - --model <> - - --tensor-parallel-size 4 - - --trust-remote-code - - -- - - --data /hf-local/modelopt/Speculative-Decoding-Dataset-v2-default - - --save /scratchspace/data - environment: - - HF_LOCAL: /hf-local - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest - - # Step 2: Dump hidden states from target model - # Uses HF backend (device_map=auto) — Step-3.5-Flash uses sliding window attention (SWA) - # which is not supported by TRT-LLM hidden state extraction. - task_1: - script: common/eagle3/dump_offline_data_hf.sh - args: - - --input-data /scratchspace/data - - --output-dir /scratchspace/offline_hidden_states - - --max-seq-len 8192 - environment: - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 3: Train EAGLE3 draft head (offline) - task_2: - script: common/eagle3/train_eagle.sh - args: - - --config modules/Model-Optimizer/modelopt_recipes/general/speculative_decoding/eagle3.yaml - - model.model_name_or_path=<> - - data.offline_data_path=/scratchspace/offline_hidden_states - - training.output_dir=/scratchspace/eagle3 - - training.training_seq_len=4096 - - training.disable_tqdm=true - - training.ar_validate_steps=500000 - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10 - - # Step 4: Benchmark speculative decoding (VLLM backend) - task_3: - script: common/specdec_bench/quick_check.sh - args: - - --draft_model_dir /scratchspace/export - - --draft_length 3 - - --output_length 4096 - - --engine VLLM - - --tp_size 4 - - --ep_size 1 - - --speculative_algorithm EAGLE3 - - --mtbench /hf-local/HuggingFaceH4/mt_bench_prompts/raw/question.jsonl - - --concurrency 1 - environment: - - HF_LOCAL: /hf-local - - HF_MODEL_CKPT: <> - slurm_config: - _factory_: "slurm_factory" - nodes: 1 - ntasks_per_node: 1 - gpus_per_node: 4 - container: vllm/vllm-openai:latest