From 1eb8ce64db3a299c338aeaaf2270ba651c3d6445 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 23 Jul 2026 12:29:47 +0000 Subject: [PATCH 01/13] Add ModelOpt QAD skill Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 160 ++++++++++++++++++ .../qad/assets/nemotron-cascade-2-blend.yaml | 44 +++++ .claude/skills/qad | 1 + examples/megatron_bridge/distill.py | 14 ++ tests/examples/megatron_bridge/test_qad.py | 10 +- 5 files changed, 226 insertions(+), 3 deletions(-) create mode 100644 .agents/skills/qad/SKILL.md create mode 100644 .agents/skills/qad/assets/nemotron-cascade-2-blend.yaml create mode 120000 .claude/skills/qad diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md new file mode 100644 index 00000000000..be8e57a88de --- /dev/null +++ b/.agents/skills/qad/SKILL.md @@ -0,0 +1,160 @@ +--- +name: qad +description: >- + Recover a measured BF16-to-PTQ accuracy gap on Slurm with ModelOpt + Quantization-Aware Distillation (QAD) through Megatron Bridge. Use when PTQ + has already been benchmarked against the matching BF16 model and the + quantized checkpoint needs short, evidence-driven distillation; also use for + QAD topology, data preparation, Slurm launch, resume, checkpoint export, or + recovery decisions. +--- + +# ModelOpt Quantization-Aware Distillation + +Use QAD only to recover a material, apples-to-apples PTQ benchmark gap. Do not +train merely because a quantized checkpoint exists. + +## 1. Read the supported workflow + +Read these sources before constructing commands: + +- `examples/megatron_bridge/README.md`, especially Post-Training Quantization, + Data Preparation, Quantization Aware Distillation, export, and Slurm usage +- `examples/dataset/MEGATRON_DATA_PREP.md`, especially token-budgeted blends +- `examples/megatron_bridge/{quantize.py,distill.py}` via `--help` + +Treat them as the source of truth for mutable flags, containers, model support, +and checkpoint formats. Do not copy their full command lines into new wrappers. +The deprecated `examples/llm_qad` flow is not the default. + +## 2. Establish the gap + +Use the `evaluation` and `compare-results` skills to run and validate the same +benchmark configuration on the BF16 source and the export of the exact +quantized Megatron checkpoint that will initialize QAD. If the existing PTQ +result came from another checkpoint or its lineage is unclear, export and +benchmark the intended Megatron checkpoint first. Record its path or identifier, +metric, direction, uncertainty, and acceptable delta. Stop if the gap is not +material or the runs are not comparable. + +## 3. Set up Slurm execution + +Read `skills/common/environment-setup.md` and detect whether the Slurm target is +local or remote. This version of the skill supports Slurm only; stop if the +target is not Slurm. + +- On local Slurm, follow `skills/common/workspace-management.md` and + `skills/common/slurm-setup.md` directly. Do not add SSH or sync steps. +- On remote Slurm, also follow `skills/common/remote-execution.md`: establish + its persistent SSH session, create matching local and remote session/model + workspaces, and sync the source plus the two-script Slurm wrapper/inner runner. + +Use the common account, partition, registry-auth, submission, and monitoring +procedures in either case. + +Keep BF16, the exact benchmarked PTQ Megatron checkpoint and its export, QAD +Megatron, exported checkpoints, data, logs, and benchmark results in the same +session/model workspace. QAD must start from the checkpoint produced by +`examples/megatron_bridge/quantize.py` because it carries the ModelOpt state; an +exported HF PTQ checkpoint is not a substitute. + +## 4. Choose topology explicitly + +Inspect model parameter count and architecture, sequence length, GPU count/type +and memory, then choose the smallest topology that fits both student and teacher: + +1. Choose node count from model-state plus activation memory; do not start with + an oversized multi-node topology. +2. Use TP only when model-state/GEMM size warrants it, and preserve attention + head and hidden-size divisibility. +3. Use CP to distribute the 32K context when activation/attention memory is the + constraint. Prefer CP before TP for a small dense model at long context. +4. Keep EP=1 for dense models. For MoE, choose EP from expert count, memory, and + available ranks; verify expert and data-parallel divisibility. +5. Add PP only when layer/model state still does not fit. + +Verify `DP = world_size / (TP * PP * CP)` is integral and +`GBS % (MBS * DP) == 0`. Record nodes, GPUs/node, TP/PP/CP/EP/DP, MBS, GPU +memory, and why each non-one dimension is needed. + +Example only: for the requested one-node, eight-H100 Qwen3-0.6B validation at +32K, TP=1, PP=1, CP=4, EP=1, DP=2, MBS=1 is a reasonable starting point. This +is validation evidence, not a default topology; derive a fresh topology for +every target model and cluster. + +## 5. Prepare only the next data tranche + +Copy `assets/nemotron-cascade-2-blend.yaml`, then set its tokenizer, output +directory, and token budget. Run: + +```bash +python -m modelopt.torch.utils.plugins.prepare_megatron_data_blend \ + --config +``` + +The utility streams `nvidia/Nemotron-Cascade-2-SFT-Data`, applies a deterministic +approximate buffer shuffle, and stops at the token budget. Never snapshot or +materialize the full dataset. For the initial step-150 gate, the default +2.6-billion-token budget covers +`150 * 512 * 32768` tokens plus a small margin. Use a much smaller explicit +budget for workflow validation. At very small budgets, use a representative +subset of source configs rather than giving a rare config too few documents +for Megatron's train/validation split. + +Pass the generated `data_blend.txt` entries as `distill.py --data_paths`. +The supported real-data `GPTDatasetConfig` shuffles documents and concatenates +them into fixed-length 32K samples, so sequences are packed instead of padded. +Do not use mock data as training evidence. + +## 6. Retain PTQ lineage, then run staged QAD + +Use the exact PTQ Megatron checkpoint established in step 2, or produce and +benchmark it before QAD. Follow the current Megatron Bridge README for PTQ, QAD, +resume, and export commands. Apply these QAD defaults: + +| Setting | Default | +| --- | --- | +| Sequence length | 32768 | +| Peak / minimum LR | `1e-5` / `1e-6` | +| LR schedule | cosine | +| Training cap | 1000 iterations | +| Global batch size | 512 | +| Dataset | `nvidia/Nemotron-Cascade-2-SFT-Data` | +| Initial eval/save/exit | iteration 150 | +| Slurm duration exit | 220 minutes for a 4-hour allocation | + +Keep `train_iters=1000` from the first run so the cosine schedule has a stable +horizon. Set `eval_interval=150`, `exit_interval=150`, and +`exit_duration_in_mins=220` for the initial stage. The duration exit is a +checkpointing safety margin, not the training target. + +Use one result-bearing Slurm job per stage and put Pyxis container flags on the +final `srun`. Fold startup validation into that job; do not submit separate GPU +preflight or smoke jobs. + +## 7. Benchmark before continuing + +At the initial exit: + +1. Confirm finite QAD/KD loss, sensible learning rate, non-pathological gradient + norm, and an iteration-150 checkpoint. +2. Export that QAD checkpoint with the current quantized Megatron exporter. +3. Run the identical benchmark on BF16, the step-2 PTQ export, and QAD-150. +4. Calculate the original gap and recovery. For higher-is-better metrics: + `gap = BF16 - PTQ`, `recovered = QAD - PTQ`, and + `recovery_fraction = recovered / gap`. + +Continue only if benchmark recovery is positive beyond run noise and loss is +stable. Choose the next absolute checkpoint from the evidence (for example 300, +500, 750, or 1000), resume the same output directory with `train_iters=1000`, +and set both `eval_interval` and `exit_interval` to that target. Stop on +recovered gap, plateau, regression, divergence, or iteration 1000. Never raise +the 1000-step cap without explicit user direction. + +## 8. Report evidence + +Report exact source revision, commands, Slurm job/account/partition, container, +paths, topology derivation, dataset configs/seed/token budget/materialized +counts, PTQ format, checkpoint iterations, loss/LR/grad trend, scheduler state, +and BF16/PTQ/QAD benchmark scores. Support success with both scheduler and log +evidence; identify the first real error when a run fails. diff --git a/.agents/skills/qad/assets/nemotron-cascade-2-blend.yaml b/.agents/skills/qad/assets/nemotron-cascade-2-blend.yaml new file mode 100644 index 00000000000..cb28ce5b147 --- /dev/null +++ b/.agents/skills/qad/assets/nemotron-cascade-2-blend.yaml @@ -0,0 +1,44 @@ +tokenizer: +output_dir: /data/nemotron-cascade-2-through-150 +target_tokens: 2_600_000_000 +sources: + - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data + config: math + split: train + content_field: messages + weight: 21.1 + - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data + config: science + split: train + content_field: messages + weight: 10.9 + - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data + config: chat + split: train + content_field: messages + weight: 56.2 + - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data + config: instruction_following + split: train + content_field: messages + weight: 3.3 + - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data + config: safety + split: train + content_field: messages + weight: 0.02 + - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data + config: conversational_agent + split: train + content_field: messages + weight: 3.3 + - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data + config: swe + split: train + content_field: messages + weight: 1.8 + - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data + config: terminal_agent + split: train + content_field: messages + weight: 3.3 diff --git a/.claude/skills/qad b/.claude/skills/qad new file mode 120000 index 00000000000..9ac1b527220 --- /dev/null +++ b/.claude/skills/qad @@ -0,0 +1 @@ +../../.agents/skills/qad \ No newline at end of file diff --git a/examples/megatron_bridge/distill.py b/examples/megatron_bridge/distill.py index 96880d2ae82..b1d2bca363a 100644 --- a/examples/megatron_bridge/distill.py +++ b/examples/megatron_bridge/distill.py @@ -167,6 +167,18 @@ def get_args(): parser.add_argument( "--eval_iters", type=int, default=32, help="Number of batches per validation stage" ) + parser.add_argument( + "--exit_interval", + type=int, + default=None, + help="Save a checkpoint and exit when the iteration is divisible by this value", + ) + parser.add_argument( + "--exit_duration_in_mins", + type=int, + default=None, + help="Save a checkpoint and exit after this many minutes", + ) # Logging arguments parser.add_argument("--log_interval", type=int, default=10, help="Write to log every steps") parser.add_argument( @@ -328,6 +340,8 @@ def _restore_student_hook(model_chunks): train_iters=args.train_iters, global_batch_size=args.gbs, micro_batch_size=args.mbs, + exit_interval=args.exit_interval, + exit_duration_in_mins=args.exit_duration_in_mins, manual_gc=True, manual_gc_interval=100, ), diff --git a/tests/examples/megatron_bridge/test_qad.py b/tests/examples/megatron_bridge/test_qad.py index f36b727b9c4..1d6f6731b7f 100644 --- a/tests/examples/megatron_bridge/test_qad.py +++ b/tests/examples/megatron_bridge/test_qad.py @@ -68,7 +68,8 @@ def test_qad(tmp_path: Path, num_gpus, create_student, is_vlm, is_moe): hf_model_path = create_student(tmp_path) quantized_megatron_path = tmp_path / "quantized_megatron" distill_output_dir = tmp_path / "qad_output" - train_iters = 2 + train_iters = 3 + early_exit_iter = 2 # TODO: VLMs disable sequence parallelism, so tensor parallelism can't be used here. # Flip to tp_size=num_gpus in nemo:26.08 container @@ -109,13 +110,16 @@ def test_qad(tmp_path: Path, num_gpus, create_student, is_vlm, is_moe): gbs=4, train_iters=train_iters, lr_warmup_iters=2, - eval_interval=train_iters, + eval_interval=early_exit_iter, eval_iters=1, log_interval=1, + exit_interval=early_exit_iter, + exit_duration_in_mins=60, ) run_example_command(distill_cmd, example_path="megatron_bridge", setup_free_port=True) distilled_megatron_path = distill_output_dir / "checkpoints" - assert (distilled_megatron_path / "latest_checkpointed_iteration.txt").exists() + tracker = distilled_megatron_path / "latest_checkpointed_iteration.txt" + assert tracker.read_text(encoding="utf-8").strip() == str(early_exit_iter) assert list(distilled_megatron_path.rglob("modelopt_state")), ( "Expected modelopt_state to be preserved in the distilled (QAD) checkpoint" ) From afd59b43bc19fa8b1dd05e07e2389b0555c89407 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 23 Jul 2026 12:40:02 +0000 Subject: [PATCH 02/13] Preserve PTQ configuration for QAD Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index be8e57a88de..4b7d4734c82 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -29,6 +29,11 @@ The deprecated `examples/llm_qad` flow is not the default. ## 2. Establish the gap +When QAD follows `evaluation` or `quant-recipe-search`, reuse its exact PTQ +checkpoint and complete config or recipe. Preserve the quantization format, +layer selection, calibration dataset, sample count, sequence length, seed, and +topology; do not substitute a cheaper or otherwise different PTQ run. + Use the `evaluation` and `compare-results` skills to run and validate the same benchmark configuration on the BF16 source and the export of the exact quantized Megatron checkpoint that will initialize QAD. If the existing PTQ From 8abffb7fb3648bf8b9e98b3c50e57741af3048b9 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 23 Jul 2026 12:42:28 +0000 Subject: [PATCH 03/13] Clarify PTQ handoff to QAD Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index 4b7d4734c82..9d23d28f9d0 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -29,10 +29,11 @@ The deprecated `examples/llm_qad` flow is not the default. ## 2. Establish the gap -When QAD follows `evaluation` or `quant-recipe-search`, reuse its exact PTQ -checkpoint and complete config or recipe. Preserve the quantization format, -layer selection, calibration dataset, sample count, sequence length, seed, and -topology; do not substitute a cheaper or otherwise different PTQ run. +When QAD follows `evaluation` or `quant-recipe-search`, reuse its complete PTQ +config or recipe to reproduce PTQ with `examples/megatron_bridge/quantize.py`; +do not use the usual HF checkpoint as the QAD student. Preserve the quantization +format, layer selection, calibration dataset, sample count, sequence length, +seed, and topology; do not substitute a cheaper or otherwise different PTQ run. Use the `evaluation` and `compare-results` skills to run and validate the same benchmark configuration on the BF16 source and the export of the exact From 494d46ceb52174cee9194d24bdc10159068471cf Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 23 Jul 2026 12:47:23 +0000 Subject: [PATCH 04/13] Reuse existing PTQ evaluation for QAD Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 60 ++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 28 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index 9d23d28f9d0..96a2980386b 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -27,21 +27,20 @@ Treat them as the source of truth for mutable flags, containers, model support, and checkpoint formats. Do not copy their full command lines into new wrappers. The deprecated `examples/llm_qad` flow is not the default. -## 2. Establish the gap - -When QAD follows `evaluation` or `quant-recipe-search`, reuse its complete PTQ -config or recipe to reproduce PTQ with `examples/megatron_bridge/quantize.py`; -do not use the usual HF checkpoint as the QAD student. Preserve the quantization -format, layer selection, calibration dataset, sample count, sequence length, -seed, and topology; do not substitute a cheaper or otherwise different PTQ run. - -Use the `evaluation` and `compare-results` skills to run and validate the same -benchmark configuration on the BF16 source and the export of the exact -quantized Megatron checkpoint that will initialize QAD. If the existing PTQ -result came from another checkpoint or its lineage is unclear, export and -benchmark the intended Megatron checkpoint first. Record its path or identifier, -metric, direction, uncertainty, and acceptable delta. Stop if the gap is not -material or the runs are not comparable. +## 2. Reuse or establish the gap + +When QAD follows `evaluation`, `compare-results`, or `quant-recipe-search`, +reuse the validated BF16/PTQ scores and exact benchmark configuration; do not +rerun valid baselines. Record their run IDs, metric, direction, uncertainty, and +acceptable delta. Use `evaluation` and `compare-results` only for missing, +unvalidated, or non-comparable results. Stop if the gap is not material. + +Reuse the preceding PTQ config or recipe to reproduce PTQ with +`examples/megatron_bridge/quantize.py`; do not use the usual HF checkpoint as +the QAD student. Preserve the quantization format, layer selection, calibration +dataset, sample count, sequence length, and seed. Choose Megatron execution +topology separately without changing quantization semantics. If any quantization +setting must change, treat it as a new PTQ candidate and evaluate it before QAD. ## 3. Set up Slurm execution @@ -58,11 +57,11 @@ target is not Slurm. Use the common account, partition, registry-auth, submission, and monitoring procedures in either case. -Keep BF16, the exact benchmarked PTQ Megatron checkpoint and its export, QAD -Megatron, exported checkpoints, data, logs, and benchmark results in the same -session/model workspace. QAD must start from the checkpoint produced by -`examples/megatron_bridge/quantize.py` because it carries the ModelOpt state; an -exported HF PTQ checkpoint is not a substitute. +Keep the reused BF16/PTQ run references and configs with the reproduced PTQ +Megatron checkpoint, QAD checkpoints, data, logs, exports, and new benchmark +results in the same session/model workspace. QAD must start from the checkpoint +produced by `examples/megatron_bridge/quantize.py` because it carries the +ModelOpt state; an exported HF PTQ checkpoint is not a substitute. ## 4. Choose topology explicitly @@ -112,11 +111,13 @@ The supported real-data `GPTDatasetConfig` shuffles documents and concatenates them into fixed-length 32K samples, so sequences are packed instead of padded. Do not use mock data as training evidence. -## 6. Retain PTQ lineage, then run staged QAD +## 6. Reproduce PTQ, then run staged QAD -Use the exact PTQ Megatron checkpoint established in step 2, or produce and -benchmark it before QAD. Follow the current Megatron Bridge README for PTQ, QAD, -resume, and export commands. Apply these QAD defaults: +Produce the required Megatron checkpoint with the PTQ config or recipe preserved +in step 2. Reuse a valid preceding PTQ evaluation; run a new PTQ evaluation only +when the baseline is missing, invalid, or non-comparable, or a quantization +setting changed. Follow the current Megatron Bridge README for PTQ, QAD, resume, +and export commands. Apply these QAD defaults: | Setting | Default | | --- | --- | @@ -145,7 +146,9 @@ At the initial exit: 1. Confirm finite QAD/KD loss, sensible learning rate, non-pathological gradient norm, and an iteration-150 checkpoint. 2. Export that QAD checkpoint with the current quantized Megatron exporter. -3. Run the identical benchmark on BF16, the step-2 PTQ export, and QAD-150. +3. Evaluate QAD-150 with the exact step-2 benchmark configuration. Reuse + validated, comparable BF16/PTQ results; run only missing, invalid, or + non-comparable baselines. 4. Calculate the original gap and recovery. For higher-is-better metrics: `gap = BF16 - PTQ`, `recovered = QAD - PTQ`, and `recovery_fraction = recovered / gap`. @@ -161,6 +164,7 @@ the 1000-step cap without explicit user direction. Report exact source revision, commands, Slurm job/account/partition, container, paths, topology derivation, dataset configs/seed/token budget/materialized -counts, PTQ format, checkpoint iterations, loss/LR/grad trend, scheduler state, -and BF16/PTQ/QAD benchmark scores. Support success with both scheduler and log -evidence; identify the first real error when a run fails. +counts, PTQ format, reused evaluation run IDs, checkpoint iterations, +loss/LR/grad trend, scheduler state, and BF16/PTQ/QAD benchmark scores. State +which scores were reused versus newly evaluated. Support success with both +scheduler and log evidence; identify the first real error when a run fails. From f8f87645f09617c80776b795fbb0c4541d159a17 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 23 Jul 2026 12:53:10 +0000 Subject: [PATCH 05/13] Clarify Megatron expert parallel topology Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index 96a2980386b..ae862471274 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -75,12 +75,19 @@ and memory, then choose the smallest topology that fits both student and teacher 3. Use CP to distribute the 32K context when activation/attention memory is the constraint. Prefer CP before TP for a small dense model at long context. 4. Keep EP=1 for dense models. For MoE, choose EP from expert count, memory, and - available ranks; verify expert and data-parallel divisibility. + available ranks, and inspect the current `distill.py` value for expert tensor + parallelism (ETP); do not assume ETP equals TP. 5. Add PP only when layer/model state still does not fit. -Verify `DP = world_size / (TP * PP * CP)` is integral and -`GBS % (MBS * DP) == 0`. Record nodes, GPUs/node, TP/PP/CP/EP/DP, MBS, GPU -memory, and why each non-one dimension is needed. +For MoE, Megatron folds two overlapping meshes onto the same ranks within each +PP stage; EP/ETP do not multiply the dense TP/CP mesh: + +- Attention DP: `DP = world_size / (TP * PP * CP)` +- Expert DP: `EDP = world_size / (ETP * EP * PP)` + +Require both divisions to be integral, `num_experts % EP == 0`, and +`GBS % (MBS * DP) == 0`. Record nodes, GPUs/node, TP/PP/CP/EP/ETP/DP/EDP, MBS, +GPU memory, and why each non-one dimension is needed. Example only: for the requested one-node, eight-H100 Qwen3-0.6B validation at 32K, TP=1, PP=1, CP=4, EP=1, DP=2, MBS=1 is a reasonable starting point. This From ef9be330ce77cf615d3660f28f6f85c8f57b2bc9 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 23 Jul 2026 13:37:43 +0000 Subject: [PATCH 06/13] Add QAD training safety controls Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 61 +++++++++++++++------- examples/megatron_bridge/distill.py | 12 +++-- tests/examples/megatron_bridge/test_qad.py | 2 + 3 files changed, 54 insertions(+), 21 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index ae862471274..6ac55f6cfe5 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -1,18 +1,18 @@ --- name: qad description: >- - Recover a measured BF16-to-PTQ accuracy gap on Slurm with ModelOpt - Quantization-Aware Distillation (QAD) through Megatron Bridge. Use when PTQ - has already been benchmarked against the matching BF16 model and the - quantized checkpoint needs short, evidence-driven distillation; also use for - QAD topology, data preparation, Slurm launch, resume, checkpoint export, or - recovery decisions. + Run explicitly requested ModelOpt Quantization-Aware Distillation (QAD) on + Slurm through Megatron Bridge to recover a measured BF16-to-PTQ accuracy gap. + Use only when the user explicitly asks for QAD, including its topology, data + preparation, Slurm launch, resume, checkpoint export, or recovery decisions. --- # ModelOpt Quantization-Aware Distillation -Use QAD only to recover a material, apples-to-apples PTQ benchmark gap. Do not -train merely because a quantized checkpoint exists. +QAD is expensive. Use it only to recover a material, apples-to-apples PTQ +benchmark gap and only when the user explicitly requests QAD for the target +model or run. Do not infer permission from a quantized checkpoint, evaluation +gap, or recipe-search result. ## 1. Read the supported workflow @@ -116,7 +116,16 @@ for Megatron's train/validation split. Pass the generated `data_blend.txt` entries as `distill.py --data_paths`. The supported real-data `GPTDatasetConfig` shuffles documents and concatenates them into fixed-length 32K samples, so sequences are packed instead of padded. -Do not use mock data as training evidence. +It deterministically holds out 1% of those shuffled documents for validation +with `split="99,1,0"`; bounded materialization uses shuffle seed 42, while the +training seed controls ordering within the Megatron splits. Do not download a +duplicate validation copy. With the defaults, two GBS-512 32K validation +batches consume 33.6M tokens from the nominal 26M-token holdout, so about 1.3 +passes per validation event is accepted. Recalculate this ratio when changing +GBS, sequence length, or `eval_iters`, and increase the token budget rather than +allowing substantially more repetition. Treat this loss as a training-health +signal; the independent benchmark remains the recovery gate. Do not use mock +data as training evidence. ## 6. Reproduce PTQ, then run staged QAD @@ -134,24 +143,39 @@ and export commands. Apply these QAD defaults: | Training cap | 1000 iterations | | Global batch size | 512 | | Dataset | `nvidia/Nemotron-Cascade-2-SFT-Data` | -| Initial eval/save/exit | iteration 150 | +| Validation | every 25 iterations; deterministic 1% holdout; 2 batches | +| Checkpoint interval | 50 iterations | +| Initial benchmark/exit | iteration 150 | | Slurm duration exit | 220 minutes for a 4-hour allocation | Keep `train_iters=1000` from the first run so the cosine schedule has a stable -horizon. Set `eval_interval=150`, `exit_interval=150`, and -`exit_duration_in_mins=220` for the initial stage. The duration exit is a +horizon. Set `save_interval=50`, `eval_interval=25`, `eval_iters=2`, +`exit_interval=150`, and `exit_duration_in_mins=220` for the initial stage. This +preserves checkpoints at iterations 50, 100, and 150. The duration exit is a checkpointing safety margin, not the training target. Use one result-bearing Slurm job per stage and put Pyxis container flags on the final `srun`. Fold startup validation into that job; do not submit separate GPU preflight or smoke jobs. -## 7. Benchmark before continuing +## 7. Monitor loss, then benchmark + +Monitor the running log instead of waiting for the stage to finish. Record total +or logits-distillation loss every log interval. The expected signal is a +decreasing smoothed trend, not a decrease at every noisy step: + +1. From iteration 100 onward, compare the median loss in each 50-step window + with the preceding window. +2. Treat non-finite loss, NaN iterations, repeated skipped iterations, or a + sharp sustained increase as immediate failure. +3. Flag one flat or rising window. If a second consecutive window also fails to + decrease, stop at the checkpoint ending that second window. Preserve the + latest known-good checkpoint and diagnose before resuming. At the initial exit: -1. Confirm finite QAD/KD loss, sensible learning rate, non-pathological gradient - norm, and an iteration-150 checkpoint. +1. Confirm the smoothed QAD/KD loss decreased, learning rate is sensible, + gradient norm is non-pathological, and checkpoints 50, 100, and 150 exist. 2. Export that QAD checkpoint with the current quantized Megatron exporter. 3. Evaluate QAD-150 with the exact step-2 benchmark configuration. Reuse validated, comparable BF16/PTQ results; run only missing, invalid, or @@ -163,9 +187,10 @@ At the initial exit: Continue only if benchmark recovery is positive beyond run noise and loss is stable. Choose the next absolute checkpoint from the evidence (for example 300, 500, 750, or 1000), resume the same output directory with `train_iters=1000`, -and set both `eval_interval` and `exit_interval` to that target. Stop on -recovered gap, plateau, regression, divergence, or iteration 1000. Never raise -the 1000-step cap without explicit user direction. +retain `eval_interval=25`, `eval_iters=2`, and `save_interval=50`, and set only +`exit_interval` to that target. Stop on recovered gap, plateau, regression, +divergence, or iteration 1000. Never raise the 1000-step cap without explicit +user direction. ## 8. Report evidence diff --git a/examples/megatron_bridge/distill.py b/examples/megatron_bridge/distill.py index b1d2bca363a..332c09224db 100644 --- a/examples/megatron_bridge/distill.py +++ b/examples/megatron_bridge/distill.py @@ -161,11 +161,15 @@ def get_args(): help="Modules to recompute with --recompute_granularity=selective. Defaults to ['core_attn']. " "Allowed: core_attn, mlp, moe, moe_act, layernorm, mla_up_proj, shared_experts.", ) + parser.add_argument("--eval_interval", type=int, default=100, help="Validate every steps") parser.add_argument( - "--eval_interval", type=int, default=100, help="Validate + checkpoint every steps" + "--eval_iters", type=int, default=32, help="Number of batches per validation stage" ) parser.add_argument( - "--eval_iters", type=int, default=32, help="Number of batches per validation stage" + "--save_interval", + type=int, + default=None, + help="Checkpoint every steps; defaults to --eval_interval", ) parser.add_argument( "--exit_interval", @@ -370,7 +374,9 @@ def _restore_student_hook(model_chunks): tokenizer_type="NullTokenizer", vocab_size=distill_provider.vocab_size ), checkpoint=CheckpointConfig( - save_interval=args.eval_interval, + save_interval=( + args.save_interval if args.save_interval is not None else args.eval_interval + ), save=checkpoint_dir, load=checkpoint_dir, # Resume from this directory (if exists) most_recent_k=5, # Keeps 5 most recent checkpoints (not metric-based) diff --git a/tests/examples/megatron_bridge/test_qad.py b/tests/examples/megatron_bridge/test_qad.py index 1d6f6731b7f..92c122bbd8f 100644 --- a/tests/examples/megatron_bridge/test_qad.py +++ b/tests/examples/megatron_bridge/test_qad.py @@ -112,6 +112,7 @@ def test_qad(tmp_path: Path, num_gpus, create_student, is_vlm, is_moe): lr_warmup_iters=2, eval_interval=early_exit_iter, eval_iters=1, + save_interval=1, log_interval=1, exit_interval=early_exit_iter, exit_duration_in_mins=60, @@ -120,6 +121,7 @@ def test_qad(tmp_path: Path, num_gpus, create_student, is_vlm, is_moe): distilled_megatron_path = distill_output_dir / "checkpoints" tracker = distilled_megatron_path / "latest_checkpointed_iteration.txt" assert tracker.read_text(encoding="utf-8").strip() == str(early_exit_iter) + assert (distilled_megatron_path / "iter_0000001").is_dir() assert list(distilled_megatron_path.rglob("modelopt_state")), ( "Expected modelopt_state to be preserved in the distilled (QAD) checkpoint" ) From 61aa5f07cd17f8800f221efe96505538620eafb7 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Fri, 24 Jul 2026 04:01:01 +0000 Subject: [PATCH 07/13] Add QAD PTQ compatibility gate Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index 6ac55f6cfe5..b9c26e0d4a5 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -42,6 +42,15 @@ dataset, sample count, sequence length, and seed. Choose Megatron execution topology separately without changing quantization semantics. If any quantization setting must change, treat it as a new PTQ candidate and evaluate it before QAD. +In the exact runtime, confirm `AutoBridge.can_handle()` accepts the model, then +reproduce PTQ with `quantize.py`. Sanity-check the checkpoint's master-rank +`.quant_summary.txt`: every enabled statically calibrated quantizer it lists +must have a finite, positive `amax`. Accept `dynamic` or format-defined `None` +only when the preserved recipe intentionally uses dynamic or MX quantization; +with model parallelism, do not treat this rank-local summary as exhaustive. +Stop on unsupported conversion, failed PTQ, missing ModelOpt state, or +unexpected `amax`. + ## 3. Set up Slurm execution Read `skills/common/environment-setup.md` and detect whether the Slurm target is @@ -129,11 +138,11 @@ data as training evidence. ## 6. Reproduce PTQ, then run staged QAD -Produce the required Megatron checkpoint with the PTQ config or recipe preserved -in step 2. Reuse a valid preceding PTQ evaluation; run a new PTQ evaluation only -when the baseline is missing, invalid, or non-comparable, or a quantization -setting changed. Follow the current Megatron Bridge README for PTQ, QAD, resume, -and export commands. Apply these QAD defaults: +Use the verified Megatron PTQ checkpoint from step 2. Reuse a valid preceding +PTQ evaluation; run a new PTQ evaluation only when the baseline is missing, +invalid, or non-comparable, or a quantization setting changed. Follow the +current Megatron Bridge README for QAD, resume, and export commands. Apply these +QAD defaults: | Setting | Default | | --- | --- | From 62fda5213fc733f4aa6e3d695cc9362350dc40a7 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Wed, 29 Jul 2026 06:51:56 +0000 Subject: [PATCH 08/13] Move QAD data blend to Megatron example Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 6 ++++-- examples/megatron_bridge/README.md | 5 +++++ .../megatron_bridge/data}/nemotron-cascade-2-blend.yaml | 1 + 3 files changed, 10 insertions(+), 2 deletions(-) rename {.agents/skills/qad/assets => examples/megatron_bridge/data}/nemotron-cascade-2-blend.yaml (94%) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index b9c26e0d4a5..67863b04b08 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -105,8 +105,10 @@ every target model and cluster. ## 5. Prepare only the next data tranche -Copy `assets/nemotron-cascade-2-blend.yaml`, then set its tokenizer, output -directory, and token budget. Run: +Copy `examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml` into the +session workspace, replace the tokenizer placeholder with the target model's +Hugging Face ID or local path, and set the output directory and token budget. +Run: ```bash python -m modelopt.torch.utils.plugins.prepare_megatron_data_blend \ diff --git a/examples/megatron_bridge/README.md b/examples/megatron_bridge/README.md index 45606d691ee..981337a3285 100644 --- a/examples/megatron_bridge/README.md +++ b/examples/megatron_bridge/README.md @@ -130,6 +130,11 @@ The distillation script expects pre-tokenized data in Megatron's binary format ( See the **[Dataset Preparation README](../dataset/README.md#tokenizing-for-megatron-frameworks)** for full instructions on tokenizing JSONL files and Hugging Face datasets and get the list of output prefixes that you can use for `--data_paths` argument. +If you do not already have a suitable dataset, start with +[data/nemotron-cascade-2-blend.yaml](data/nemotron-cascade-2-blend.yaml). It defines a general-purpose +mixture of SFT data for QAD. Copy it, set the tokenizer for the target model, and adjust the output directory, +token budget, sources, and weights as needed before preparing data. + ### Distillation with Real Data Example usage to distill a 4B student (HF) from an 8B teacher (HF) on 8 GPUs (TP=8, PP=1): diff --git a/.agents/skills/qad/assets/nemotron-cascade-2-blend.yaml b/examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml similarity index 94% rename from .agents/skills/qad/assets/nemotron-cascade-2-blend.yaml rename to examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml index cb28ce5b147..d815a59fecd 100644 --- a/.agents/skills/qad/assets/nemotron-cascade-2-blend.yaml +++ b/examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml @@ -1,3 +1,4 @@ +# Set to the target model's Hugging Face ID or local tokenizer path. tokenizer: output_dir: /data/nemotron-cascade-2-through-150 target_tokens: 2_600_000_000 From 41ec0ac5454ec026eb1217a9741c7486acc1539c Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Wed, 29 Jul 2026 08:17:30 +0000 Subject: [PATCH 09/13] Address QAD review feedback Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 24 ++++--- examples/megatron_bridge/distill.py | 33 +++++++-- .../examples/megatron_bridge/test_distill.py | 68 +++++++++++++++++++ 3 files changed, 109 insertions(+), 16 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index 67863b04b08..6308d9ed599 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -162,8 +162,9 @@ QAD defaults: Keep `train_iters=1000` from the first run so the cosine schedule has a stable horizon. Set `save_interval=50`, `eval_interval=25`, `eval_iters=2`, `exit_interval=150`, and `exit_duration_in_mins=220` for the initial stage. This -preserves checkpoints at iterations 50, 100, and 150. The duration exit is a -checkpointing safety margin, not the training target. +preserves checkpoints at iterations 50, 100, and 150 when the stage reaches its +iteration exit. The duration exit is a checkpointing safety margin, not the +training target. Use one result-bearing Slurm job per stage and put Pyxis container flags on the final `srun`. Fold startup validation into that job; do not submit separate GPU @@ -183,7 +184,11 @@ decreasing smoothed trend, not a decrease at every noisy step: decrease, stop at the checkpoint ending that second window. Preserve the latest known-good checkpoint and diagnose before resuming. -At the initial exit: +Record the actual exit iteration. If the duration exit happens before checkpoint +150 exists, mark the stage incomplete and resume or relaunch the same output +directory. Do not export, benchmark, or label an earlier checkpoint as QAD-150. + +Once checkpoint 150 exists: 1. Confirm the smoothed QAD/KD loss decreased, learning rate is sensible, gradient norm is non-pathological, and checkpoints 50, 100, and 150 exist. @@ -205,9 +210,10 @@ user direction. ## 8. Report evidence -Report exact source revision, commands, Slurm job/account/partition, container, -paths, topology derivation, dataset configs/seed/token budget/materialized -counts, PTQ format, reused evaluation run IDs, checkpoint iterations, -loss/LR/grad trend, scheduler state, and BF16/PTQ/QAD benchmark scores. State -which scores were reused versus newly evaluated. Support success with both -scheduler and log evidence; identify the first real error when a run fails. +Report the exact ModelOpt source revision, commands, Slurm job/account/partition, +container, paths, topology derivation, dataset IDs/configs/seed/token +budget/materialized counts, PTQ format, reused evaluation run IDs, checkpoint +iterations, loss/LR/grad trend, scheduler state, and BF16/PTQ/QAD benchmark +scores. State which scores were reused versus newly evaluated. Support success +with both scheduler and log evidence; identify the first real error when a run +fails. diff --git a/examples/megatron_bridge/distill.py b/examples/megatron_bridge/distill.py index 867ac2c9edd..16dd37d5f8d 100644 --- a/examples/megatron_bridge/distill.py +++ b/examples/megatron_bridge/distill.py @@ -59,6 +59,20 @@ import modelopt.torch.puzzletron.plugins.mbridge # noqa: F401 +def _positive_int(value: str) -> int: + parsed = int(value) + if parsed <= 0: + raise argparse.ArgumentTypeError("must be a positive integer") + return parsed + + +def _nonnegative_int(value: str) -> int: + parsed = int(value) + if parsed < 0: + raise argparse.ArgumentTypeError("must be a non-negative integer") + return parsed + + def get_args(): """Parse command-line arguments.""" parser = argparse.ArgumentParser(description="Distillation for Megatron-Bridge.") @@ -161,25 +175,30 @@ def get_args(): help="Modules to recompute with --recompute_granularity=selective. Defaults to ['core_attn']. " "Allowed: core_attn, mlp, moe, moe_act, layernorm, mla_up_proj, shared_experts.", ) - parser.add_argument("--eval_interval", type=int, default=100, help="Validate every steps") parser.add_argument( - "--eval_iters", type=int, default=32, help="Number of batches per validation stage" + "--eval_interval", type=_positive_int, default=100, help="Validate every steps" + ) + parser.add_argument( + "--eval_iters", + type=_nonnegative_int, + default=32, + help="Number of batches per validation stage; set to 0 to disable validation", ) parser.add_argument( "--save_interval", - type=int, + type=_positive_int, default=None, help="Checkpoint every steps; defaults to --eval_interval", ) parser.add_argument( "--exit_interval", - type=int, + type=_positive_int, default=None, help="Save a checkpoint and exit when the iteration is divisible by this value", ) parser.add_argument( "--exit_duration_in_mins", - type=int, + type=_positive_int, default=None, help="Save a checkpoint and exit after this many minutes", ) @@ -234,8 +253,8 @@ def get_args(): args.student_hf_model = args.student_hf_path if args.checkpoint_keep_last < -1: raise ValueError("--checkpoint_keep_last must be >= -1.") - if args.validate_only and (args.eval_interval <= 0 or args.eval_iters <= 0): - raise ValueError("--validate_only requires --eval_interval > 0 and --eval_iters > 0.") + if args.validate_only and args.eval_iters == 0: + raise ValueError("--validate_only requires --eval_iters > 0.") print_args(args) diff --git a/tests/examples/megatron_bridge/test_distill.py b/tests/examples/megatron_bridge/test_distill.py index 1a5bf9568bf..54872d60eac 100644 --- a/tests/examples/megatron_bridge/test_distill.py +++ b/tests/examples/megatron_bridge/test_distill.py @@ -14,8 +14,11 @@ # limitations under the License. """Tests for prune_minitron.py and distill.py scripts.""" +import importlib +import sys from pathlib import Path +import pytest import torch from _test_utils.examples.run_command import extend_cmd_parts, run_example_command from _test_utils.torch.puzzletron.utils import create_and_save_small_hf_model @@ -29,6 +32,71 @@ from modelopt.torch.puzzletron.anymodel import convert_model +@pytest.mark.parametrize( + ("option", "value", "message"), + [ + ("--eval_interval", "0", "must be a positive integer"), + ("--eval_iters", "-1", "must be a non-negative integer"), + ("--save_interval", "0", "must be a positive integer"), + ("--exit_interval", "-1", "must be a positive integer"), + ("--exit_duration_in_mins", "0", "must be a positive integer"), + ], +) +def test_distill_rejects_invalid_intervals(monkeypatch, capsys, option, value, message): + example_dir = Path(__file__).parents[3] / "examples" / "megatron_bridge" + monkeypatch.syspath_prepend(str(example_dir)) + distill = importlib.import_module("distill") + monkeypatch.setattr( + sys, + "argv", + [ + "distill.py", + "--student_hf_path", + "unused", + "--teacher_hf_path", + "unused", + "--output_dir", + "unused", + "--train_iters", + "1", + "--use_mock_data", + option, + value, + ], + ) + + with pytest.raises(SystemExit, match="2"): + distill.get_args() + + assert message in capsys.readouterr().err + + +def test_distill_allows_zero_eval_iters(monkeypatch): + example_dir = Path(__file__).parents[3] / "examples" / "megatron_bridge" + monkeypatch.syspath_prepend(str(example_dir)) + distill = importlib.import_module("distill") + monkeypatch.setattr( + sys, + "argv", + [ + "distill.py", + "--student_hf_path", + "unused", + "--teacher_hf_path", + "unused", + "--output_dir", + "unused", + "--train_iters", + "1", + "--use_mock_data", + "--eval_iters", + "0", + ], + ) + + assert distill.get_args().eval_iters == 0 + + def test_distill_llm(tmp_path, num_gpus): teacher_hf_path = create_tiny_qwen3_dir(tmp_path, with_tokenizer=True) train_iters = 2 From 49a456d44a89b98988ffd369984235006bd3b109 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 30 Jul 2026 02:44:38 +0000 Subject: [PATCH 10/13] Refine QAD data and resume workflow Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 266 +++++------------- examples/megatron_bridge/README.md | 4 +- .../data/nemotron-cascade-2-blend.yaml | 4 +- .../examples/megatron_bridge/test_distill.py | 15 +- 4 files changed, 88 insertions(+), 201 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index 6308d9ed599..726bc3a3314 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -9,142 +9,61 @@ description: >- # ModelOpt Quantization-Aware Distillation -QAD is expensive. Use it only to recover a material, apples-to-apples PTQ -benchmark gap and only when the user explicitly requests QAD for the target -model or run. Do not infer permission from a quantized checkpoint, evaluation -gap, or recipe-search result. +QAD is expensive. Run it only when the user explicitly authorizes QAD for the +target model or run. A Day-0, PTQ, evaluation, comparison, or recipe-search +request alone is not authorization to start QAD. -## 1. Read the supported workflow +## Follow the supported workflow -Read these sources before constructing commands: +Before constructing commands, read: -- `examples/megatron_bridge/README.md`, especially Post-Training Quantization, - Data Preparation, Quantization Aware Distillation, export, and Slurm usage -- `examples/dataset/MEGATRON_DATA_PREP.md`, especially token-budgeted blends +- `examples/megatron_bridge/README.md`, especially PTQ, data preparation, QAD, + export, and Slurm usage - `examples/megatron_bridge/{quantize.py,distill.py}` via `--help` - -Treat them as the source of truth for mutable flags, containers, model support, -and checkpoint formats. Do not copy their full command lines into new wrappers. -The deprecated `examples/llm_qad` flow is not the default. - -## 2. Reuse or establish the gap - -When QAD follows `evaluation`, `compare-results`, or `quant-recipe-search`, -reuse the validated BF16/PTQ scores and exact benchmark configuration; do not -rerun valid baselines. Record their run IDs, metric, direction, uncertainty, and -acceptable delta. Use `evaluation` and `compare-results` only for missing, -unvalidated, or non-comparable results. Stop if the gap is not material. - -Reuse the preceding PTQ config or recipe to reproduce PTQ with -`examples/megatron_bridge/quantize.py`; do not use the usual HF checkpoint as -the QAD student. Preserve the quantization format, layer selection, calibration -dataset, sample count, sequence length, and seed. Choose Megatron execution -topology separately without changing quantization semantics. If any quantization -setting must change, treat it as a new PTQ candidate and evaluate it before QAD. - -In the exact runtime, confirm `AutoBridge.can_handle()` accepts the model, then -reproduce PTQ with `quantize.py`. Sanity-check the checkpoint's master-rank -`.quant_summary.txt`: every enabled statically calibrated quantizer it lists -must have a finite, positive `amax`. Accept `dynamic` or format-defined `None` -only when the preserved recipe intentionally uses dynamic or MX quantization; -with model parallelism, do not treat this rank-local summary as exhaustive. -Stop on unsupported conversion, failed PTQ, missing ModelOpt state, or -unexpected `amax`. - -## 3. Set up Slurm execution - -Read `skills/common/environment-setup.md` and detect whether the Slurm target is -local or remote. This version of the skill supports Slurm only; stop if the -target is not Slurm. - -- On local Slurm, follow `skills/common/workspace-management.md` and - `skills/common/slurm-setup.md` directly. Do not add SSH or sync steps. -- On remote Slurm, also follow `skills/common/remote-execution.md`: establish - its persistent SSH session, create matching local and remote session/model - workspaces, and sync the source plus the two-script Slurm wrapper/inner runner. - -Use the common account, partition, registry-auth, submission, and monitoring -procedures in either case. - -Keep the reused BF16/PTQ run references and configs with the reproduced PTQ -Megatron checkpoint, QAD checkpoints, data, logs, exports, and new benchmark -results in the same session/model workspace. QAD must start from the checkpoint -produced by `examples/megatron_bridge/quantize.py` because it carries the -ModelOpt state; an exported HF PTQ checkpoint is not a substitute. - -## 4. Choose topology explicitly - -Inspect model parameter count and architecture, sequence length, GPU count/type -and memory, then choose the smallest topology that fits both student and teacher: - -1. Choose node count from model-state plus activation memory; do not start with - an oversized multi-node topology. -2. Use TP only when model-state/GEMM size warrants it, and preserve attention - head and hidden-size divisibility. -3. Use CP to distribute the 32K context when activation/attention memory is the - constraint. Prefer CP before TP for a small dense model at long context. -4. Keep EP=1 for dense models. For MoE, choose EP from expert count, memory, and - available ranks, and inspect the current `distill.py` value for expert tensor - parallelism (ETP); do not assume ETP equals TP. -5. Add PP only when layer/model state still does not fit. - -For MoE, Megatron folds two overlapping meshes onto the same ranks within each -PP stage; EP/ETP do not multiply the dense TP/CP mesh: - -- Attention DP: `DP = world_size / (TP * PP * CP)` -- Expert DP: `EDP = world_size / (ETP * EP * PP)` - -Require both divisions to be integral, `num_experts % EP == 0`, and -`GBS % (MBS * DP) == 0`. Record nodes, GPUs/node, TP/PP/CP/EP/ETP/DP/EDP, MBS, -GPU memory, and why each non-one dimension is needed. - -Example only: for the requested one-node, eight-H100 Qwen3-0.6B validation at -32K, TP=1, PP=1, CP=4, EP=1, DP=2, MBS=1 is a reasonable starting point. This -is validation evidence, not a default topology; derive a fresh topology for -every target model and cluster. - -## 5. Prepare only the next data tranche - -Copy `examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml` into the -session workspace, replace the tokenizer placeholder with the target model's -Hugging Face ID or local path, and set the output directory and token budget. -Run: - -```bash -python -m modelopt.torch.utils.plugins.prepare_megatron_data_blend \ - --config -``` - -The utility streams `nvidia/Nemotron-Cascade-2-SFT-Data`, applies a deterministic -approximate buffer shuffle, and stops at the token budget. Never snapshot or -materialize the full dataset. For the initial step-150 gate, the default -2.6-billion-token budget covers -`150 * 512 * 32768` tokens plus a small margin. Use a much smaller explicit -budget for workflow validation. At very small budgets, use a representative -subset of source configs rather than giving a rare config too few documents -for Megatron's train/validation split. - -Pass the generated `data_blend.txt` entries as `distill.py --data_paths`. -The supported real-data `GPTDatasetConfig` shuffles documents and concatenates -them into fixed-length 32K samples, so sequences are packed instead of padded. -It deterministically holds out 1% of those shuffled documents for validation -with `split="99,1,0"`; bounded materialization uses shuffle seed 42, while the -training seed controls ordering within the Megatron splits. Do not download a -duplicate validation copy. With the defaults, two GBS-512 32K validation -batches consume 33.6M tokens from the nominal 26M-token holdout, so about 1.3 -passes per validation event is accepted. Recalculate this ratio when changing -GBS, sequence length, or `eval_iters`, and increase the token budget rather than -allowing substantially more repetition. Treat this loss as a training-health -signal; the independent benchmark remains the recovery gate. Do not use mock -data as training evidence. - -## 6. Reproduce PTQ, then run staged QAD - -Use the verified Megatron PTQ checkpoint from step 2. Reuse a valid preceding -PTQ evaluation; run a new PTQ evaluation only when the baseline is missing, -invalid, or non-comparable, or a quantization setting changed. Follow the -current Megatron Bridge README for QAD, resume, and export commands. Apply these -QAD defaults: +- `skills/common/{environment-setup,workspace-management,slurm-setup}.md`; also + `skills/common/remote-execution.md` for remote Slurm + +Treat the example README and `--help` output as authoritative for mutable flags, +commands, containers, and checkpoint formats. This skill supports Slurm only. + +## Execute in this order + +1. **Confirm the gap.** Reuse only validated, comparable BF16/PTQ results and + the exact benchmark configuration from preceding evaluation or recipe + search; run missing, invalid, or non-comparable baselines. Set the recovery + target to the user's acceptable BF16 delta, or the benchmark-noise envelope + if none is supplied. Stop if the initial gap already meets that target. +2. **Reproduce PTQ and verify compatibility.** In the target runtime, require + `AutoBridge.can_handle()` for the target model and PTQ through `quantize.py` + to succeed while preserving the exact preceding PTQ config or recipe: + format, layer selection, calibration data/count, sequence length, and seed. + A changed quantization setting is a new PTQ candidate and must be evaluated + before QAD. In the master-rank `.quant_summary.txt`, require finite positive + `amax` for enabled static quantizers; accept `dynamic`/format-defined `None` + only when the recipe intends it. Treat the summary as rank-local under model + parallelism. +3. **Choose topology explicitly.** Derive the smallest fitting node count and + TP/PP/CP/EP/ETP from student and teacher architecture, 32K activation memory, + and available GPU memory. Prefer CP before TP for small long-context models; + keep EP=1 for dense models. For MoE require: + + - `DP = world_size / (TP * PP * CP)` + - `EDP = world_size / (ETP * EP * PP)` + - integral DP/EDP, `num_experts % EP == 0`, and + `GBS % (MBS * DP) == 0` + +4. **Prepare the full capped dataset once.** Copy + `examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml`, set the target + tokenizer and workspace path, and materialize it before training. Use the + generated random sample for packed 32K sequences and its 1% validation + holdout; do not download separate validation data or use mock data as + evidence. +5. **Run staged QAD.** Use one result-bearing Slurm job per stage and fold startup + validation into it; do not submit separate GPU preflight jobs. Use the + defaults below and evaluate checkpoint 150 before deciding whether to + continue. + +## Default training policy | Setting | Default | | --- | --- | @@ -154,66 +73,33 @@ QAD defaults: | Training cap | 1000 iterations | | Global batch size | 512 | | Dataset | `nvidia/Nemotron-Cascade-2-SFT-Data` | +| Materialized token budget | 17.3B, prepared once before training | | Validation | every 25 iterations; deterministic 1% holdout; 2 batches | | Checkpoint interval | 50 iterations | | Initial benchmark/exit | iteration 150 | | Slurm duration exit | 220 minutes for a 4-hour allocation | -Keep `train_iters=1000` from the first run so the cosine schedule has a stable -horizon. Set `save_interval=50`, `eval_interval=25`, `eval_iters=2`, -`exit_interval=150`, and `exit_duration_in_mins=220` for the initial stage. This -preserves checkpoints at iterations 50, 100, and 150 when the stage reaches its -iteration exit. The duration exit is a checkpointing safety margin, not the -training target. - -Use one result-bearing Slurm job per stage and put Pyxis container flags on the -final `srun`. Fold startup validation into that job; do not submit separate GPU -preflight or smoke jobs. - -## 7. Monitor loss, then benchmark - -Monitor the running log instead of waiting for the stage to finish. Record total -or logits-distillation loss every log interval. The expected signal is a -decreasing smoothed trend, not a decrease at every noisy step: - -1. From iteration 100 onward, compare the median loss in each 50-step window - with the preceding window. -2. Treat non-finite loss, NaN iterations, repeated skipped iterations, or a - sharp sustained increase as immediate failure. -3. Flag one flat or rising window. If a second consecutive window also fails to - decrease, stop at the checkpoint ending that second window. Preserve the - latest known-good checkpoint and diagnose before resuming. - -Record the actual exit iteration. If the duration exit happens before checkpoint -150 exists, mark the stage incomplete and resume or relaunch the same output -directory. Do not export, benchmark, or label an earlier checkpoint as QAD-150. - -Once checkpoint 150 exists: - -1. Confirm the smoothed QAD/KD loss decreased, learning rate is sensible, - gradient norm is non-pathological, and checkpoints 50, 100, and 150 exist. -2. Export that QAD checkpoint with the current quantized Megatron exporter. -3. Evaluate QAD-150 with the exact step-2 benchmark configuration. Reuse - validated, comparable BF16/PTQ results; run only missing, invalid, or - non-comparable baselines. -4. Calculate the original gap and recovery. For higher-is-better metrics: - `gap = BF16 - PTQ`, `recovered = QAD - PTQ`, and - `recovery_fraction = recovered / gap`. - -Continue only if benchmark recovery is positive beyond run noise and loss is -stable. Choose the next absolute checkpoint from the evidence (for example 300, -500, 750, or 1000), resume the same output directory with `train_iters=1000`, -retain `eval_interval=25`, `eval_iters=2`, and `save_interval=50`, and set only -`exit_interval` to that target. Stop on recovered gap, plateau, regression, -divergence, or iteration 1000. Never raise the 1000-step cap without explicit -user direction. - -## 8. Report evidence - -Report the exact ModelOpt source revision, commands, Slurm job/account/partition, -container, paths, topology derivation, dataset IDs/configs/seed/token -budget/materialized counts, PTQ format, reused evaluation run IDs, checkpoint -iterations, loss/LR/grad trend, scheduler state, and BF16/PTQ/QAD benchmark -scores. State which scores were reused versus newly evaluated. Support success -with both scheduler and log evidence; identify the first real error when a run -fails. +Keep `train_iters=1000` from the first launch. A duration exit before checkpoint +150 is incomplete: resume the same run and do not label or benchmark an earlier +checkpoint as QAD-150. + +Monitor smoothed QAD/KD loss, learning rate, and gradient norm. Stop on non-finite +loss, repeated skipped iterations, or a sustained spike. Starting at iteration +100, stop and diagnose if two consecutive 50-step windows fail to lower median +loss. + +Export and evaluate checkpoint 150 with the exact BF16/PTQ benchmark +configuration. Stop if it meets the recovery target. Otherwise continue only +when partial recovery is positive beyond run noise and training is healthy. +Choose later absolute checkpoints from that evidence. + +Resume the same Megatron output directory without changing prepared data, data +paths/cache, seed, checkpoint lineage, optimizer, scheduler, iteration, or +consumed-sample state. Among Megatron training arguments, change only the +absolute `exit_interval`; keep topology unchanged, never restart from PTQ, and +never reset training progress. Stop when the recovery target is met, recovery +plateaus or regresses, training diverges, or iteration 1000 is reached. + +Report exact revisions and commands; Slurm job/container/topology; sampled data +configuration and counts; PTQ recipe; checkpoint, loss, optimizer, and scheduler +state; and comparable BF16/PTQ/QAD results. diff --git a/examples/megatron_bridge/README.md b/examples/megatron_bridge/README.md index 981337a3285..5618234fddb 100644 --- a/examples/megatron_bridge/README.md +++ b/examples/megatron_bridge/README.md @@ -133,7 +133,9 @@ for full instructions on tokenizing JSONL files and Hugging Face datasets and ge If you do not already have a suitable dataset, start with [data/nemotron-cascade-2-blend.yaml](data/nemotron-cascade-2-blend.yaml). It defines a general-purpose mixture of SFT data for QAD. Copy it, set the tokenizer for the target model, and adjust the output directory, -token budget, sources, and weights as needed before preparing data. +sources, and weights as needed before preparing data. Its default 17.3-billion-token budget covers 1000 +iterations at global batch size 512 and sequence length 32768, including a 1% validation holdout and margin. +Recalculate the budget when changing those settings, and keep the prepared data unchanged when resuming. ### Distillation with Real Data diff --git a/examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml b/examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml index d815a59fecd..ca4afdd4bdb 100644 --- a/examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml +++ b/examples/megatron_bridge/data/nemotron-cascade-2-blend.yaml @@ -1,7 +1,7 @@ # Set to the target model's Hugging Face ID or local tokenizer path. tokenizer: -output_dir: /data/nemotron-cascade-2-through-150 -target_tokens: 2_600_000_000 +output_dir: /data/nemotron-cascade-2-through-1000 +target_tokens: 17_300_000_000 sources: - hf_dataset: nvidia/Nemotron-Cascade-2-SFT-Data config: math diff --git a/tests/examples/megatron_bridge/test_distill.py b/tests/examples/megatron_bridge/test_distill.py index 54872d60eac..7922e710ebe 100644 --- a/tests/examples/megatron_bridge/test_distill.py +++ b/tests/examples/megatron_bridge/test_distill.py @@ -14,7 +14,6 @@ # limitations under the License. """Tests for prune_minitron.py and distill.py scripts.""" -import importlib import sys from pathlib import Path @@ -31,6 +30,11 @@ from modelopt.torch.puzzletron.anymodel import convert_model +# distill.py imports sibling example modules by filename, so make that directory importable first. +sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "examples" / "megatron_bridge")) + +import distill + @pytest.mark.parametrize( ("option", "value", "message"), @@ -43,9 +47,6 @@ ], ) def test_distill_rejects_invalid_intervals(monkeypatch, capsys, option, value, message): - example_dir = Path(__file__).parents[3] / "examples" / "megatron_bridge" - monkeypatch.syspath_prepend(str(example_dir)) - distill = importlib.import_module("distill") monkeypatch.setattr( sys, "argv", @@ -65,16 +66,14 @@ def test_distill_rejects_invalid_intervals(monkeypatch, capsys, option, value, m ], ) - with pytest.raises(SystemExit, match="2"): + with pytest.raises(SystemExit) as exc_info: distill.get_args() + assert exc_info.value.code == 2 assert message in capsys.readouterr().err def test_distill_allows_zero_eval_iters(monkeypatch): - example_dir = Path(__file__).parents[3] / "examples" / "megatron_bridge" - monkeypatch.syspath_prepend(str(example_dir)) - distill = importlib.import_module("distill") monkeypatch.setattr( sys, "argv", From 23e83d71fba22b46afd43ebccfe6552c87af2527 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 30 Jul 2026 03:42:56 +0000 Subject: [PATCH 11/13] Refine QAD recovery evaluation policy Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 69 +++++++++++++++++++++++-------------- 1 file changed, 43 insertions(+), 26 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index 726bc3a3314..a5dcce64335 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -58,10 +58,10 @@ commands, containers, and checkpoint formats. This skill supports Slurm only. generated random sample for packed 32K sequences and its 1% validation holdout; do not download separate validation data or use mock data as evidence. -5. **Run staged QAD.** Use one result-bearing Slurm job per stage and fold startup - validation into it; do not submit separate GPU preflight jobs. Use the - defaults below and evaluate checkpoint 150 before deciding whether to - continue. +5. **Run and monitor QAD.** Submit one QAD training job and fold startup + validation into it; do not submit separate GPU preflight jobs or split + training into iteration stages. Let training continue while evaluating saved + checkpoints, and cancel the training job when a stop condition below is met. ## Default training policy @@ -74,31 +74,48 @@ commands, containers, and checkpoint formats. This skill supports Slurm only. | Global batch size | 512 | | Dataset | `nvidia/Nemotron-Cascade-2-SFT-Data` | | Materialized token budget | 17.3B, prepared once before training | -| Validation | every 25 iterations; deterministic 1% holdout; 2 batches | +| Training validation | every 25 iterations; deterministic 1% holdout; 2 batches | | Checkpoint interval | 50 iterations | -| Initial benchmark/exit | iteration 150 | +| Loss logging | every 10 iterations | +| Recovery benchmark | 150, then every 100 iterations while training runs | | Slurm duration exit | 220 minutes for a 4-hour allocation | -Keep `train_iters=1000` from the first launch. A duration exit before checkpoint -150 is incomplete: resume the same run and do not label or benchmark an earlier -checkpoint as QAD-150. - -Monitor smoothed QAD/KD loss, learning rate, and gradient norm. Stop on non-finite -loss, repeated skipped iterations, or a sustained spike. Starting at iteration -100, stop and diagnose if two consecutive 50-step windows fail to lower median -loss. - -Export and evaluate checkpoint 150 with the exact BF16/PTQ benchmark -configuration. Stop if it meets the recovery target. Otherwise continue only -when partial recovery is positive beyond run noise and training is healthy. -Choose later absolute checkpoints from that evidence. - -Resume the same Megatron output directory without changing prepared data, data -paths/cache, seed, checkpoint lineage, optimizer, scheduler, iteration, or -consumed-sample state. Among Megatron training arguments, change only the -absolute `exit_interval`; keep topology unchanged, never restart from PTQ, and -never reset training progress. Stop when the recovery target is met, recovery -plateaus or regresses, training diverges, or iteration 1000 is reached. +Keep `train_iters=1000` from the first launch and do not set `exit_interval`. +Benchmark saved checkpoints 150, 250, 350, and so on while the training job +continues. If `exit_duration_in_mins` ends the allocation first, resume the same +run. + +Monitor smoothed QAD/KD loss, learning rate, and gradient norm. Cancel immediately +on non-finite loss, repeated skipped iterations, or a sustained spike; preserve +the latest complete checkpoint and do not wait for another save. With +`log_interval=10`, require the loss aggregate reported at iteration 50 +(iterations 41–50) to be lower than the one at iteration 10 (iterations 1–10); +otherwise diagnose and cancel under the evidence-driven rule below. + +At each recovery checkpoint, first export and evaluate only the one to three +benchmarks with the largest measured PTQ drops, using their exact BF16/PTQ +configurations. At checkpoint 150, compare them with PTQ; at 250, 350, and later, +compare them with the prior QAD checkpoint. If the targeted set improves beyond +run noise, run the remaining original PTQ benchmark suite at that same +checkpoint. Otherwise do not run the full suite. Declare the recovery target +met only from a full-suite result at that checkpoint. Wait until a recovery +checkpoint is fully committed before export; never read one still being written. + +For normal duration resumes, use the same Megatron output directory without +changing prepared data, data paths/cache, seed, topology, checkpoint lineage, +optimizer, scheduler, iteration, or consumed-sample state; never restart from +PTQ or reset training progress. If emergency cancellation occurs before the +first complete QAD checkpoint, report that no resumable QAD state exists and +diagnose before relaunching from PTQ. + +Cancel the training job when the recovery target is met. Otherwise continue +while training is healthy and recovery has not regressed beyond run noise. At +each recovery checkpoint, compare the median of the five latest 10-step loss +aggregates with the preceding five. If benchmark change is flat within run +noise, continue only when the latest loss median is lower; otherwise cancel. +Cancel on recovery regression beyond run noise. For an evidence-driven +cancellation, wait only for an in-progress scheduled save to commit; do not +continue to create a future checkpoint. Never train past iteration 1000. Report exact revisions and commands; Slurm job/container/topology; sampled data configuration and counts; PTQ recipe; checkpoint, loss, optimizer, and scheduler From b2188d4a5976e4a0d3a28a175691e099ccb0679b Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 30 Jul 2026 04:16:34 +0000 Subject: [PATCH 12/13] Clarify QAD scheduling and baseline comparisons Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index a5dcce64335..8a7eec91da5 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -58,10 +58,10 @@ commands, containers, and checkpoint formats. This skill supports Slurm only. generated random sample for packed 32K sequences and its 1% validation holdout; do not download separate validation data or use mock data as evidence. -5. **Run and monitor QAD.** Submit one QAD training job and fold startup - validation into it; do not submit separate GPU preflight jobs or split - training into iteration stages. Let training continue while evaluating saved - checkpoints, and cancel the training job when a stop condition below is met. +5. **Run and monitor QAD.** Run one QAD training job at a time and fold startup + validation into it; do not submit separate GPU preflight jobs or split at + recovery iterations. Let training continue while evaluating saved + checkpoints, and cancel it when a stop condition below is met. ## Default training policy @@ -82,8 +82,11 @@ commands, containers, and checkpoint formats. This skill supports Slurm only. Keep `train_iters=1000` from the first launch and do not set `exit_interval`. Benchmark saved checkpoints 150, 250, 350, and so on while the training job -continues. If `exit_duration_in_mins` ends the allocation first, resume the same -run. +continues. After a representative allocation, estimate jobs to the next +checkpoint as `ceil(remaining_iters / observed_iters_per_allocation)`, using +end-to-end progress that includes startup, validation, and saves. Before then, +treat the early seconds-per-iteration estimate as a lower bound and provision +one additional allocation. Resume after a duration exit under the rule below. Monitor smoothed QAD/KD loss, learning rate, and gradient norm. Cancel immediately on non-finite loss, repeated skipped iterations, or a sustained spike; preserve @@ -94,11 +97,12 @@ otherwise diagnose and cancel under the evidence-driven rule below. At each recovery checkpoint, first export and evaluate only the one to three benchmarks with the largest measured PTQ drops, using their exact BF16/PTQ -configurations. At checkpoint 150, compare them with PTQ; at 250, 350, and later, -compare them with the prior QAD checkpoint. If the targeted set improves beyond -run noise, run the remaining original PTQ benchmark suite at that same -checkpoint. Otherwise do not run the full suite. Declare the recovery target -met only from a full-suite result at that checkpoint. Wait until a recovery +configurations. At checkpoint 150, new recovery means QAD-150 over PTQ; later it +means improvement over the prior evaluated QAD checkpoint. In all cases, judge +BF16-gap recovery against the fixed BF16/PTQ baselines. When the targeted set +shows new recovery beyond run noise, run the remaining original PTQ benchmark +suite at that same checkpoint; otherwise do not. Declare the recovery target met +only from a full-suite result at that checkpoint. Wait until a recovery checkpoint is fully committed before export; never read one still being written. For normal duration resumes, use the same Megatron output directory without From 78501168312efa7fc4d8cd552ae21fb38cd9e982 Mon Sep 17 00:00:00 2001 From: Meng Xin Date: Thu, 30 Jul 2026 04:43:32 +0000 Subject: [PATCH 13/13] Streamline QAD run policy Signed-off-by: Meng Xin --- .agents/skills/qad/SKILL.md | 72 +++++++++++++------------------------ 1 file changed, 25 insertions(+), 47 deletions(-) diff --git a/.agents/skills/qad/SKILL.md b/.agents/skills/qad/SKILL.md index 8a7eec91da5..da6880e7da6 100644 --- a/.agents/skills/qad/SKILL.md +++ b/.agents/skills/qad/SKILL.md @@ -30,9 +30,8 @@ commands, containers, and checkpoint formats. This skill supports Slurm only. 1. **Confirm the gap.** Reuse only validated, comparable BF16/PTQ results and the exact benchmark configuration from preceding evaluation or recipe - search; run missing, invalid, or non-comparable baselines. Set the recovery - target to the user's acceptable BF16 delta, or the benchmark-noise envelope - if none is supplied. Stop if the initial gap already meets that target. + search; run missing, invalid, or non-comparable baselines. Stop if the PTQ + gap to BF16 is already below 1%. 2. **Reproduce PTQ and verify compatibility.** In the target runtime, require `AutoBridge.can_handle()` for the target model and PTQ through `quantize.py` to succeed while preserving the exact preceding PTQ config or recipe: @@ -80,47 +79,26 @@ commands, containers, and checkpoint formats. This skill supports Slurm only. | Recovery benchmark | 150, then every 100 iterations while training runs | | Slurm duration exit | 220 minutes for a 4-hour allocation | -Keep `train_iters=1000` from the first launch and do not set `exit_interval`. -Benchmark saved checkpoints 150, 250, 350, and so on while the training job -continues. After a representative allocation, estimate jobs to the next -checkpoint as `ceil(remaining_iters / observed_iters_per_allocation)`, using -end-to-end progress that includes startup, validation, and saves. Before then, -treat the early seconds-per-iteration estimate as a lower bound and provision -one additional allocation. Resume after a duration exit under the rule below. - -Monitor smoothed QAD/KD loss, learning rate, and gradient norm. Cancel immediately -on non-finite loss, repeated skipped iterations, or a sustained spike; preserve -the latest complete checkpoint and do not wait for another save. With -`log_interval=10`, require the loss aggregate reported at iteration 50 -(iterations 41–50) to be lower than the one at iteration 10 (iterations 1–10); -otherwise diagnose and cancel under the evidence-driven rule below. - -At each recovery checkpoint, first export and evaluate only the one to three -benchmarks with the largest measured PTQ drops, using their exact BF16/PTQ -configurations. At checkpoint 150, new recovery means QAD-150 over PTQ; later it -means improvement over the prior evaluated QAD checkpoint. In all cases, judge -BF16-gap recovery against the fixed BF16/PTQ baselines. When the targeted set -shows new recovery beyond run noise, run the remaining original PTQ benchmark -suite at that same checkpoint; otherwise do not. Declare the recovery target met -only from a full-suite result at that checkpoint. Wait until a recovery -checkpoint is fully committed before export; never read one still being written. - -For normal duration resumes, use the same Megatron output directory without -changing prepared data, data paths/cache, seed, topology, checkpoint lineage, -optimizer, scheduler, iteration, or consumed-sample state; never restart from -PTQ or reset training progress. If emergency cancellation occurs before the -first complete QAD checkpoint, report that no resumable QAD state exists and -diagnose before relaunching from PTQ. - -Cancel the training job when the recovery target is met. Otherwise continue -while training is healthy and recovery has not regressed beyond run noise. At -each recovery checkpoint, compare the median of the five latest 10-step loss -aggregates with the preceding five. If benchmark change is flat within run -noise, continue only when the latest loss median is lower; otherwise cancel. -Cancel on recovery regression beyond run noise. For an evidence-driven -cancellation, wait only for an in-progress scheduled save to commit; do not -continue to create a future checkpoint. Never train past iteration 1000. - -Report exact revisions and commands; Slurm job/container/topology; sampled data -configuration and counts; PTQ recipe; checkpoint, loss, optimizer, and scheduler -state; and comparable BF16/PTQ/QAD results. +## Run policy + +- Keep `train_iters=1000` and leave `exit_interval` unset. +- From initial step timing, submit only enough sequential jobs to reach + checkpoint 150; never submit through iteration 1000 upfront. At each recovery + checkpoint, submit to the next only after its targeted evaluation and any + triggered full suite, and only if the BF16 gap remains at least 1% and + recovery has neither plateaued nor regressed. +- Give all training jobs the same run-specific job name and + `--dependency=singleton`; record job IDs and, on any stop, cancel pending jobs + before the active job. +- Cancel on non-finite loss, repeated skipped iterations, or a sustained spike. + At iteration 50, require the loss aggregate to be lower than at iteration 10. +- At each recovery checkpoint, first evaluate the one to three benchmarks with + the largest PTQ drops. Run the remaining original PTQ suite at that checkpoint + only after recovery beyond run noise. +- Cancel when the full-suite gap to BF16 is below 1%, benchmark recovery + regresses beyond run noise, or benchmark recovery and loss both plateau. +- After a duration exit, resume the latest QAD checkpoint in the same output + directory with unchanged prepared data paths/cache, seed, topology, optimizer, + scheduler, iteration, and consumed-sample state; do not restart from PTQ. +- Report the PTQ recipe, data sample, Slurm topology, loss/state, checkpoints, + and comparable BF16/PTQ/QAD results.