Skip to content

[docs] distributed training - #44775

Open
stevhliu wants to merge 9 commits into
huggingface:mainfrom
stevhliu:nd-parallelism
Open

[docs] distributed training#44775
stevhliu wants to merge 9 commits into
huggingface:mainfrom
stevhliu:nd-parallelism

Conversation

@stevhliu

@stevhliu stevhliu commented Mar 16, 2026

Copy link
Copy Markdown
Member

CI

Updates the distributed training docs with recent changes to the DistributedConfig API.

  • Replace the deprecated tp_plan and tp_size in from_pretrained with distributed_config=DistributedConfig(tp_size=N)
  • Add new DistributedConfig doc to cover TP, FSDP2, sharding plans, save_pretrained(distributed_checkpoint=True), and optimizer save/load
  • Add new overview doc for how to choose a strategy and provide a better entry point to each individual parallelism strategy. Also differentiates between using Trainer/Accelerate and load-time DistributedConfig
  • Extends debugging doc with other issues a user may potentially run into (taken from source) instead of mostly DeepSpeed
  • Minor autolink formatting (use # instead of .)

Many of the changed files are due to formatting. The main ones to focus on are:

  • distributed_config.md
  • distributed_overview.md
  • perf_train_gpu_many.md

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 29037662996:1
Result: success | Jobs: 2 | Tests: 24 | Failures: 0 | Duration: 1m 57s

@stevhliu
stevhliu marked this pull request as ready for review July 29, 2026 23:43
@stevhliu
stevhliu requested review from 3outeille and SunMarc July 29, 2026 23:43
@stevhliu stevhliu changed the title [docs] n-d parallelism [docs] distributed training Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants