Turkey (Turkish)
Germany (German)
Worldwide (English)
Accelerate requires launching the command on all nodes with correct machine_rank values.
Halkali merkez neighborhood fatih st ozgur apt no 46 , Kucukcekmece , Istanbul , 34303 , TR
Diagnose watchdog collective timeouts, rank hangs, barriers, machine_rank, main_process_ip, NCCL networking, DDP, DeepSpeed ZeRO and FSDP issues.
Accelerate requires launching the command on all nodes with correct machine_rank values.
ZeRO-1 shards optimizer states, ZeRO-2 adds gradients, ZeRO-3 adds parameters; deeper sharding increases communication cost.
FSDP FULL_SHARD maps conceptually to ZeRO-3, while SHARD_GRAD_OP maps to ZeRO-2.
If one rank crashes earlier with CUDA OOM, data-loader exception or shape mismatch, other ranks may wait in a collective until the watchdog times out. Search all rank logs for the earliest exception before the timeout.
On a single node, PCIe/NVLink topology and process mapping dominate. Multi-node adds IP/port, firewall, routing, MTU and interface selection. Reduce the problem to a minimal two-GPU single-node test when possible.
`num_machines`, `num_processes`, `machine_rank`, `main_process_ip`, `main_process_port` and `same_network` must agree. Accelerate recommends the rank-0 intranet IP for lower latency. Change machine_rank per node.
accelerate envaccelerate config show 2>/dev/null || truehostname -Invidia-smi topo -mAccelerate explicitly requires launching on every node, not only the master. If one node never starts, others can appear to hang at rendezvous/barrier.
PyTorch ProcessGroupNCCL exposes environment variables for watchdog and async error handling. Enable detailed logging during reproduction and identify which collective and rank stalled.
export NCCL_DEBUG=INFOexport TORCH_DISTRIBUTED_DEBUG=DETAILexport TORCH_NCCL_ASYNC_ERROR_HANDLING=1On multi-NIC hosts, NCCL may pick an unsuitable public, Docker or management interface, causing poor bandwidth or timeouts. Verify routes, interface IPs and node-to-node bandwidth/latency, plus the rendezvous port.
DDP keeps a model replica per GPU and all-reduces gradients. FSDP/ZeRO shard more state to save memory but increase communication. If a small model already fits per GPU, ZeRO-3 may reduce throughput unnecessarily.
If one rank finishes earlier, others may wait forever in collectives. Verify distributed sampler behavior, drop_last, filtering and exceptions so ranks reach synchronization points consistently.
Debugging 8 GPUs across multiple nodes directly is hard. Validate single GPU, two-GPU DDP, full single node and only then multi-node. This sharply narrows the configuration search space.
| Hang point | First suspect | Check |
|---|---|---|
| Launch/rendezvous | IP/port/rank | Accelerate config |
| First all-reduce | NCCL/interface/topology | NCCL debug + topo |
| After some steps | One-rank exception/OOM | All rank logs |
| Epoch end | Uneven dataloader | Sampler/drop_last |
Do not blindly upgrade packages in a working training environment. Record GPU, driver, CUDA/PyTorch runtime, Transformers, Accelerate, PEFT, TRL, bitsandbytes/Diffusers, model revision and dataset fingerprint for every run. Reproduce minimally before changing production training.
No. An earlier OOM or exception on one rank can make others time out in collectives.
No. It saves more memory but increases communication due to parameter sharding.
Evaluate model size, precision, context, batch, LoRA/QLoRA or full fine-tuning and multi-GPU needs together instead of choosing by GPU name alone.