Why accelerate distributed launches fail, and how its config actually needs to match your hardware
ValueError: You can't train a model that has been loaded with `device_map='auto'` in any distributed mode
Also appears as
- ValueError: DistributedType.MULTI_GPU does not match the number of processes launched
- RuntimeError: Please run `accelerate config` to set up your environment before launching training
Short answer
Accelerate errors during launch almost always mean the saved configuration does not match the current machine's actual GPU count, node count, or distributed type, or that the model was loaded with device_map='auto' inference-style sharding and then also handed to accelerate's training-mode preparation, which are two incompatible placement strategies. Regenerating the config for the current machine, or passing explicit CLI overrides, resolves most cases.
Affects: HuggingFace accelerate 0.20 and later, used standalone or underneath Trainer, DeepSpeed, or PEFT, on single-node multi-GPU and multi-node clusters
Regenerate or explicitly override the accelerate config for this machine
- 1Run accelerate env to print what accelerate currently believes about your environment and compare it against the actual GPU count and node setup.
- 2If the config is stale, rerun accelerate config interactively, or delete the cached default_config.yaml and let it regenerate.
- 3For quick fixes without regenerating the whole config, pass explicit overrides directly on the CLI: accelerate launch --num_processes=N --num_machines=M --multi_gpu train.py.
- 4If you are loading the model with device_map='auto' for inference-style sharding, do not also call accelerator.prepare() on that model for training; pick one placement strategy.
- 5For multi-node runs, regenerate machine_rank and main_process_ip per node rather than copying one node's config file to the others unchanged.
How to confirm this is your problem
- Job launches but immediately errors about distributed type, device_map, or process count mismatch before any training step runs
- The same code and config worked on a different machine but fails here after a change in GPU count or node topology
- Mixed precision behavior (fp16 or bf16) does not match what you expect, or a scaler or dtype error appears
- Copying a working accelerate config to a new node causes that node to hang or error at launch
Root causes and fixes
The saved accelerate config does not match the current machine's actual GPU count, node count, or distributed type
accelerate config writes a YAML file capturing the exact hardware topology you answered questions about at the time: number of GPUs, number of machines, and distributed backend. If that config is later reused on a machine with a different GPU count, or after moving from a single-GPU dev box to a multi-GPU or multi-node cluster, accelerate launches with the wrong topology and errors or misbehaves immediately.
Fix: Rerun accelerate config on the actual machine you are launching from, or bypass the cached config entirely by passing explicit flags like --num_processes and --num_machines directly to accelerate launch.
accelerate env accelerate config
A model loaded with device_map='auto' is then also passed to accelerator.prepare() for distributed training preparation
device_map='auto' is designed for splitting a model across devices for inference or generation, computing its own device assignment based on available memory. Accelerator.prepare() assumes it is responsible for placement (via DDP, FSDP, or DeepSpeed) and wrapping the model accordingly. Combining both means two different systems both believe they control device placement, which raises an explicit error to prevent silent corruption of the model's state.
Fix: Choose one strategy: use device_map='auto' for inference-only workflows without wrapping in accelerator.prepare(), or load the model normally and let accelerator.prepare() handle placement entirely for training.
Mixed precision setting in the accelerate config does not match what the training script or DeepSpeed config expects
accelerate's mixed_precision setting controls whether it wraps the optimizer step with a gradient scaler and how it casts activations. If your training script or an underlying DeepSpeed config independently sets a different precision, the two layers can disagree about whether a loss-scaling step is needed, producing dtype errors or silently degraded numerical behavior.
Fix: Set mixed precision in exactly one place, preferably the accelerate config, and verify with accelerate env that the effective setting matches what your training script assumes.
An accelerate config generated on a single-GPU machine was copied unchanged to a multi-node cluster without updating machine_rank or main_process_ip
Multi-node accelerate configs need a unique machine_rank per node and a shared main_process_ip that every node can reach, values that are meaningless or wrong when copied verbatim from a single-machine config. Every node ending up with the same machine_rank causes a rendezvous conflict instead of a clean multi-node launch.
Fix: Generate the config per node, or template it from a single source with the correct machine_rank and IP substituted for each node, rather than distributing one static file to the whole cluster.
A stale accelerate cache uses deprecated keys from an older accelerate version that the current version does not recognize
accelerate has occasionally renamed or removed config keys across major versions; a config file written by an old version and never regenerated can silently be missing required fields or contain fields the current version ignores, producing confusing startup errors unrelated to your actual hardware.
Fix: After upgrading accelerate, delete and regenerate default_config.yaml rather than assuming an old config remains compatible.
Diagnostic commands
Print accelerate's current understanding of the environment
accelerate env
Compare the reported distributed_type, num_processes, and num_machines against what you actually intend to launch with; any mismatch here is the direct cause of most launch failures.
Inspect the raw saved config file
cat ~/.cache/huggingface/accelerate/default_config.yaml
Check machine_rank, main_process_ip, num_machines, and num_processes values directly, especially on multi-node setups where a copied config is a common mistake.
Test the launch configuration with accelerate's built-in check
accelerate test
Runs a minimal distributed smoke test using the current config; a failure here isolates the problem to accelerate's setup itself, before you spend time debugging your actual training script.
Stopping it from happening again
- Regenerate or explicitly template the accelerate config as part of your deployment pipeline for every new machine, rather than reusing a config across different hardware.
- Keep mixed precision, batch size, and distributed type settings in a single source of truth and document which one owns them.
- Run accelerate test as a pre-flight check in CI or on new cluster nodes before launching a real training job.
- Version-control your accelerate config templates per environment instead of relying on the interactive prompt each time.
When this becomes an architecture problem
If your team is regularly hand-editing accelerate configs per machine and still hitting mismatches, standardize on a templated, version-controlled config generation step in your deployment tooling, or move fully to a DeepSpeed or Kubernetes-native launch mechanism, since manual per-node config management does not scale reliably past a handful of machines.
Frequently asked questions
Do I need to run accelerate config on every machine separately?
You need a config whose values, such as num_processes, num_machines, machine_rank, and main_process_ip, are correct for each machine's role in the job. You can generate it interactively per machine, or template one config and substitute the per-node values programmatically. Copying one machine's finished config unchanged to another with different hardware or role is what typically breaks.
Can I skip accelerate config entirely and just use CLI flags?
Yes. Passing --num_processes, --num_machines, --multi_gpu, and similar flags directly to accelerate launch overrides the cached config for that run, which is often simpler and less error-prone than maintaining config files, especially for straightforward single-node multi-GPU setups.
Why does device_map='auto' conflict with accelerate training specifically?
device_map='auto' is accelerate's own inference-oriented sharding utility, and it computes a device placement it expects to remain unchanged. Accelerator.prepare(), used for training, applies its own placement and wrapping logic. Using both means two parts of the same library disagree about who owns device placement, which accelerate detects and blocks rather than risk incorrect results.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Fine-Tuning GPU-Hours Estimator
Estimate GPU-hours for LoRA, QLoRA, and full fine-tuning on the same model size and dataset, so you can compare method tradeoffs before choosing.
Free ToolMulti-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Free ToolLoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Related problems
Expected all tensors to be on the same device error in multi-GPU code
This error means a tensor was created or moved to a specific device, often cuda:0 or cpu, that does not match the device another tensor in the same operation lives on, which happens most often when leftover manual .to(device) calls from single-GPU code collide with automatic sharding from device_map='auto' or DistributedDataParallel. The fix is to trace exactly which tensor has the wrong device with a quick print of tensor.device, not to guess and add more .to() calls.
DeepSpeed ZeRO configuration errors at training startup
DeepSpeed refuses to start when its config's batch-size fields do not agree with each other, since train_batch_size must equal train_micro_batch_size_per_gpu times gradient_accumulation_steps times world_size exactly. A second common failure comes from enabling ZeRO stage 3 with CPU or NVMe offload on hardware that lacks enough system RAM or fast enough storage, which surfaces as tensor or contiguity errors rather than a clear resource message.
Multi-node training hangs with no error after rendezvous
A multi-node job that hangs with no error almost always means not every rank actually joined the process group: a mismatched world size, a wrong MASTER_ADDR or MASTER_PORT, a firewall blocking the ephemeral ports NCCL negotiates after rendezvous, or a node that silently OOMed are the four most common causes. Because NCCL blocks silently while waiting for missing ranks, there is often no error at all until you manually intervene or hit a long default timeout.
NCCL error during multi-GPU training or inference
An NCCL error during multi-GPU training or inference is almost always a symptom of a rank that crashed, a version mismatch across processes, or bad GPU topology, not a bug in NCCL itself. Enable NCCL_DEBUG=INFO first and read the per-rank logs before touching timeouts or retry logic.
GuideFine-Tuning Failure Modes: What Actually Goes Wrong
Fine-tuning failure modes that actually derail enterprise projects: catastrophic forgetting, eval overfitting, data leakage, and how to catch each one.
GuideMulti-Node LLM Training Infrastructure: Networking and Storage
Which networking has the lowest latency for multi-node LLM training? InfiniBand NDR vs RoCEv2, NVLink inside the node, plus storage and topology guidance.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.
Try it here, free
Automotive Launch Readiness Review Template
A fill-in launch readiness review document covering PPAP, capacity, tooling, logistics and risks, with a decision and approval block.
Keep exploring
Free tools
- Automotive Launch Readiness Review TemplateA fill-in launch readiness review document covering PPAP, capacity, tooling, logistics and risks, with a decision and approval block.
- ERP Master Data AI Cleanup EstimatorTurn record count, error rate, and per-record review time into the hours and dollars saved by using AI to accelerate master data cleanup versus a fully manual review.
- Safe Launch Plan TemplateDocument the added controls, staffing and exit criteria for the first weeks of production on a new automotive program.
ERP fixes and how-tos
- Fix NetSuite CANNOT_CONVERT ErrorCANNOT_CONVERT means SuiteScript could not coerce a value into a field's expected data type. Learn the common setValue and format.parse causes and the fix.
- Error validating SYSPRO online profile: what it means and how to fix itWhy SYSPRO fails to validate an online profile, plus the usual storage and runtime error messages, and a clean order of checks to fix each one.
- Troubleshooting SyteLine MRP and APS regeneration that fails or hangsSyteLine MRP or APS regeneration runs that hang, time out, or fail usually trace to bad BOM/routing data, a locked scheduling engine process, or insufficient app server resources. Here is the diagnostic order.
Guides
- Multi-Node LLM Training Infrastructure: Networking and Storage That Actually ScaleWhich networking has the lowest latency for multi-node LLM training? InfiniBand NDR vs RoCEv2, NVLink inside the node, plus storage and topology guidance.
- Infor LN Print Management: Complete Configuration GuideConfigure Infor LN print management including device setup, output routing, PDF generation, and Document Output Management for enterprise document workflows.
- Configuring Handheld Devices for SyteLine OperationsConfigure handheld scanners and mobile devices for SyteLine operations. Set up Zebra, Honeywell, and Datalogic devices for CloudSuite Industrial warehouse workflows.
Troubleshooting
- Why multi-node training hangs with no error, and the four things that actually cause itMulti-node training usually hangs from a world size mismatch, wrong MASTER_ADDR or MASTER_PORT, a blocked firewall port, or a silently OOM-killed node.
- Why you get 'expected all tensors to be on the same device', and where the mismatched tensor actually comes fromThis device-mismatch error almost always comes from a leftover manual .to(device) call clashing with device_map='auto' sharding. Find the hardcoded device call.