# Why accelerate distributed launches fail, and how its config actually needs to match your hardware > Source: https://netray.co/troubleshoot/accelerate-config-issues > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Accelerate errors during launch almost always mean the saved configuration does not match the current machine's actual GPU count, node count, or distributed type, or that the model was loaded with device_map='auto' inference-style sharding and then also handed to accelerate's training-mode preparation, which are two incompatible placement strategies. Regenerating the config for the current machine, or passing explicit CLI overrides, resolves most cases. ## Error Title HuggingFace Accelerate config mismatch causes wrong distributed launch ## Error Signature ValueError: You can't train a model that has been loaded with `device_map='auto'` in any distributed mode ## Alternate Signatures - ValueError: DistributedType.MULTI_GPU does not match the number of processes launched - RuntimeError: Please run `accelerate config` to set up your environment before launching training ## Category distributed ## Stack - accelerate - pytorch - deepspeed - huggingface ## Affects HuggingFace accelerate 0.20 and later, used standalone or underneath Trainer, DeepSpeed, or PEFT, on single-node multi-GPU and multi-node clusters ## Quick Fix ### Regenerate or explicitly override the accelerate config for this machine **Steps:** - Run accelerate env to print what accelerate currently believes about your environment and compare it against the actual GPU count and node setup. - If the config is stale, rerun accelerate config interactively, or delete the cached default_config.yaml and let it regenerate. - For quick fixes without regenerating the whole config, pass explicit overrides directly on the CLI: accelerate launch --num_processes=N --num_machines=M --multi_gpu train.py. - If you are loading the model with device_map='auto' for inference-style sharding, do not also call accelerator.prepare() on that model for training; pick one placement strategy. - For multi-node runs, regenerate machine_rank and main_process_ip per node rather than copying one node's config file to the others unchanged. ## Symptoms - Job launches but immediately errors about distributed type, device_map, or process count mismatch before any training step runs - The same code and config worked on a different machine but fails here after a change in GPU count or node topology - Mixed precision behavior (fp16 or bf16) does not match what you expect, or a scaler or dtype error appears - Copying a working accelerate config to a new node causes that node to hang or error at launch ## Causes ### The saved accelerate config does not match the current machine's actual GPU count, node count, or distributed type **Likelihood:** most-common **Explanation:** accelerate config writes a YAML file capturing the exact hardware topology you answered questions about at the time: number of GPUs, number of machines, and distributed backend. If that config is later reused on a machine with a different GPU count, or after moving from a single-GPU dev box to a multi-GPU or multi-node cluster, accelerate launches with the wrong topology and errors or misbehaves immediately. **Fix:** Rerun accelerate config on the actual machine you are launching from, or bypass the cached config entirely by passing explicit flags like --num_processes and --num_machines directly to accelerate launch. **Commands:** - accelerate env - accelerate config ### A model loaded with device_map='auto' is then also passed to accelerator.prepare() for distributed training preparation **Likelihood:** common **Explanation:** device_map='auto' is designed for splitting a model across devices for inference or generation, computing its own device assignment based on available memory. Accelerator.prepare() assumes it is responsible for placement (via DDP, FSDP, or DeepSpeed) and wrapping the model accordingly. Combining both means two different systems both believe they control device placement, which raises an explicit error to prevent silent corruption of the model's state. **Fix:** Choose one strategy: use device_map='auto' for inference-only workflows without wrapping in accelerator.prepare(), or load the model normally and let accelerator.prepare() handle placement entirely for training. ### Mixed precision setting in the accelerate config does not match what the training script or DeepSpeed config expects **Likelihood:** common **Explanation:** accelerate's mixed_precision setting controls whether it wraps the optimizer step with a gradient scaler and how it casts activations. If your training script or an underlying DeepSpeed config independently sets a different precision, the two layers can disagree about whether a loss-scaling step is needed, producing dtype errors or silently degraded numerical behavior. **Fix:** Set mixed precision in exactly one place, preferably the accelerate config, and verify with accelerate env that the effective setting matches what your training script assumes. ### An accelerate config generated on a single-GPU machine was copied unchanged to a multi-node cluster without updating machine_rank or main_process_ip **Likelihood:** occasional **Explanation:** Multi-node accelerate configs need a unique machine_rank per node and a shared main_process_ip that every node can reach, values that are meaningless or wrong when copied verbatim from a single-machine config. Every node ending up with the same machine_rank causes a rendezvous conflict instead of a clean multi-node launch. **Fix:** Generate the config per node, or template it from a single source with the correct machine_rank and IP substituted for each node, rather than distributing one static file to the whole cluster. ### A stale accelerate cache uses deprecated keys from an older accelerate version that the current version does not recognize **Likelihood:** rare **Explanation:** accelerate has occasionally renamed or removed config keys across major versions; a config file written by an old version and never regenerated can silently be missing required fields or contain fields the current version ignores, producing confusing startup errors unrelated to your actual hardware. **Fix:** After upgrading accelerate, delete and regenerate default_config.yaml rather than assuming an old config remains compatible. ## Diagnostics ### Print accelerate's current understanding of the environment **Command:** accelerate env **Interpretation:** Compare the reported distributed_type, num_processes, and num_machines against what you actually intend to launch with; any mismatch here is the direct cause of most launch failures. ### Inspect the raw saved config file **Command:** cat ~/.cache/huggingface/accelerate/default_config.yaml **Interpretation:** Check machine_rank, main_process_ip, num_machines, and num_processes values directly, especially on multi-node setups where a copied config is a common mistake. ### Test the launch configuration with accelerate's built-in check **Command:** accelerate test **Interpretation:** Runs a minimal distributed smoke test using the current config; a failure here isolates the problem to accelerate's setup itself, before you spend time debugging your actual training script. ## Prevention - Regenerate or explicitly template the accelerate config as part of your deployment pipeline for every new machine, rather than reusing a config across different hardware. - Keep mixed precision, batch size, and distributed type settings in a single source of truth and document which one owns them. - Run accelerate test as a pre-flight check in CI or on new cluster nodes before launching a real training job. - Version-control your accelerate config templates per environment instead of relying on the interactive prompt each time. ## When To Escalate If your team is regularly hand-editing accelerate configs per machine and still hitting mismatches, standardize on a templated, version-controlled config generation step in your deployment tooling, or move fully to a DeepSpeed or Kubernetes-native launch mechanism, since manual per-node config management does not scale reliably past a handful of machines. ## Faq ### Do I need to run accelerate config on every machine separately? **Answer:** You need a config whose values, such as num_processes, num_machines, machine_rank, and main_process_ip, are correct for each machine's role in the job. You can generate it interactively per machine, or template one config and substitute the per-node values programmatically. Copying one machine's finished config unchanged to another with different hardware or role is what typically breaks. ### Can I skip accelerate config entirely and just use CLI flags? **Answer:** Yes. Passing --num_processes, --num_machines, --multi_gpu, and similar flags directly to accelerate launch overrides the cached config for that run, which is often simpler and less error-prone than maintaining config files, especially for straightforward single-node multi-GPU setups. ### Why does device_map='auto' conflict with accelerate training specifically? **Answer:** device_map='auto' is accelerate's own inference-oriented sharding utility, and it computes a device placement it expects to remain unchanged. Accelerator.prepare(), used for training, applies its own placement and wrapping logic. Using both means two parts of the same library disagree about who owns device placement, which accelerate detects and blocks rather than risk incorrect results. ## Search terms accelerate config error, accelerate launch distributed type mismatch, device_map auto distributed training error, accelerate env mismatch, accelerate multi node config, huggingface accelerate launch fails --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json