# Why DeepSpeed ZeRO throws config errors at startup, and how offload settings actually trade off > Source: https://netray.co/troubleshoot/deepspeed-zero-config-errors > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer DeepSpeed refuses to start when its config's batch-size fields do not agree with each other, since train_batch_size must equal train_micro_batch_size_per_gpu times gradient_accumulation_steps times world_size exactly. A second common failure comes from enabling ZeRO stage 3 with CPU or NVMe offload on hardware that lacks enough system RAM or fast enough storage, which surfaces as tensor or contiguity errors rather than a clear resource message. ## Error Title DeepSpeed ZeRO configuration errors at training startup ## Error Signature AssertionError: Check batch related parameters. train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size ## Alternate Signatures - ValueError: Please specify either train_batch_size or train_micro_batch_size_per_gpu in the DeepSpeed config - RuntimeError: Tensors must be CUDA and dense when using ZeRO stage 3 with offload ## Category distributed ## Stack - deepspeed - pytorch - accelerate - cuda ## Affects DeepSpeed ZeRO stages 1 through 3, with or without HuggingFace accelerate or Trainer integration, any multi-GPU fine-tuning or pretraining job ## Quick Fix ### Fix the batch-size math before touching ZeRO stage or offload settings **Steps:** Open your DeepSpeed JSON config and set only two of train_batch_size, train_micro_batch_size_per_gpu, and gradient_accumulation_steps, leaving the third as 'auto' so DeepSpeed derives it correctly. Confirm world_size (total GPUs across all nodes) matches what you expect, since train_batch_size implicitly depends on it. If using HuggingFace Trainer or accelerate alongside DeepSpeed, remove batch size settings from the Trainer or accelerate config and let the DeepSpeed config be the single source of truth, or vice versa, but never both. Start with ZeRO stage 2 and no offload to confirm the model and data pipeline work, then move to stage 3 or add offload one change at a time. If enabling offload, verify system RAM (for CPU offload) or NVMe read and write speed (for NVMe offload) is sufficient before assuming a resulting error is a config bug. ## Symptoms - Training fails immediately at DeepSpeed initialization with an assertion about batch size parameters - Job runs fine at ZeRO stage 2 but fails or hangs the moment stage 3 or offload is enabled - Error messages reference contiguous tensors, dense tensors, or parameter partitioning rather than an obvious out-of-memory message - Training starts but is far slower than expected once offload is enabled, with heavy CPU or disk activity ## Causes ### The DeepSpeed config specifies a train_batch_size that is mathematically inconsistent with the other batch fields **Likelihood:** most-common **Explanation:** DeepSpeed treats train_batch_size, train_micro_batch_size_per_gpu, and gradient_accumulation_steps as a strict equation, not independent settings, because it needs to know exactly how many gradient accumulation steps to run per optimizer step across the whole cluster. If the numbers do not multiply out correctly, most often because world_size changed without updating the other fields, DeepSpeed refuses to start rather than guess. **Fix:** Set train_micro_batch_size_per_gpu and gradient_accumulation_steps explicitly, and set train_batch_size to 'auto' so the values can never drift out of sync when your GPU count changes. ### ZeRO stage 3 is enabled but a required setting (contiguous_gradients, overlap_comm, or reduce_bucket_size) is misconfigured **Likelihood:** common **Explanation:** Stage 3 partitions parameters, gradients, and optimizer state across all ranks and reconstructs full parameters just-in-time during forward and backward passes. This reconstruction depends on contiguous memory layout assumptions; a mismatched or overly aggressive bucket size setting for a model's specific layer shapes can produce a tensor contiguity error during backward rather than a clear configuration warning. **Fix:** Start from a published ZeRO stage 3 example config for a similarly sized model rather than hand-tuning every field, and change one setting at a time while confirming training still runs. ### CPU or NVMe offload is enabled but the host lacks enough system RAM or fast enough storage to support it **Likelihood:** common **Explanation:** Offloading optimizer states or parameters to CPU RAM or NVMe trades GPU VRAM for system memory and I/O bandwidth. If system RAM is smaller than the offloaded data, or NVMe throughput is far below what the offload engine expects, the process can fail with allocation or timeout errors that read like config problems but are actually a hardware capacity mismatch. **Fix:** Calculate offloaded memory requirements, roughly proportional to optimizer state size, against actual available system RAM before enabling offload, and benchmark NVMe throughput if using NVMe offload. **Commands:** - free -h ### DeepSpeed configuration and HuggingFace Trainer or accelerate configuration both try to control batch size or precision **Likelihood:** occasional **Explanation:** When DeepSpeed is used through the Trainer or accelerate integration, both layers read overlapping settings from different places. If the two are set inconsistently, whichever layer initializes last can override the other in a way that produces the exact assertion the deeper layer expects to prevent. **Fix:** Pick one layer to own batch size and precision settings, usually the DeepSpeed JSON when using 'auto' values, and confirm the other layer is not separately overriding the same fields. ### DeepSpeed's version does not match the installed CUDA or PyTorch build, so its custom CUDA ops fail to compile or load **Likelihood:** rare **Explanation:** DeepSpeed ships several custom CUDA kernels compiled against a specific CUDA toolkit and PyTorch ABI. Installing a DeepSpeed version built against a different CUDA version than what is actually present can produce confusing errors that look like configuration mistakes but are actually a build or ABI mismatch. **Fix:** Run ds_report to check which ops compiled successfully and match against your installed CUDA and PyTorch versions, reinstalling DeepSpeed after confirming CUDA toolkit compatibility if any op shows as not installed. **Commands:** - ds_report ## Diagnostics ### Validate the DeepSpeed config's op compatibility **Command:** ds_report **Interpretation:** Lists each DeepSpeed op and whether it compiled successfully against your current CUDA and PyTorch install; any op marked not installed or incompatible explains downstream errors that otherwise look unrelated. ### Manually verify the batch-size equation **Command:** python -c "print(4 * 8 * 8)" **Interpretation:** Substitute your actual train_micro_batch_size_per_gpu, gradient_accumulation_steps, and world_size; the result must equal your intended train_batch_size exactly, or that alone explains the assertion error. ### Check available system RAM before enabling CPU offload **Command:** free -h **Interpretation:** Compare available memory against the estimated offloaded optimizer state size, roughly twice the model parameter count in FP32 for Adam; insufficient headroom here explains offload-related failures or extreme slowdowns. ## Prevention - Always set 'auto' for at least one of the three batch-size fields so DeepSpeed derives it, instead of hardcoding all three. - Start every new model or GPU count at ZeRO stage 2 with no offload to confirm the baseline works, then add complexity incrementally. - Benchmark actual NVMe or CPU RAM offload throughput on your specific hardware before committing a production training plan to it. - Keep DeepSpeed, PyTorch, and CUDA versions pinned together in your container image and re-validate with ds_report after any upgrade. ## When To Escalate If a model genuinely does not fit even with ZeRO stage 3 and full offload on your current hardware, or offload throughput makes training impractically slow, that is a capacity and hardware decision (more GPUs, more VRAM per GPU, or faster NVMe and interconnect) rather than a configuration problem to keep tuning around. ## Faq ### Why does DeepSpeed care about train_batch_size if I already set micro batch size and gradient accumulation? **Answer:** DeepSpeed uses train_batch_size as a consistency check across the whole cluster, since the effective global batch size depends on world_size, which can silently change when you scale the job to more or fewer GPUs. Setting it to 'auto' avoids the check ever going stale. ### Is CPU offload always slower than keeping everything on GPU? **Answer:** Yes, offload always adds overhead compared to fitting everything in GPU memory, because data has to move across the PCIe bus to and from system RAM or NVMe. It is a deliberate tradeoff: offload lets you train models that would not otherwise fit at all, at the cost of throughput, and is best used only for the portion of memory you cannot avoid offloading. ### Can I mix ZeRO stage 3 with tensor parallelism? **Answer:** Yes, and DeepSpeed and other frameworks increasingly support this combination for very large models, but it substantially increases configuration complexity. Get ZeRO working alone first, then add tensor parallelism, changing one dimension of parallelism at a time so you can isolate which layer introduced any new error. ## Search terms deepspeed zero config error, train_batch_size not equal to micro_batch_per_gpu, deepspeed zero stage 3 offload, deepspeed cpu offload slow, deepspeed ds_report, zero stage 3 contiguous tensors error --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json