# Why you get 'expected all tensors to be on the same device', and where the mismatched tensor actually comes from > Source: https://netray.co/troubleshoot/expected-all-tensors-same-device > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer This error means a tensor was created or moved to a specific device, often cuda:0 or cpu, that does not match the device another tensor in the same operation lives on, which happens most often when leftover manual .to(device) calls from single-GPU code collide with automatic sharding from device_map='auto' or DistributedDataParallel. The fix is to trace exactly which tensor has the wrong device with a quick print of tensor.device, not to guess and add more .to() calls. ## Error Title Expected all tensors to be on the same device error in multi-GPU code ## Error Signature RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! ## Alternate Signatures - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! - RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same ## Category distributed ## Stack - pytorch - transformers - accelerate - huggingface ## Affects PyTorch and HuggingFace transformers models using device_map='auto', DataParallel, or manual multi-GPU placement, most common when migrating single-GPU code to multi-GPU ## Quick Fix ### Find the specific hardcoded device call before adding more .to() calls **Steps:** Search your codebase for every literal .to("cuda") or .cuda() call and check whether any of them run after the model was loaded with device_map='auto'. If using device_map='auto', print model.hf_device_map to see the actual per-layer placement, then make sure inputs are moved to the device of the model's first layer, not a hardcoded cuda:0. Check any custom data collator or preprocessing function for tensors it creates directly (labels, position_ids, attention_mask), since these often default to CPU or a hardcoded device instead of matching the batch. If using Trainer or accelerate, let the framework handle device placement entirely and remove any manual .to(device) calls on the model or optimizer that predate the multi-GPU setup. Add a one-line assertion right before the failing operation during debugging to pinpoint exactly which tensor is wrong, then remove the assertion once fixed. ## Symptoms - Error appears only after moving from single-GPU to multi-GPU or device_map='auto' setups, never on a single device - The error names two specific devices, often cuda:0 and another GPU, or cuda:0 and cpu - Custom training loops or custom collators are involved; the error rarely occurs with unmodified library training loops - The failure happens consistently on the same operation, such as loss computation or a position embedding lookup, every run ## Causes ### A manual .to(device) call left over from earlier single-GPU code now conflicts with a model sharded automatically across multiple devices **Likelihood:** most-common **Explanation:** When code is first written for a single GPU, it is common to hardcode .to("cuda") or .cuda() on the model, inputs, or specific tensors. Once the model is later loaded with device_map='auto', which places different layers on different GPUs, that hardcoded call pins one tensor to a fixed device while the model expects inputs to match whatever device each layer actually landed on, causing a mismatch the moment that layer runs. **Fix:** Remove hardcoded .to("cuda") calls on the model itself when using device_map='auto', and move inputs to the device of the model's embedding layer rather than a fixed index. ### A custom data collator or preprocessing step creates new tensors (labels, position_ids, custom masks) that default to CPU or a different device **Likelihood:** common **Explanation:** Collators often build auxiliary tensors from scratch, which default to CPU unless a device is explicitly specified. If the rest of the batch has already been moved to GPU by the training loop but the collator's output has not, the mismatch surfaces the first time the two tensors interact, such as in a loss or attention computation. **Fix:** Either leave all collator output on CPU and let the Trainer or accelerate move the whole batch to the correct device afterward, or explicitly set the device to match the batch when the collator creates new tensors. ### device_map='auto' automatic sharding is combined with an explicit .to("cuda:0") call on the model or specific inputs **Likelihood:** common **Explanation:** device_map='auto' uses the accelerate library to compute a per-layer device assignment based on available VRAM across GPUs, and expects nothing else to override that placement afterward. A subsequent .to("cuda:0") on the whole model attempts to move every parameter to a single device, which can partially succeed or silently break the mapping accelerate had already set up, leaving some tensors on their originally assigned device and others moved. **Fix:** Never call .to(device) on a model loaded with device_map='auto'; if you need to move only the inputs, query the model's actual device placement instead of assuming cuda:0. ### A cached buffer (rotary embedding cache, KV cache, precomputed positional encoding) is created once at init time on whatever device was active then **Likelihood:** occasional **Explanation:** Some custom model implementations precompute and cache tensors like rotary embedding tables the first time they are needed, using whatever device is current at that moment. If that first call happens before multi-GPU placement is finalized, or from a different rank than later calls, the cached buffer stays pinned to the wrong device for subsequent calls. **Fix:** Register such buffers with register_buffer() so they move automatically with the module's .to() calls, or explicitly move the cache to match the input tensor's device at each call rather than caching a device assumption. ### A custom stopping criteria, logits processor, or generation utility allocates a new tensor without specifying a device **Likelihood:** rare **Explanation:** Custom generation utilities sometimes create small tensors, such as a done flag or an EOS check tensor, which default to CPU, then combine them with GPU logits tensors during the generation loop, producing the same device mismatch pattern in a code path that is easy to overlook since it is not part of the main forward pass. **Fix:** Explicitly pass a matching device to every tensor-creation call inside custom generation logic, matching whatever tensor it will be combined with. ## Diagnostics ### Print actual per-layer device placement for device_map='auto' models **Command:** python -c "print(model.hf_device_map)" **Interpretation:** Shows exactly which layer lives on which device; use this to confirm where inputs need to be moved instead of assuming cuda:0 for everything. ### Print the device of every tensor going into the failing operation **Command:** python -c "print({k: v.device for k, v in batch.items()})" **Interpretation:** Any key showing a device different from the rest of the batch, or from the model's first layer, is the specific tensor causing the mismatch and should be traced back to where it was created. ### Search the codebase for hardcoded device calls **Command:** grep -rn 'cuda(' --include='*.py' . **Interpretation:** Every hit is a candidate for the leftover single-GPU code that conflicts with automatic multi-GPU placement; review each one against whether the model uses device_map='auto' or DDP. ## Prevention - Avoid hardcoding device strings anywhere in model or data pipeline code; always derive the device from an existing tensor or the model's own placement. - Register any custom cached buffers with register_buffer() so PyTorch's built-in device-movement machinery handles them automatically. - Code-review multi-GPU pull requests specifically for leftover .to("cuda") or .cuda() calls inherited from single-GPU development. - Add an integration test that runs the training or inference path on at least two GPUs in CI, since this class of bug is invisible on single-GPU test runs. ## When To Escalate If device placement bugs keep recurring across a growing custom codebase built around device_map='auto', it is worth standardizing on a single, well-tested placement pattern, or a maintained framework like accelerate or DeepSpeed, rather than continuing to hand-roll device management, since the failure mode scales with code complexity, not GPU count. ## Faq ### Why does this error only show up with multiple GPUs, never with one? **Answer:** On a single GPU, every tensor moved to 'cuda' lands on the same physical device, so there is nothing to mismatch. Multi-GPU setups introduce more than one valid device target, which is what exposes any code that assumed a single fixed device. ### Should I add .to(model.device) everywhere to fix this? **Answer:** Only if the model actually lives on a single device. If you are using device_map='auto', the model itself spans multiple devices and has no single device, so blindly calling .to(model.device) will not work and can mask the real issue; use model.hf_device_map or the first layer's device instead. ### Does DataParallel have this problem too, or only device_map='auto'? **Answer:** DataParallel replicates the whole model onto each GPU and handles device placement internally for its own forward pass, so it is less prone to this specific error than device_map='auto' sharding, but custom code that creates tensors outside the managed forward call can still trigger the same mismatch. ## Search terms expected all tensors to be on the same device, cuda:0 and cuda:1 device mismatch, device_map auto device error, pytorch tensor device mismatch multi gpu, hf_device_map error --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json