# Why vLLM's tensor-parallel-size must divide attention heads evenly, and how to fix the error > Source: https://netray.co/troubleshoot/vllm-tensor-parallel-size-error > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer vLLM shards attention heads and KV heads evenly across GPUs for tensor parallelism, so --tensor-parallel-size must be a divisor of the model's head count and must equal the number of GPUs vLLM can actually see. Pick a TP size from the model's valid divisors, commonly 1, 2, 4, or 8, and make sure CUDA_VISIBLE_DEVICES exposes exactly that many GPUs. ## Error Title vLLM tensor-parallel-size must divide the number of attention heads ## Error Signature ValueError: Total number of attention heads (32) must be divisible by tensor parallel size (3) ## Alternate Signatures - AssertionError: tensor_parallel_size (X) must be less than or equal to the number of GPUs (Y) - RuntimeError: CUDA error: invalid device ordinal ## Category serving ## Stack - vllm - cuda ## Affects vLLM 0.3 and later, any multi-GPU host; occurs whenever --tensor-parallel-size doesn't evenly divide the model's num_attention_heads or num_key_value_heads, or doesn't match visible GPU count ## Quick Fix ### Fix it in a few minutes **Steps:** - Find the model's num_attention_heads and num_key_value_heads in its config.json. - Pick a --tensor-parallel-size that evenly divides both, commonly 1, 2, 4, or 8 for standard architectures. - Confirm CUDA_VISIBLE_DEVICES exposes exactly that many physical GPUs, for example 0,1,2,3 for TP=4. - Re-launch with vllm serve MODEL_ID --tensor-parallel-size 4. - If the head count only divides evenly by an odd or prime number, use fewer GPUs than you own for tensor parallelism and consider pipeline parallelism for the rest. ## Symptoms - Server crashes at initialization with a ValueError naming attention heads and your requested TP size. - Works with --tensor-parallel-size 2 or 4 but fails at 3, 5, 6, or other non-divisor values. - Different models on the same host need different TP sizes even with the same GPU count. - Error mentions invalid device ordinal when TP size exceeds visible GPU count. ## Causes ### Requested TP size is not a divisor of the model's attention head count **Likelihood:** most-common **Explanation:** Tensor parallelism splits each attention layer's heads evenly across ranks so each GPU computes an equal, non-overlapping share of the heads. If the head count doesn't divide evenly, vLLM has no valid way to shard the weight matrices and refuses to start rather than silently pad or drop heads. **Fix:** Look up num_attention_heads, and num_key_value_heads for GQA models, in the model's config.json, and choose a TP size from its integer divisors. **Commands:** python -c "from transformers import AutoConfig; c = AutoConfig.from_pretrained('MODEL_ID'); print(c.num_attention_heads, getattr(c, 'num_key_value_heads', None))" ### TP size exceeds the number of GPUs actually visible to the process **Likelihood:** common **Explanation:** vLLM spawns one worker process per TP rank and assigns it a GPU ordinal. If you ask for --tensor-parallel-size 4 but CUDA_VISIBLE_DEVICES only exposes 2 devices, or the container only has 2 GPUs passed through, the later ranks fail to acquire a device. **Fix:** Set --tensor-parallel-size to match the length of CUDA_VISIBLE_DEVICES exactly, and verify with nvidia-smi -L inside the same environment vLLM runs in. **Commands:** - nvidia-smi -L - echo $CUDA_VISIBLE_DEVICES ### GQA models have fewer KV heads than attention heads, and only KV heads divide cleanly **Likelihood:** common **Explanation:** Grouped-query attention models such as Llama 3, Mistral, and Qwen have far fewer key/value heads than query heads. Both must be shardable by TP size, so a TP size that divides the larger attention head count but not the smaller KV head count will still fail. **Fix:** Check num_key_value_heads specifically, not just num_attention_heads; TP size must divide both. ### Container or scheduler only allocated a subset of the requested GPUs **Likelihood:** occasional **Explanation:** In Kubernetes or Docker, a resource request for N GPUs that only gets M less than N allocated, due to a misconfigured device plugin or resource quota, will present fewer devices than vLLM's launch flag assumes, producing the same divisibility or ordinal errors even though your manifest looks correct. **Fix:** Verify the actual GPU count inside the running container, not just the requested count in the deployment spec. **Commands:** - kubectl exec POD_NAME -- nvidia-smi -L ### Mixing tensor-parallel-size with pipeline-parallel-size incorrectly **Likelihood:** rare **Explanation:** When both --tensor-parallel-size and --pipeline-parallel-size are set, the total GPU count must equal TP times PP. A mismatch between that product and visible devices produces the same class of device-ordinal errors even when TP alone would have divided cleanly. **Fix:** Confirm tensor_parallel_size multiplied by pipeline_parallel_size equals your total GPU count before launch. ## Diagnostics ### Get the model's head counts **Command:** python -c "from transformers import AutoConfig; c = AutoConfig.from_pretrained('MODEL_ID'); print('heads:', c.num_attention_heads, 'kv_heads:', getattr(c, 'num_key_value_heads', 'n/a'))" **Interpretation:** Both numbers must be evenly divisible by your chosen TP size; if only one is, TP size has to divide the smaller KV head count. ### List visible GPUs in the exact runtime environment **Command:** nvidia-smi -L **Interpretation:** Count the lines; --tensor-parallel-size must equal this count, or TP times PP if using pipeline parallelism too. ### Check CUDA_VISIBLE_DEVICES scoping **Command:** echo $CUDA_VISIBLE_DEVICES **Interpretation:** If unset, all GPUs on the host are visible; if set, only the listed ordinals are, and vLLM's TP size must match that list's length. ## Prevention - Standardize on GPU counts per node, such as 2, 4, or 8, that are known divisors for the model families you deploy. - Script a pre-flight check that reads config.json head counts and validates --tensor-parallel-size before launching in CI/CD. - Document per-model TP size in your deployment manifests so on-call engineers don't have to rederive it during an incident. - Test new model releases' head-count divisibility against your standard GPU counts before promoting to production. ## When To Escalate If a model's head count only divides evenly by GPU counts you don't have available, or by 1, that's an architecture-hardware mismatch: consider pipeline parallelism instead of tensor parallelism, a differently-shaped model variant, or provisioning nodes with a GPU count that matches the model's valid divisors. ## Faq ### Why must tensor-parallel-size divide the head count evenly? **Answer:** Each GPU in a TP group computes a disjoint, equal-sized slice of the attention heads for every layer. If the head count can't be split into equal integer groups, vLLM has no correct way to partition the weight matrices, so it fails fast instead of producing subtly wrong outputs. ### What TP sizes are safe for common models? **Answer:** Most modern open-weight models such as Llama, Qwen, and Mistral use head counts that are powers of two or have small prime factors, so TP of 1, 2, 4, or 8 usually works. Always confirm against the specific model's config.json rather than assuming. ### Can I use 3 or 5 GPUs for tensor parallelism? **Answer:** Only if the model's attention and KV head counts are divisible by that number, which is uncommon for mainstream architectures. If you only have 3 or 5 GPUs and the model doesn't support that split, use fewer GPUs for TP and consider pipeline parallelism for the remainder. ## Search terms vllm tensor parallel size error, attention heads must be divisible, vllm tensor parallelism gpu count, vllm invalid device ordinal, tensor-parallel-size vllm, vllm multi gpu serving error --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json