# Why tokenizer padding breaks your training run, and how to configure it correctly > Source: https://netray.co/troubleshoot/tokenizer-padding-errors-training > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Padding errors during training happen because many base models ship without a defined pad token at all, because the common workaround of setting pad_token equal to eos_token teaches the model that end-of-sequence and padding look identical (so it can learn to never emit a real stop signal), or because left-padding is used when right-padding was needed for the training collator, or vice versa. Add a distinct pad token when possible, and always right-pad for causal LM training. ## Error Title Tokenizer padding and truncation errors during training ## Error Signature ValueError: Asking to pad but the tokenizer does not have a padding token ## Alternate Signatures - model generates forever and never stops because pad_token equals eos_token - ValueError: Unable to create tensor, you should probably activate truncation ## Category fine-tuning ## Stack - huggingface - transformers - pytorch ## Affects Fine-tuning any causal language model with HuggingFace transformers and a data collator that pads batches ## Quick Fix ### Fastest path to correct padding configuration **Steps:** Check if the tokenizer already has a pad_token; if tokenizer.pad_token is None, first check whether the model card recommends a specific token before improvising. If no pad token exists and none is recommended, add a new one with tokenizer.add_special_tokens({'pad_token': '[PAD]'}) and resize the model's embeddings to match with model.resize_token_embeddings(len(tokenizer)). If you must reuse an existing token as pad, prefer a dedicated unused token over eos_token when one exists, since equating pad and eos causes the model to see padding as a stop signal during training. Set tokenizer.padding_side = 'right' for training (causal LM training expects labels aligned so padding at the end doesn't interfere with the causal mask), even though generation typically uses 'left' padding. Confirm your data collator masks padding tokens in labels with -100 so the loss function never computes cross-entropy over padding positions. ## Symptoms - Trainer raises an error stating the tokenizer has no padding token when trying to batch variable-length sequences - Fine-tuned model never emits an end-of-sequence token during generation, always running to max_new_tokens - Training runs without error but batched sequences of different lengths produce garbled or degraded outputs - Loss values look reasonable but inference behavior is inconsistent between single-example and batched generation - Padding side warnings appear about left vs right padding for a causal language model ## Causes ### Base model tokenizer has no pad token defined at all **Likelihood:** most-common **Explanation:** Many base models (particularly Llama-family and GPT-style models) were pretrained without ever needing to batch variable-length sequences with padding, so their tokenizer configuration simply has no pad_token defined. Any data collator that tries to pad a batch to a common length then fails immediately because there is no token id it can use to fill the shorter sequences. **Fix:** Check the model's documentation for a recommended pad token first (some architectures specify one), and if none exists, add a genuinely new special token as pad_token and resize the model's token embedding matrix to accommodate it. **Commands:** - python -c "from transformers import AutoTokenizer; t=AutoTokenizer.from_pretrained('MODEL'); print(t.pad_token)" - tokenizer.add_special_tokens({'pad_token': '[PAD]'}) - model.resize_token_embeddings(len(tokenizer)) ### pad_token set equal to eos_token **Likelihood:** common **Explanation:** A widely copied workaround sets tokenizer.pad_token = tokenizer.eos_token to avoid the missing-pad-token error quickly. This works for batching but means the model sees the identical token id used both to mean legitimate end-of-response and to mean meaningless padding filler during training, which teaches it a confused, weaker signal for when to actually stop generating, since padding positions are supposed to be masked out of the loss but implementation bugs sometimes fail to mask them correctly. **Fix:** Prefer adding a genuinely distinct pad token over reusing eos_token when possible. If you must reuse eos_token as pad (some architectures don't tolerate embedding resizing well), be extremely careful that your data collator masks all padding positions with -100 in labels so the model never actually trains on padding-as-eos confusion. ### Wrong padding side for causal LM training **Likelihood:** common **Explanation:** Causal language models predict each token from the tokens before it, so during generation, left-padding is used to keep the actual content aligned at the end of the sequence where new tokens are appended. During training, however, right-padding is typically expected so that the label masking and position of the real sequence content align correctly with how the loss and attention mask are constructed; using the wrong side for the wrong phase produces subtly broken attention or label alignment. **Fix:** Explicitly set tokenizer.padding_side = 'right' before tokenizing training data, and separately set it to 'left' only when configuring the tokenizer for batched generation/inference, rather than assuming one setting works for both phases. **Commands:** - tokenizer.padding_side = 'right' # for training ### Data collator not masking padding tokens in labels **Likelihood:** occasional **Explanation:** If a custom data collator copies input_ids directly into labels without also setting -100 at every padding position, the loss function computes cross-entropy over padding tokens as if they were meaningful targets. This adds noise to the loss and, combined with pad_token equaling eos_token, can specifically corrupt the model's learned stopping behavior. **Fix:** Use a collator (like DataCollatorForLanguageModeling with mlm=False, or a custom one) that explicitly sets labels to -100 wherever attention_mask is 0, and verify this with a direct batch inspection rather than assuming the default behavior is correct for your setup. ### Model embedding size not resized after adding a new pad token **Likelihood:** rare **Explanation:** Adding a new special token via add_special_tokens increases the tokenizer's vocabulary size, but the model's embedding and output projection layers keep their original dimensions unless explicitly resized. Training or even just tokenizing with the new token without calling resize_token_embeddings can produce an index-out-of-range error the first time that token id is actually used in a forward pass. **Fix:** Always call model.resize_token_embeddings(len(tokenizer)) immediately after adding any new special token, and verify the new embedding row is either randomly initialized reasonably or explicitly initialized to the mean of existing embeddings for more stable early training. ## Diagnostics ### Check whether a pad token already exists **Command:** python -c "from transformers import AutoTokenizer; t=AutoTokenizer.from_pretrained('MODEL'); print(t.pad_token, t.pad_token_id)" **Interpretation:** If pad_token is None, you must either add a new token or explicitly reuse an existing one before any padded batching will work. ### Confirm padding side setting **Command:** python -c "print(tokenizer.padding_side)" **Interpretation:** Should read 'right' during training and can be switched to 'left' specifically for generation; if it's set to the wrong value for the phase you're running, sequence alignment issues can silently degrade quality. ### Verify padding positions are masked in labels **Command:** python -c "b=next(iter(dl)); print((b['labels'][b['attention_mask']==0] == -100).all())" **Interpretation:** Should print True. If False, some padding positions have real token ids as labels instead of -100, meaning the loss is being computed over padding, which corrupts training signal quality. ### Confirm model embedding size matches tokenizer vocab size **Command:** python -c "print(model.get_input_embeddings().weight.shape[0], len(tokenizer))" **Interpretation:** These two numbers must match. If the tokenizer's length is larger (because a new token was added) but the model wasn't resized, you'll get an index error the moment that token id is used. ## Prevention - Standardize a tokenizer setup snippet used across all training scripts that checks for and correctly configures pad_token before anything else runs. - Never silently default to pad_token = eos_token without explicitly verifying your collator masks padding out of the loss. - Set and document padding_side explicitly and separately for training versus inference code paths rather than relying on tokenizer defaults. - Add an automated assertion in your training script that model embedding size matches tokenizer vocabulary size before training starts. - Include a quick generation smoke test after every fine-tuning run to confirm the model actually stops generating with a real EOS rather than running to max_new_tokens. ## When To Escalate If you're standardizing fine-tuning across many different base model families with inconsistent tokenizer conventions (some with pad tokens, some without, different padding side defaults), building a shared, tested tokenizer configuration library is worth doing rather than re-solving this per project. If subtle padding-related quality issues are only showing up in production at scale (batched inference quality worse than single-request quality), that's worth a focused audit of your serving-side padding configuration specifically. ## Faq ### Is it safe to set pad_token equal to eos_token? **Answer:** It works for batching without errors, but it risks the model associating padding with a legitimate stop signal, which can weaken its learned stopping behavior if padding tokens aren't perfectly masked out of the loss everywhere. Adding a genuinely separate pad token and resizing embeddings is the more robust choice when the architecture allows it. ### Should I use left or right padding for fine-tuning? **Answer:** Use right padding during training so sequence content and label alignment work correctly with standard causal LM data collators. Left padding is typically used only during batched generation/inference, where you want the real content aligned at the end of the sequence where new tokens get appended. ### Do I need to resize the model after adding a pad token? **Answer:** Yes, always call model.resize_token_embeddings(len(tokenizer)) after adding any new special token including a pad token, otherwise the model's embedding matrix will not have a row for the new token id and you'll hit an index error the first time it's actually used. ### Why does my model never stop generating after fine-tuning? **Answer:** This is commonly caused by a padding and EOS configuration problem: either pad_token was set equal to eos_token without proper label masking, or training examples were missing the EOS token entirely. Check both your padding setup and that input_ids for training examples end with a real eos_token_id. ## Search terms tokenizer padding error training, no padding token defined, pad_token equals eos_token, left vs right padding causal LM, resize_token_embeddings, labels padding mask -100, model never stops generating, add_special_tokens pad token --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json