Why tokenizer padding breaks your training run, and how to configure it correctly

Many base models (particularly Llama-family and GPT-style models) were pretrained without ever needing to batch variable-length sequences with padding, so their tokenizer configuration simply has no pad_token defined. Any data collator that tries to pad a batch to a common length then fails immediately because there is no token id it can use to fill the shorter sequences.

Fix: Check the model's documentation for a recommended pad token first (some architectures specify one), and if none exists, add a genuinely new special token as pad_token and resize the model's token embedding matrix to accommodate it.

python -c "from transformers import AutoTokenizer; t=AutoTokenizer.from_pretrained('MODEL'); print(t.pad_token)"
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
model.resize_token_embeddings(len(tokenizer))

A widely copied workaround sets tokenizer.pad_token = tokenizer.eos_token to avoid the missing-pad-token error quickly. This works for batching but means the model sees the identical token id used both to mean legitimate end-of-response and to mean meaningless padding filler during training, which teaches it a confused, weaker signal for when to actually stop generating, since padding positions are supposed to be masked out of the loss but implementation bugs sometimes fail to mask them correctly.

Fix: Prefer adding a genuinely distinct pad token over reusing eos_token when possible. If you must reuse eos_token as pad (some architectures don't tolerate embedding resizing well), be extremely careful that your data collator masks all padding positions with -100 in labels so the model never actually trains on padding-as-eos confusion.

Causal language models predict each token from the tokens before it, so during generation, left-padding is used to keep the actual content aligned at the end of the sequence where new tokens are appended. During training, however, right-padding is typically expected so that the label masking and position of the real sequence content align correctly with how the loss and attention mask are constructed; using the wrong side for the wrong phase produces subtly broken attention or label alignment.

Fix: Explicitly set tokenizer.padding_side = 'right' before tokenizing training data, and separately set it to 'left' only when configuring the tokenizer for batched generation/inference, rather than assuming one setting works for both phases.

tokenizer.padding_side = 'right'  # for training

If a custom data collator copies input_ids directly into labels without also setting -100 at every padding position, the loss function computes cross-entropy over padding tokens as if they were meaningful targets. This adds noise to the loss and, combined with pad_token equaling eos_token, can specifically corrupt the model's learned stopping behavior.

Fix: Use a collator (like DataCollatorForLanguageModeling with mlm=False, or a custom one) that explicitly sets labels to -100 wherever attention_mask is 0, and verify this with a direct batch inspection rather than assuming the default behavior is correct for your setup.

Adding a new special token via add_special_tokens increases the tokenizer's vocabulary size, but the model's embedding and output projection layers keep their original dimensions unless explicitly resized. Training or even just tokenizing with the new token without calling resize_token_embeddings can produce an index-out-of-range error the first time that token id is actually used in a forward pass.

Fix: Always call model.resize_token_embeddings(len(tokenizer)) immediately after adding any new special token, and verify the new embedding row is either randomly initialized reasonably or explicitly initialized to the mean of existing embeddings for more stable early training.