Why your training dataset won't load or trains incorrectly, and how to fix the format

TRL's SFTTrainer, a custom HuggingFace Trainer, and raw JSONL loading each expect a specific field name and structure. A dataset built with a custom key like {"input", "output"} will fail to map to a trainer expecting {"prompt", "completion"} or a text field, because there is no field-name auto-detection happening by default in most training scripts.

Fix: Standardize your dataset to one supported schema (text field with the full formatted conversation, or messages as a list of role/content dicts) and rename keys during a preprocessing pass before training rather than trying to make the trainer flexible to arbitrary key names.

python -c "import datasets; ds=datasets.load_dataset('json', data_files='train.jsonl'); print(ds['train'][0])"

If some rows use a messages list of role/content turns and others use a flat prompt/completion pair (often from merging datasets from different sources), the data loading and formatting function only handles whichever schema it was written for, and rows in the other format either raise a KeyError or get silently formatted incorrectly, producing garbage training examples that don't error but still hurt the model.

Fix: Convert every example to a single consistent schema during a preprocessing step, and validate the converted dataset by rendering several examples through your formatting function and manually reading the output text.

If the response portion of a training example is not terminated with the tokenizer's EOS token before tokenization, the model never sees an example of when generation should stop, so it learns to keep generating past the intended answer indefinitely. This is a frequent issue with tokenizers that add EOS at the sequence end by default in some modes but not others depending on how you call the tokenizer.

Fix: Explicitly append the EOS token to every response string before tokenizing (or confirm your tokenizer call includes add_special_tokens correctly), and verify by checking that input_ids[-1] equals tokenizer.eos_token_id for training examples.

python -c "print(tokenizer.eos_token, tokenizer.eos_token_id)"

When a training example's token count exceeds max_length, standard truncation cuts from the end, which often removes part or all of the intended response (the labels), leaving the model to train on a prompt with a partial or entirely missing target. Because truncation happens silently by default, this can go unnoticed for a significant fraction of a dataset's longer examples.

Fix: Measure the token length distribution of your dataset before training and set max_length to cover at least the 95th-99th percentile, or filter out (rather than silently truncate) examples that exceed a length you've decided is acceptable to drop.

python -c "import datasets; ds=datasets.load_from_disk('train'); lens=[len(x['input_ids']) for x in ds]; import statistics as s; print(max(lens), s.quantiles(lens, n=100)[94])"

A JSONL file with a stray trailing comma, an unescaped quote inside a string field, or mixed encodings (a line saved with a different encoding than the rest of the file) can cause the JSON parser to fail on that specific line, and depending on the loader, this either halts the entire load or silently skips the malformed rows without informing you how many were dropped.

Fix: Validate the file by parsing it independently line by line with Python's json module before handing it to any training framework, logging the line number and error message for any line that fails so you can fix the source data directly.

python -c "import json\nbad=0\nfor i,l in enumerate(open('train.jsonl', encoding='utf-8')):\n    try: json.loads(l)\n    except Exception as e: print(i, e); bad+=1\nprint('bad lines:', bad)"