# Why your training dataset won't load or trains incorrectly, and how to fix the format

> Source: https://netray.co/troubleshoot/training-dataset-format-errors
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Dataset format errors happen because the trainer expects a specific schema (either a messages list of role/content dicts, or a prompt/completion pair, or a single text field) and your JSONL doesn't match it, because samples are missing an EOS token so the model never learns to stop generating, or because a fixed max_length silently truncates long examples and cuts off labels partway through the intended response. Confirm your exact schema against what SFTTrainer or your data collator expects before training.

## Error Title

Training dataset format errors during fine-tuning

## Error Signature

KeyError: 'text' (or 'messages', 'prompt', 'completion')

## Alternate Signatures

- ValueError: Unable to create tensor, you should probably activate truncation and/or padding
- jsonlines.InvalidLineError: line contains invalid json

## Category

fine-tuning

## Stack

- huggingface
- transformers
- pytorch

## Affects

SFT with TRL's SFTTrainer, custom HuggingFace Trainer scripts, any JSONL-based instruction dataset

## Quick Fix

### Fastest path to a correctly formatted dataset

**Steps:**

Print the first 3 raw rows of your dataset and identify which schema they actually use: text, {"prompt", "completion"}, or {"messages": [{"role", "content"}, ...]}.

Match that schema to what your trainer expects; TRL's SFTTrainer wants either a text field or a messages field it can pass through apply_chat_template, not an arbitrary custom key.

Confirm every example ends with the tokenizer's EOS token before truncation is applied, by checking input_ids[-1] equals tokenizer.eos_token_id after tokenization for several samples.

Check your max_length or max_seq_length setting against your longest examples' token counts, since silent truncation cuts labels for the longest responses without any warning.

Validate the entire file parses as JSONL by loading it independently with the json module line by line before handing it to the trainer, to catch malformed lines early.

## Symptoms

- Trainer raises a KeyError naming a field like 'text', 'messages', 'prompt', or 'completion' that it expected but didn't find
- Loading the dataset raises a JSON parsing error on a specific line number
- Model never produces an end-of-sequence token during generation, running until max_new_tokens is hit every time
- Training completes without error but the model performs poorly specifically on longer examples
- Tokenization warns about sequences exceeding max_length and being truncated

## Causes

### Dataset schema does not match what the trainer expects

**Likelihood:** most-common

**Explanation:** TRL's SFTTrainer, a custom HuggingFace Trainer, and raw JSONL loading each expect a specific field name and structure. A dataset built with a custom key like {"input", "output"} will fail to map to a trainer expecting {"prompt", "completion"} or a text field, because there is no field-name auto-detection happening by default in most training scripts.

**Fix:** Standardize your dataset to one supported schema (text field with the full formatted conversation, or messages as a list of role/content dicts) and rename keys during a preprocessing pass before training rather than trying to make the trainer flexible to arbitrary key names.

**Commands:**

- python -c "import datasets; ds=datasets.load_dataset('json', data_files='train.jsonl'); print(ds['train'][0])"

### Mixing messages-style and prompt/completion-style examples in one file

**Likelihood:** common

**Explanation:** If some rows use a messages list of role/content turns and others use a flat prompt/completion pair (often from merging datasets from different sources), the data loading and formatting function only handles whichever schema it was written for, and rows in the other format either raise a KeyError or get silently formatted incorrectly, producing garbage training examples that don't error but still hurt the model.

**Fix:** Convert every example to a single consistent schema during a preprocessing step, and validate the converted dataset by rendering several examples through your formatting function and manually reading the output text.

### Missing EOS token at the end of the response

**Likelihood:** common

**Explanation:** If the response portion of a training example is not terminated with the tokenizer's EOS token before tokenization, the model never sees an example of when generation should stop, so it learns to keep generating past the intended answer indefinitely. This is a frequent issue with tokenizers that add EOS at the sequence end by default in some modes but not others depending on how you call the tokenizer.

**Fix:** Explicitly append the EOS token to every response string before tokenizing (or confirm your tokenizer call includes add_special_tokens correctly), and verify by checking that input_ids[-1] equals tokenizer.eos_token_id for training examples.

**Commands:**

- python -c "print(tokenizer.eos_token, tokenizer.eos_token_id)"

### Fixed max_length silently truncates long examples, cutting off labels

**Likelihood:** common

**Explanation:** When a training example's token count exceeds max_length, standard truncation cuts from the end, which often removes part or all of the intended response (the labels), leaving the model to train on a prompt with a partial or entirely missing target. Because truncation happens silently by default, this can go unnoticed for a significant fraction of a dataset's longer examples.

**Fix:** Measure the token length distribution of your dataset before training and set max_length to cover at least the 95th-99th percentile, or filter out (rather than silently truncate) examples that exceed a length you've decided is acceptable to drop.

**Commands:**

python -c "import datasets; ds=datasets.load_from_disk('train'); lens=[len(x['input_ids']) for x in ds]; import statistics as s; print(max(lens), s.quantiles(lens, n=100)[94])"

### Malformed JSONL with invalid lines or inconsistent encoding

**Likelihood:** rare

**Explanation:** A JSONL file with a stray trailing comma, an unescaped quote inside a string field, or mixed encodings (a line saved with a different encoding than the rest of the file) can cause the JSON parser to fail on that specific line, and depending on the loader, this either halts the entire load or silently skips the malformed rows without informing you how many were dropped.

**Fix:** Validate the file by parsing it independently line by line with Python's json module before handing it to any training framework, logging the line number and error message for any line that fails so you can fix the source data directly.

**Commands:**

python -c "import json\nbad=0\nfor i,l in enumerate(open('train.jsonl', encoding='utf-8')):\n    try: json.loads(l)\n    except Exception as e: print(i, e); bad+=1\nprint('bad lines:', bad)"

## Diagnostics

### Print raw dataset rows to confirm schema

**Command:** python -c "import datasets; ds=datasets.load_dataset('json', data_files='train.jsonl'); print(ds['train'][0]); print(ds['train'][1])"

**Interpretation:** Confirms exactly which keys exist in your data (text, messages, prompt/completion, or something custom) so you can match your formatting function to what's actually there rather than what you assumed was there.

### Check that examples end with EOS after tokenization

**Command:** python -c "ids=tokenizer(example_text)['input_ids']; print(ids[-3:], tokenizer.eos_token_id)"

**Interpretation:** The last token id should match tokenizer.eos_token_id. If it doesn't, the model is not seeing where to stop and will likely run on during generation.

### Measure token length distribution against max_length

**Command:** python -c "lens=[len(tokenizer(x['text'])['input_ids']) for x in ds]; print(max(lens))"

**Interpretation:** If the maximum length substantially exceeds your configured max_length, a meaningful fraction of your dataset is being truncated, likely cutting into labels for your longest and often most information-dense examples.

### Validate JSONL parses line by line

**Command:** python -c "import json; [json.loads(l) for l in open('train.jsonl', encoding='utf-8')]"

**Interpretation:** Raises immediately at the first invalid line with its line number if any row is malformed JSON, letting you fix the exact source row rather than debugging a confusing downstream trainer error.

## Prevention

- Pick one dataset schema (messages or prompt/completion or text) at project start and enforce it with a validation script run before every training job.
- Add an automated check for EOS token presence and length-distribution reporting as a required preprocessing step, not an optional one.
- Never silently truncate: log or reject examples that exceed max_length so you know exactly how much data is affected.
- Keep a small set of held-out format sanity-check examples (including edge cases like very long responses) that you re-verify after any preprocessing script change.
- Version your dataset preprocessing scripts alongside the raw data so format changes are traceable.

## When To Escalate

If your organization is merging training data from many sources (support tickets, documentation, chat logs) with inconsistent formats on an ongoing basis, ad hoc format-fixing scripts stop scaling and you need a proper data pipeline with schema validation built in. If dataset quality issues keep surfacing as training problems rather than being caught before training starts, that's a signal to invest in a dedicated data preparation and validation stage as part of your fine-tuning process.

## Faq

### What dataset schema does TRL's SFTTrainer expect?

**Answer:** SFTTrainer can work with a text field containing the fully formatted training string, or a messages field structured as a list of role/content dictionaries that gets passed through the tokenizer's chat template. It does not automatically understand arbitrary custom field names like input/output without a formatting function you provide.

### Why does my fine-tuned model never stop generating?

**Answer:** This almost always means training examples were missing the EOS token at the end of the response, so the model never learned an example of where a response should end. Check that input_ids for your training examples end with tokenizer.eos_token_id and add it explicitly if it's missing.

### Does truncation happen silently during tokenization?

**Answer:** Yes, by default, tokenizers truncate to max_length without raising an error, simply cutting tokens from the end (or configured side) of the sequence. This can silently remove the response portion of long training examples unless you explicitly check length distributions or filter examples before training.

### How do I check if my JSONL file has invalid lines?

**Answer:** Parse the file independently with Python's json module, line by line, wrapped in a try/except that logs the line number and error for any failure. This isolates malformed rows before they reach a training framework, where the resulting error message is often less specific about which line caused the problem.

## Search terms

training dataset format error, JSONL schema fine-tuning, messages vs prompt completion, missing EOS token training, silent truncation labels, SFTTrainer KeyError, invalid JSON line training data, dataset preprocessing fine-tuning

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json