Choosing the Right Format: Instruction, Chat, or Completion
Instruction format (a single instruction plus a response, sometimes with an optional input field) suits single-turn tasks like classification, extraction, or summarization drawn from structured records. Chat format, with explicit system, user, and assistant role turns, is the right choice when you are teaching multi-turn behavior or need the fine-tune to inherit a specific persona and refusal behavior consistent with how the model will actually be called in production. Completion format, plain text continuation with no role structure, is reserved for continued pretraining on domain corpora rather than instruction fine-tuning.
Match the format to the exact chat template of your target base model (Llama, Qwen, and Mistral families each use different special tokens), since a mismatched template at training time silently degrades instruction-following quality without throwing an error.
- Instruction format: single-turn extraction, classification, summarization tasks with structured source records
- Chat format: multi-turn behavior, persona consistency, matches production API call shape
- Completion format: continued pretraining on domain text, not instruction fine-tuning
- Always match the base model's exact chat template, a mismatch degrades quality silently
Sourcing and Extracting Data From ERP, Tickets, and Documents
The highest-value enterprise datasets come from systems that already contain a verified correct outcome: closed support tickets with a resolution, historical ERP transactions with the human decision recorded, engineering change orders with the approved final state. Extract pairs of input context and correct output directly from these systems rather than having someone write examples from memory, since memory-written examples systematically miss the messy edge cases that make up a meaningful share of real production traffic. Pull a wide time window (12 months minimum) to capture seasonal and process variation, and deliberately include the ugly cases: incomplete records, corrected mistakes, and disputed outcomes, since a dataset of only clean happy-path examples produces a model that is confidently wrong the first time it sees a real edge case.
Deduplication: Exact, Near-Duplicate, and Semantic
Exact deduplication by content hash is the easy first pass and catches copy-pasted templates and repeated boilerplate responses that would otherwise get overweighted during training. Near-duplicate detection, typically via MinHash with locality-sensitive hashing, catches examples that differ only in a few tokens (a support response with a swapped customer name, an ERP record with a different quantity), which exact hashing misses entirely. 95 catches paraphrased duplicates that neither hash-based method finds, though it is more compute-intensive and usually only worth running once as a final pass on a dataset that has already been through exact and near-duplicate filtering.
Deduplication also matters critically for preventing train-eval contamination, covered separately in evaluation design.
Try it here, free
Fine-Tuning Dataset Size Estimator
Estimate the number of training examples, total tokens, and human curation hours needed for a fine-tuning dataset based on task complexity and quality bar.
PII and Sensitive Data Scrubbing
Scrub PII before data leaves any system boundary you do not fully control, not as a post-hoc cleanup step after data has already been copied to a training environment. A combined approach works best: regex-based detection for structured identifiers (phone numbers, SSNs, account numbers, email addresses) combined with a named-entity-recognition model for unstructured mentions of names, addresses, and organizations that regex alone will miss. For manufacturing and defense clients, scrubbing extends beyond personal data to export-controlled technical details, part numbers tied to controlled programs, and customer-identifying information embedded in free-text fields, which is exactly the category of scrubbing that justifies running the entire pipeline on-premises rather than through a third-party data labeling service.
- Regex pass for structured identifiers: phone, email, SSN, account and employee ID formats
- NER model pass for unstructured mentions: names, addresses, organizations in free text
- Manufacturing and defense: extend scrubbing to export-controlled part numbers and program identifiers
- Scrub before data leaves the controlled environment, never as a cleanup step afterward
Quality Filtering and Dataset Sizing
Quality filtering removes truncated records, boilerplate auto-responses, examples where the recorded outcome was later corrected or overturned, and outliers flagged by a perplexity check against a reference model. For dataset size, task-specific behavior shaping with LoRA typically needs 500 to 5,000 well-curated examples to show a measurable effect, while broader capability transfer or teaching a wide range of task variants needs 10,000 to 100,000-plus examples. More data with lower average quality consistently underperforms less data with higher quality in practice, so a curation pass that cuts a raw 50,000-example pull down to a clean 8,000-example set is usually the right tradeoff, not a loss.
Frequently asked questions
What is the right fine-tuning dataset format?
Store one training example per line in JSONL. For instruction tuning, each record holds an instruction, an optional input and the target output. For chat fine-tuning, each record holds a messages list of system, user and assistant turns, rendered with the base model's own chat template, because a template mismatch degrades quality without raising any error. Use plain completion records only for continued pretraining on domain text. Validate the file by checking that every line parses, every record ends on an assistant turn, and no example exceeds the training context length.
How much data do I need to fine-tune a model for an enterprise task?
Task-specific behavior shaping with LoRA typically shows measurable improvement with 500 to 5,000 well-curated examples. Broader capability transfer or a wide range of task variants within one fine-tune usually needs 10,000 to 100,000-plus examples. Quality matters more than raw volume: a smaller, well-curated dataset consistently outperforms a larger dataset with more noise and duplication.
How do you remove PII from training data before fine-tuning?
Combine regex-based detection for structured identifiers like phone numbers, emails, and account IDs with a named-entity-recognition model to catch unstructured mentions of names, addresses, and organizations in free text. Scrub before the data leaves any system boundary you control, not as a later cleanup step. For regulated industries, extend scrubbing to export-controlled technical details and program-identifying information as well.
What deduplication method works best for fine-tuning datasets?
Run exact hash deduplication first to catch identical or copy-pasted records, then near-duplicate detection via MinHash with locality-sensitive hashing to catch examples that differ by only a few tokens. A final semantic pass using embedding cosine similarity around a 0.92 to 0.95 threshold catches paraphrased duplicates neither method finds, though it is the most compute-intensive step and best run last.
Should I use instruction format or chat format for fine-tuning?
Use instruction format for single-turn tasks like extraction, classification, or summarization drawn from structured records. Use chat format with explicit system, user, and assistant turns when teaching multi-turn behavior or when the fine-tune needs to inherit a persona and refusal behavior matching how the model will actually be called in production. Always match the exact chat template of your target base model.
Sitting on years of ERP tickets, engineering documents, or support transcripts you want to turn into a fine-tuning dataset without it ever leaving your network? Netray builds the extraction, dedup, and PII-scrubbing pipeline on your own infrastructure end to end.