# Why your fine-tuning run is so much slower than it should be, and how to speed it up

> Source: https://netray.co/troubleshoot/fine-tuning-slower-than-expected
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Slow fine-tuning is usually a GPU sitting idle waiting on the CPU dataloader, a batch size too small to keep the GPU's compute units busy, missing FlashAttention or another fused kernel so attention runs in a much slower fallback path, or CPU/NVMe offloading (for models too large to fit fully in VRAM) that is inherently bound by PCIe bandwidth rather than GPU compute. Check GPU utilization with nvidia-smi first to know which of these you're actually facing.

## Error Title

Fine-tuning is much slower than expected

## Error Signature

training throughput far below expected tokens/sec or steps/sec for the GPU and model size

## Alternate Signatures

- GPU utilization stuck below 50% during training (checked via nvidia-smi)
- each training step takes several times longer than benchmark numbers suggest it should

## Category

fine-tuning

## Stack

- pytorch
- huggingface
- transformers
- accelerate
- deepspeed

## Affects

LoRA and full fine-tuning on single or multi-GPU setups, HuggingFace transformers, Accelerate, and DeepSpeed

## Quick Fix

### Fastest path to faster training

**Steps:**

Run nvidia-smi in a loop during training and check GPU utilization; if it's frequently below 70-80%, the bottleneck is likely the dataloader or CPU-side preprocessing, not the GPU itself.

Increase per-device batch size and/or gradient accumulation steps until GPU memory is well utilized (watch nvidia-smi memory.used), since small batches underutilize modern GPU compute.

Enable FlashAttention 2 (attn_implementation='flash_attention_2') if your model and GPU support it, since the default eager attention implementation is meaningfully slower, especially at longer sequence lengths.

Increase dataloader_num_workers and enable pin_memory=True so data loading and tokenization happen in parallel with GPU compute instead of blocking it.

If using CPU or NVMe offloading (DeepSpeed ZeRO-Offload) because the model doesn't fit in VRAM, understand that offloading is inherently PCIe-bandwidth-bound; the real fix is more or bigger GPUs, not further offload tuning.

## Symptoms

- nvidia-smi shows GPU utilization frequently dropping to 0-30% between steps rather than staying consistently high
- Training throughput (tokens/sec or steps/sec) is far below published benchmark numbers for similar hardware and model size
- Increasing batch size doesn't meaningfully speed up wall-clock training time per epoch
- CPU usage is pegged at 100% on one or more cores while GPU sits comparatively idle
- Training with CPU or disk offloading is dramatically slower than training without it (expected, but often more severe than anticipated)

## Causes

### Batch size too small to utilize the GPU's compute units

**Likelihood:** most-common

**Explanation:** Modern GPUs have far more parallel compute capacity than a very small batch can occupy; with a tiny batch, most of the GPU's tensor cores sit idle during each forward and backward pass, and per-step overhead (kernel launches, synchronization) becomes a larger fraction of total time relative to actual compute, making each step slower than it needs to be for the work performed.

**Fix:** Increase per_device_train_batch_size as far as VRAM allows, and use gradient_accumulation_steps to reach your desired effective batch size without exceeding memory, which recovers most of the throughput lost to a too-small batch.

### Sequence packing not used, wasting compute on padding

**Likelihood:** common

**Explanation:** Without packing, a batch of variable-length examples is padded to the length of its longest member, meaning shorter examples waste compute processing padding tokens that produce no useful gradient. On datasets with high length variance, this can mean a large fraction of every batch's compute is spent on tokens that contribute nothing to learning.

**Fix:** Enable sequence packing (TRL's SFTTrainer supports packing=True) which concatenates multiple short examples into fixed-length sequences separated by EOS tokens, dramatically reducing wasted compute on padding for datasets with variable-length examples.

### Missing FlashAttention or another fused attention kernel

**Likelihood:** common

**Explanation:** The default eager attention implementation computes the full attention matrix with several separate, unfused GPU operations, each with its own memory read/write overhead. FlashAttention and similar fused kernels compute the same mathematical result with a single, memory-efficient fused kernel, which is significantly faster and also uses less memory, especially as sequence length grows.

**Fix:** Load the model with attn_implementation='flash_attention_2' (or 'sdpa' as a solid built-in alternative if FlashAttention installation isn't feasible) instead of leaving the default eager implementation active.

**Commands:**

- model = AutoModelForCausalLM.from_pretrained(model_id, attn_implementation='flash_attention_2')

### Dataloader-bound: CPU preprocessing and tokenization can't keep up with the GPU

**Likelihood:** common

**Explanation:** If tokenization, data augmentation, or disk I/O for reading training examples happens synchronously on the main process with too few worker processes, the GPU finishes its compute for a batch and then sits idle waiting for the next batch to be prepared, which shows up as low GPU utilization even though the model and hardware are otherwise capable of much higher throughput.

**Fix:** Increase dataloader_num_workers (start around 4-8 and tune based on CPU core count), enable pin_memory=True, and pre-tokenize the dataset once and cache it to disk rather than re-tokenizing on the fly every epoch.

### PCIe-bound CPU or NVMe offloading for models that don't fit in VRAM

**Likelihood:** occasional

**Explanation:** When using DeepSpeed ZeRO-Offload or similar techniques to move optimizer states or parameters to CPU RAM or NVMe storage because the model is too large for available VRAM, every offloaded tensor must move across the PCIe bus each time it's needed, and PCIe bandwidth is orders of magnitude slower than GPU-to-GPU memory bandwidth, making offloaded training inherently much slower regardless of how well it's tuned.

**Fix:** Recognize this as a hardware capacity limitation rather than a tuning problem: the durable fix is more VRAM (larger or additional GPUs) or a smaller/quantized model that fits without offloading, not further offload configuration adjustments.

### Mixed precision or TF32 not enabled

**Likelihood:** rare

**Explanation:** Training in full fp32 precision without enabling TF32 (on Ampere and newer GPUs) or bf16/fp16 mixed precision leaves substantial GPU compute throughput on the table, since these formats allow the GPU's tensor cores to run at significantly higher throughput than full fp32 matrix operations.

**Fix:** Enable bf16 or fp16 mixed precision training explicitly, and ensure torch.backends.cuda.matmul.allow_tf32 = True is set if any fp32 operations remain in your pipeline.

## Diagnostics

### Monitor GPU utilization live during training

**Command:** nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv -l 2

**Interpretation:** Utilization consistently below 70-80% points to a bottleneck outside the GPU itself (dataloader, small batch, or offloading), while consistently high utilization with still-slow wall-clock time suggests the workload itself (attention implementation, precision) needs optimizing.

### Check which attention implementation is active

**Command:** python -c "print(model.config._attn_implementation)"

**Interpretation:** Should read flash_attention_2 or sdpa for good performance. A value of eager means you're leaving meaningful speed and memory efficiency on the table, especially at longer sequence lengths.

### Profile dataloader wait time versus compute time

**Command:** python -c "from torch.profiler import profile; # profile a few training steps and inspect DataLoader time"

**Interpretation:** A large fraction of wall-clock time attributed to data loading rather than model compute confirms a dataloader bottleneck, pointing you toward more workers, caching, or pre-tokenization rather than GPU-side tuning.

### Check whether offloading is active and how much is offloaded

**Command:** python -c "print(ds_config.get('zero_optimization', {}).get('offload_optimizer'))"

**Interpretation:** If offload_optimizer or offload_param is configured, expect meaningfully lower throughput than a fully in-VRAM run; this confirms PCIe-bound offloading as an expected cause rather than a bug to chase further.

## Prevention

- Benchmark a short training run (50-100 steps) with nvidia-smi monitoring before committing to a full multi-hour job, to catch utilization problems early.
- Pre-tokenize and cache datasets to disk once rather than re-tokenizing on the fly every epoch.
- Default new training scripts to flash_attention_2 or sdpa rather than eager attention.
- Size GPU capacity to fit the model and desired batch size without offloading whenever the budget allows, since offloading should be a deliberate fallback, not a default.
- Track tokens/sec as a standard metric across runs so throughput regressions are caught immediately rather than discovered after a slow run completes.

## When To Escalate

If GPU utilization is already high and you've enabled FlashAttention, packing, and mixed precision, but throughput still falls short of what the hardware should deliver, that points to a deeper configuration or parallelism strategy issue (tensor parallelism setup, interconnect bottleneck, or DeepSpeed ZeRO stage choice) worth a focused review. If you're regularly forced into CPU/NVMe offloading because your current GPU fleet can't fit the models you need to fine-tune, that's a capacity planning and hardware decision, not a software tuning one.

## Faq

### What GPU utilization should I expect during fine-tuning?

**Answer:** A well-tuned training run typically keeps GPU utilization at 80-95% for most of each step, with brief dips at batch boundaries. Utilization frequently dropping to 0-30% indicates a dataloader, batch size, or offloading bottleneck rather than the GPU itself being the limiting factor.

### Does FlashAttention actually make a big difference for fine-tuning speed?

**Answer:** Yes, particularly as sequence length grows. FlashAttention avoids materializing the full attention matrix and uses a single fused, memory-efficient kernel, commonly providing significant speedups and memory savings over the default eager attention implementation, especially for context lengths beyond a few thousand tokens.

### Why is CPU offloading so much slower than training fully on GPU?

**Answer:** Offloaded parameters or optimizer states must move across the PCIe bus between CPU RAM and GPU VRAM every time they're needed, and PCIe bandwidth is dramatically lower than GPU memory bandwidth. This makes offloaded training inherently much slower regardless of tuning; it trades speed for the ability to fit a model that otherwise wouldn't fit in available VRAM at all.

### Does increasing batch size always speed up training?

**Answer:** Only up to the point where the GPU's compute units are fully utilized; beyond that, further batch size increases mainly consume more memory without meaningfully improving throughput, and can eventually cause out-of-memory errors. Monitor GPU utilization to know when you've reached diminishing returns.

## Search terms

fine-tuning slow, low GPU utilization training, flash attention training speed, sequence packing fine-tuning, dataloader bottleneck training, PCIe offload slow training, deepspeed zero offload slow, increase training throughput

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json