Why your fine-tuning run is so much slower than it should be, and how to speed it up

Modern GPUs have far more parallel compute capacity than a very small batch can occupy; with a tiny batch, most of the GPU's tensor cores sit idle during each forward and backward pass, and per-step overhead (kernel launches, synchronization) becomes a larger fraction of total time relative to actual compute, making each step slower than it needs to be for the work performed.

Fix: Increase per_device_train_batch_size as far as VRAM allows, and use gradient_accumulation_steps to reach your desired effective batch size without exceeding memory, which recovers most of the throughput lost to a too-small batch.

Without packing, a batch of variable-length examples is padded to the length of its longest member, meaning shorter examples waste compute processing padding tokens that produce no useful gradient. On datasets with high length variance, this can mean a large fraction of every batch's compute is spent on tokens that contribute nothing to learning.

Fix: Enable sequence packing (TRL's SFTTrainer supports packing=True) which concatenates multiple short examples into fixed-length sequences separated by EOS tokens, dramatically reducing wasted compute on padding for datasets with variable-length examples.

The default eager attention implementation computes the full attention matrix with several separate, unfused GPU operations, each with its own memory read/write overhead. FlashAttention and similar fused kernels compute the same mathematical result with a single, memory-efficient fused kernel, which is significantly faster and also uses less memory, especially as sequence length grows.

Fix: Load the model with attn_implementation='flash_attention_2' (or 'sdpa' as a solid built-in alternative if FlashAttention installation isn't feasible) instead of leaving the default eager implementation active.

model = AutoModelForCausalLM.from_pretrained(model_id, attn_implementation='flash_attention_2')

If tokenization, data augmentation, or disk I/O for reading training examples happens synchronously on the main process with too few worker processes, the GPU finishes its compute for a batch and then sits idle waiting for the next batch to be prepared, which shows up as low GPU utilization even though the model and hardware are otherwise capable of much higher throughput.

Fix: Increase dataloader_num_workers (start around 4-8 and tune based on CPU core count), enable pin_memory=True, and pre-tokenize the dataset once and cache it to disk rather than re-tokenizing on the fly every epoch.

When using DeepSpeed ZeRO-Offload or similar techniques to move optimizer states or parameters to CPU RAM or NVMe storage because the model is too large for available VRAM, every offloaded tensor must move across the PCIe bus each time it's needed, and PCIe bandwidth is orders of magnitude slower than GPU-to-GPU memory bandwidth, making offloaded training inherently much slower regardless of how well it's tuned.

Fix: Recognize this as a hardware capacity limitation rather than a tuning problem: the durable fix is more VRAM (larger or additional GPUs) or a smaller/quantized model that fits without offloading, not further offload configuration adjustments.

Training in full fp32 precision without enabling TF32 (on Ampere and newer GPUs) or bf16/fp16 mixed precision leaves substantial GPU compute throughput on the table, since these formats allow the GPU's tensor cores to run at significantly higher throughput than full fp32 matrix operations.

Fix: Enable bf16 or fp16 mixed precision training explicitly, and ensure torch.backends.cuda.matmul.allow_tf32 = True is set if any fp32 operations remain in your pipeline.