Why merging a LoRA adapter runs out of memory even though training and inference worked
torch.cuda.OutOfMemoryError: CUDA out of memory during model.merge_and_unload()
Also appears as
- RuntimeError: CUDA out of memory while merging PEFT adapter weights
- OOM when calling merge_and_unload on a QLoRA fine-tuned model
Short answer
Merging a LoRA adapter mathematically requires the base weights in a real-valued (not 4-bit quantized) format so the adapter delta can be added in, which means a QLoRA workflow that trained happily in 4-bit suddenly needs a full fp16/bf16 copy of the base model just to merge, often doubling or more the memory footprint versus either training or inference alone. The fix is to merge on CPU, merge in a lower-footprint dtype, or skip merging entirely by serving the adapter unmerged.
Affects: Any PEFT/LoRA workflow calling merge_and_unload() or an equivalent adapter merge, most common merging QLoRA adapters that were trained against a 4-bit base model
Complete the merge without OOMing
- 1Load the base model in fp16/bf16 (not 4-bit) specifically for the merge step, since merge_and_unload() needs real-valued weights to add the adapter delta correctly.
- 2Perform the merge on CPU if GPU memory is insufficient even in fp16: load the base model with device_map="cpu", merge there, then move the merged model to GPU only after the merge completes.
- 3If GPU merging is required for speed, free all other GPU memory first (unload any training-time optimizer state, clear cached activations) so the full merge has maximum headroom.
- 4Consider skipping the merge entirely: most serving frameworks (vLLM's multi-LoRA serving, PEFT's inference mode) can serve the adapter unmerged, applying it at inference time without ever materializing a merged full-precision copy.
- 5If you must produce a merged checkpoint for deployment simplicity, do it once as an offline batch job on a machine with ample CPU RAM, rather than as part of your live serving pipeline.
How to confirm this is your problem
- Training and inference with the adapter both work fine; only the explicit merge_and_unload() call OOMs
- The failure happens specifically after fine-tuning completes, when producing a deployable merged checkpoint
- The base model was loaded and trained in 4-bit (QLoRA) but the OOM happens when the code attempts to work with full-precision weights for merging
- Available VRAM is enough for either training or inference alone but not for holding a full-precision base model copy simultaneously with the adapter
Root causes and fixes
merge_and_unload requires full-precision base weights, which is more memory than the 4-bit weights used during QLoRA training
The LoRA merge operation computes base_weight + (lora_B @ lora_A) * scale, an arithmetic operation that needs the base weight in a real-valued dtype (fp16/bf16/fp32); it cannot be done directly against 4-bit quantized weights without first dequantizing them. This means a QLoRA setup that trained comfortably in roughly a quarter of the fp16 memory footprint suddenly needs the full fp16 footprint (or more, transiently) just to perform the merge.
Fix: Explicitly load a full-precision copy of the base model for the merge step only, understanding that this step has a genuinely higher memory requirement than either the training or the eventual inference phase.
base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.bfloat16) model = PeftModel.from_pretrained(base_model, ADAPTER_PATH) merged = model.merge_and_unload()
The merge is attempted on the same GPU that still holds training-time state (optimizer, gradients, cached activations)
If the merge code runs in the same process or session right after training without explicitly releasing the optimizer, gradient buffers, and any cached activations from the training run, that leftover memory competes with the full-precision base model copy the merge needs, even though none of it is actually required for the merge operation itself.
Fix: Explicitly delete the optimizer and any training-specific objects, call gc.collect() and torch.cuda.empty_cache(), before loading the full-precision model for merging.
del optimizer, trainer; import gc; gc.collect(); torch.cuda.empty_cache()
Merging is performed on GPU when CPU RAM would comfortably fit the operation with no time pressure
The merge is a one-time, non-latency-sensitive operation typically run offline rather than as part of a live serving path, which makes it a good candidate for CPU execution; running it on GPU by default, simply because that is where the earlier training happened, forces a GPU memory budget on an operation that does not need GPU speed.
Fix: Load the base model and adapter with device_map="cpu" specifically for the merge, accepting a slower but far more memory-headroom-tolerant merge process, then move only the final merged weights to GPU for serving.
base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.bfloat16, device_map="cpu")
Multiple adapters or a large adapter rank are being merged simultaneously, multiplying the transient memory needed during the merge operation
Merging several LoRA adapters in sequence within the same process, or using an unusually high LoRA rank, increases the size of the intermediate lora_B @ lora_A product that must be computed and added to the base weights, adding to the peak transient memory beyond just the base model's own footprint.
Fix: Merge adapters one at a time, releasing memory between merges, and confirm the LoRA rank in use is consistent with your training configuration rather than an inflated default.
The merge script accidentally keeps both the unmerged PeftModel wrapper and the merged output in memory simultaneously
merge_and_unload() is designed to return a new model object; if the calling code retains a reference to the original wrapped PeftModel object alongside the newly merged one (for example by not reassigning the variable, or logging/debugging code holding a reference), both the pre-merge and post-merge weight copies stay resident at once, roughly doubling peak memory unnecessarily.
Fix: Reassign the model variable to the merge output and ensure no other reference to the pre-merge wrapped object remains, explicitly deleting it if needed before proceeding.
model = model.merge_and_unload() del base_model; gc.collect()
Diagnostic commands
Check memory immediately before the merge call
python -c "import torch; print(torch.cuda.memory_allocated()/1e9, 'GB allocated before merge')"
If this is already a large fraction of total VRAM before the merge even starts, leftover training-time state is competing with the merge; free it first rather than assuming the merge itself is unreasonably expensive.
Confirm the base model's loaded dtype and device
python -c "print(next(base_model.parameters()).dtype, next(base_model.parameters()).device)"
Confirm you are loading fp16/bf16 (not accidentally fp32, which doubles memory again) and on the device (cpu or cuda) you intended for the merge step.
Estimate peak merge memory requirement
python -c "params=7e9; print('bf16 base', params*2/1e9, 'GB, plus adapter and transient product, roughly', params*2*1.15/1e9, 'GB peak')"This rough estimate (base weights plus about 15 percent overhead for adapter and intermediate tensors) tells you whether your available VRAM or CPU RAM should comfortably cover the merge before you even attempt it.
Stopping it from happening again
- Plan the merge step's memory requirement (full-precision base model size) separately from training and inference memory during project planning, not as a surprise after training completes.
- Default to CPU merging for any adapter merge that is not on a latency-critical path, since it removes GPU memory as a constraint entirely.
- Consider whether merging is necessary at all; multi-LoRA-capable serving frameworks can often serve adapters unmerged with negligible overhead.
- Clear training-time GPU state explicitly before starting a merge in the same session, rather than assuming it was already released.
When this becomes an architecture problem
If your deployment pipeline requires producing merged full-precision checkpoints for many adapters or many base models as a routine, ongoing operation rather than a one-off step, that throughput need may justify a dedicated CPU-RAM-heavy merge server or workflow separate from your GPU training and serving infrastructure, which is a pipeline architecture decision rather than a one-time memory fix.
Frequently asked questions
Why does merging need more memory than training the LoRA adapter did?
QLoRA training keeps the base model in 4-bit precision throughout, since gradients only need to flow through the small adapter matrices, not the frozen base weights. Merging, however, computes base_weight + adapter_delta as a real arithmetic operation, which requires the base weights in a real-valued dtype like fp16 or bf16, roughly 4x the memory of the 4-bit weights used during training.
Can I merge directly against 4-bit quantized weights to avoid this?
Not directly with standard merge_and_unload(); the operation needs real-valued weights for the addition to be mathematically correct. Some quantization-aware merge techniques exist for specific formats, but the standard and most broadly supported path is to dequantize to fp16/bf16 for the merge, then optionally re-quantize the merged result afterward if you want a quantized deployment artifact.
Do I actually need to merge the adapter at all?
Often not. Frameworks like vLLM support serving LoRA adapters unmerged and applying them at inference time with low overhead, including serving multiple adapters against one base model simultaneously. Merging is mainly useful when you want a single self-contained checkpoint for simplicity or when your serving framework does not support unmerged adapter serving.
Is it safe to merge on CPU instead of GPU?
Yes, correctness is identical since the merge is a straightforward tensor addition; it is simply slower than on GPU. Since merging is typically a one-time offline step rather than part of a latency-sensitive serving path, trading merge speed for the much larger memory headroom of CPU RAM is a reasonable and commonly used approach.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Free ToolQLoRA vs Full Fine-Tuning Cost Calculator
See the GPU memory footprint, GPU-hour requirement, and dollar cost gap between QLoRA and full fine-tuning for the same model size and dataset.
Free ToolLLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Related problems
CUDA out of memory during fine-tuning
Training needs far more memory than inference for the same model because it must hold weights, gradients, optimizer states, and activations simultaneously. AdamW alone adds about 8 bytes per parameter for its two fp32 moment buffers, so full fine-tuning of a 7B model can need 60-70+ GB versus about 14 GB for inference of the same weights. The fix is LoRA/QLoRA to shrink trainable parameters, gradient checkpointing to shrink activation memory, or a paged/8-bit optimizer to shrink optimizer state.
LoRA adapter fails to load onto the base model
A LoRA adapter usually fails to load because it was trained against a different base model than the one you're loading it onto, because the rank or alpha in adapter_config.json doesn't match what was actually trained, because the installed PEFT version is incompatible with how the adapter was saved, or because the adapter directory is missing its config file. Check adapter_config.json's base_model_name_or_path first.
CUDA out of memory when loading an LLM
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
GuideFine-Tuning LLMs On-Prem with Enterprise Data
Fine-tune LLMs on-prem with enterprise data: LoRA vs full fine-tuning, dataset prep, GPU requirements, eval, and when RAG beats tuning altogether.
GuideFine-Tuning Failure Modes: What Actually Goes Wrong
Fine-tuning failure modes that actually derail enterprise projects: catastrophic forgetting, eval overfitting, data leakage, and how to catch each one.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.