# Why merging a LoRA adapter runs out of memory even though training and inference worked

> Source: https://netray.co/troubleshoot/oom-during-lora-adapter-merge
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: On-Prem AI Troubleshooting

## Short answer

Merging a LoRA adapter mathematically requires the base weights in a real-valued (not 4-bit quantized) format so the adapter delta can be added in, which means a QLoRA workflow that trained happily in 4-bit suddenly needs a full fp16/bf16 copy of the base model just to merge, often doubling or more the memory footprint versus either training or inference alone. The fix is to merge on CPU, merge in a lower-footprint dtype, or skip merging entirely by serving the adapter unmerged.

## Error Title

Out of memory when merging a LoRA adapter into the base model

## Error Signature

torch.cuda.OutOfMemoryError: CUDA out of memory during model.merge_and_unload()

## Alternate Signatures

- RuntimeError: CUDA out of memory while merging PEFT adapter weights
- OOM when calling merge_and_unload on a QLoRA fine-tuned model

## Category

gpu-memory

## Stack

- pytorch
- huggingface
- transformers
- accelerate

## Affects

Any PEFT/LoRA workflow calling merge_and_unload() or an equivalent adapter merge, most common merging QLoRA adapters that were trained against a 4-bit base model

## Quick Fix

### Complete the merge without OOMing

**Steps:**

Load the base model in fp16/bf16 (not 4-bit) specifically for the merge step, since merge_and_unload() needs real-valued weights to add the adapter delta correctly.

Perform the merge on CPU if GPU memory is insufficient even in fp16: load the base model with device_map="cpu", merge there, then move the merged model to GPU only after the merge completes.

If GPU merging is required for speed, free all other GPU memory first (unload any training-time optimizer state, clear cached activations) so the full merge has maximum headroom.

Consider skipping the merge entirely: most serving frameworks (vLLM's multi-LoRA serving, PEFT's inference mode) can serve the adapter unmerged, applying it at inference time without ever materializing a merged full-precision copy.

If you must produce a merged checkpoint for deployment simplicity, do it once as an offline batch job on a machine with ample CPU RAM, rather than as part of your live serving pipeline.

## Symptoms

- Training and inference with the adapter both work fine; only the explicit merge_and_unload() call OOMs
- The failure happens specifically after fine-tuning completes, when producing a deployable merged checkpoint
- The base model was loaded and trained in 4-bit (QLoRA) but the OOM happens when the code attempts to work with full-precision weights for merging
- Available VRAM is enough for either training or inference alone but not for holding a full-precision base model copy simultaneously with the adapter

## Causes

### merge_and_unload requires full-precision base weights, which is more memory than the 4-bit weights used during QLoRA training

**Likelihood:** most-common

**Explanation:** The LoRA merge operation computes base_weight + (lora_B @ lora_A) * scale, an arithmetic operation that needs the base weight in a real-valued dtype (fp16/bf16/fp32); it cannot be done directly against 4-bit quantized weights without first dequantizing them. This means a QLoRA setup that trained comfortably in roughly a quarter of the fp16 memory footprint suddenly needs the full fp16 footprint (or more, transiently) just to perform the merge.

**Fix:** Explicitly load a full-precision copy of the base model for the merge step only, understanding that this step has a genuinely higher memory requirement than either the training or the eventual inference phase.

**Commands:**

- base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.bfloat16)
- model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
- merged = model.merge_and_unload()

### The merge is attempted on the same GPU that still holds training-time state (optimizer, gradients, cached activations)

**Likelihood:** common

**Explanation:** If the merge code runs in the same process or session right after training without explicitly releasing the optimizer, gradient buffers, and any cached activations from the training run, that leftover memory competes with the full-precision base model copy the merge needs, even though none of it is actually required for the merge operation itself.

**Fix:** Explicitly delete the optimizer and any training-specific objects, call gc.collect() and torch.cuda.empty_cache(), before loading the full-precision model for merging.

**Commands:**

- del optimizer, trainer; import gc; gc.collect(); torch.cuda.empty_cache()

### Merging is performed on GPU when CPU RAM would comfortably fit the operation with no time pressure

**Likelihood:** common

**Explanation:** The merge is a one-time, non-latency-sensitive operation typically run offline rather than as part of a live serving path, which makes it a good candidate for CPU execution; running it on GPU by default, simply because that is where the earlier training happened, forces a GPU memory budget on an operation that does not need GPU speed.

**Fix:** Load the base model and adapter with device_map="cpu" specifically for the merge, accepting a slower but far more memory-headroom-tolerant merge process, then move only the final merged weights to GPU for serving.

**Commands:**

- base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.bfloat16, device_map="cpu")

### Multiple adapters or a large adapter rank are being merged simultaneously, multiplying the transient memory needed during the merge operation

**Likelihood:** occasional

**Explanation:** Merging several LoRA adapters in sequence within the same process, or using an unusually high LoRA rank, increases the size of the intermediate lora_B @ lora_A product that must be computed and added to the base weights, adding to the peak transient memory beyond just the base model's own footprint.

**Fix:** Merge adapters one at a time, releasing memory between merges, and confirm the LoRA rank in use is consistent with your training configuration rather than an inflated default.

### The merge script accidentally keeps both the unmerged PeftModel wrapper and the merged output in memory simultaneously

**Likelihood:** rare

**Explanation:** merge_and_unload() is designed to return a new model object; if the calling code retains a reference to the original wrapped PeftModel object alongside the newly merged one (for example by not reassigning the variable, or logging/debugging code holding a reference), both the pre-merge and post-merge weight copies stay resident at once, roughly doubling peak memory unnecessarily.

**Fix:** Reassign the model variable to the merge output and ensure no other reference to the pre-merge wrapped object remains, explicitly deleting it if needed before proceeding.

**Commands:**

- model = model.merge_and_unload()
- del base_model; gc.collect()

## Diagnostics

### Check memory immediately before the merge call

**Command:** python -c "import torch; print(torch.cuda.memory_allocated()/1e9, 'GB allocated before merge')"

**Interpretation:** If this is already a large fraction of total VRAM before the merge even starts, leftover training-time state is competing with the merge; free it first rather than assuming the merge itself is unreasonably expensive.

### Confirm the base model's loaded dtype and device

**Command:** python -c "print(next(base_model.parameters()).dtype, next(base_model.parameters()).device)"

**Interpretation:** Confirm you are loading fp16/bf16 (not accidentally fp32, which doubles memory again) and on the device (cpu or cuda) you intended for the merge step.

### Estimate peak merge memory requirement

**Command:** python -c "params=7e9; print('bf16 base', params*2/1e9, 'GB, plus adapter and transient product, roughly', params*2*1.15/1e9, 'GB peak')"

**Interpretation:** This rough estimate (base weights plus about 15 percent overhead for adapter and intermediate tensors) tells you whether your available VRAM or CPU RAM should comfortably cover the merge before you even attempt it.

## Prevention

- Plan the merge step's memory requirement (full-precision base model size) separately from training and inference memory during project planning, not as a surprise after training completes.
- Default to CPU merging for any adapter merge that is not on a latency-critical path, since it removes GPU memory as a constraint entirely.
- Consider whether merging is necessary at all; multi-LoRA-capable serving frameworks can often serve adapters unmerged with negligible overhead.
- Clear training-time GPU state explicitly before starting a merge in the same session, rather than assuming it was already released.

## When To Escalate

If your deployment pipeline requires producing merged full-precision checkpoints for many adapters or many base models as a routine, ongoing operation rather than a one-off step, that throughput need may justify a dedicated CPU-RAM-heavy merge server or workflow separate from your GPU training and serving infrastructure, which is a pipeline architecture decision rather than a one-time memory fix.

## Faq

### Why does merging need more memory than training the LoRA adapter did?

**Answer:** QLoRA training keeps the base model in 4-bit precision throughout, since gradients only need to flow through the small adapter matrices, not the frozen base weights. Merging, however, computes base_weight + adapter_delta as a real arithmetic operation, which requires the base weights in a real-valued dtype like fp16 or bf16, roughly 4x the memory of the 4-bit weights used during training.

### Can I merge directly against 4-bit quantized weights to avoid this?

**Answer:** Not directly with standard merge_and_unload(); the operation needs real-valued weights for the addition to be mathematically correct. Some quantization-aware merge techniques exist for specific formats, but the standard and most broadly supported path is to dequantize to fp16/bf16 for the merge, then optionally re-quantize the merged result afterward if you want a quantized deployment artifact.

### Do I actually need to merge the adapter at all?

**Answer:** Often not. Frameworks like vLLM support serving LoRA adapters unmerged and applying them at inference time with low overhead, including serving multiple adapters against one base model simultaneously. Merging is mainly useful when you want a single self-contained checkpoint for simplicity or when your serving framework does not support unmerged adapter serving.

### Is it safe to merge on CPU instead of GPU?

**Answer:** Yes, correctness is identical since the merge is a straightforward tensor addition; it is simply slower than on GPU. Since merging is typically a one-time offline step rather than part of a latency-sensitive serving path, trading merge speed for the much larger memory headroom of CPU RAM is a reasonable and commonly used approach.

## Search terms

oom merging lora adapter, merge_and_unload out of memory, qlora merge cuda out of memory, peft merge adapter memory error, lora adapter merge cpu, cannot merge 4bit lora adapter

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json