100 documented failures

On-Prem AI Troubleshooting Library

Every error here has stalled a real deployment. Each page gives you the short answer first, then the ranked root causes, the diagnostic commands to confirm which one you hit, and how to stop it recurring.

Written by engineers who run open-weight models on customer hardware, including air-gapped networks in aerospace, defense and regulated manufacturing.

GPU Memory & OOM10 fixes

CUDA out-of-memory errors, VRAM exhaustion, and memory that never gets released.

CUDA out of memory when loading an LLM

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 23.69 GiB total capacity; 21.84 GiB already allocated; 412.00 MiB free; 22.06 GiB reserved in total by PyTorch)

This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.

View the fix

CUDA out of memory during fine-tuning

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.24 GiB. GPU 0 has a total capacity of 79.15 GiB of which 623.12 MiB is free

Training needs far more memory than inference for the same model because it must hold weights, gradients, optimizer states, and activations simultaneously. AdamW alone adds about 8 bytes per parameter for its two fp32 moment buffers, so full fine-tuning of a 7B model can need 60-70+ GB versus about 14 GB for inference of the same weights. The fix is LoRA/QLoRA to shrink trainable parameters, gradient checkpointing to shrink activation memory, or a paged/8-bit optimizer to shrink optimizer state.

View the fix

vLLM fails to start because there is not enough memory for the KV cache

ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine

vLLM reserves a fixed pool of GPU memory (gpu_memory_utilization, default 0.9) for weights plus KV cache, and if the weights already consume most of that budget there is nothing left for even one sequence's KV cache blocks. The fix is to raise gpu_memory_utilization toward the physical limit, lower max_model_len so each sequence's KV cache is smaller, or serve a quantized checkpoint so more of the budget is available for cache.

View the fix

GPU memory stays full after inference finishes

nvidia-smi still shows the process holding X GiB of GPU memory after the script has finished running or the request has completed

This is expected PyTorch behavior, not a leak: the caching allocator keeps freed GPU memory reserved for future allocations instead of returning it to the driver, so nvidia-smi shows the process's total reserved memory rather than what is actually in use. The real leak to check for is a growing number across requests (Python references keeping tensors alive), not a single high plateau after one inference call.

View the fix

CUDA out of memory even though nvidia-smi shows free VRAM

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU 0 has a total capacity of 24.00 GiB of which 3.21 GiB is free

This almost always means memory fragmentation: the allocator has enough total free memory but no single contiguous block large enough for the requested allocation, because the address space is broken into many small free-and-used segments from prior allocations of different sizes. The fix is enabling PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, reducing allocation size variability, or restarting the process to reset the address space.

View the fix

The model is too large to fit on a single GPU

ValueError: The model size exceeds the available GPU memory. Consider using a smaller model, quantization, or model parallelism

This is a hard arithmetic ceiling, not a bug: a 70B parameter model needs about 140 GB in bf16, which exceeds every single current GPU. There are exactly three valid fixes: quantize the weights to reduce bytes-per-parameter, shard the model across multiple GPUs with tensor or pipeline parallelism, or choose a smaller model that fits your single-GPU budget at the precision you need.

View the fix

PyTorch GPU memory fragmentation causing intermittent OOM

RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU 0; ...; Y MiB free; ...); see documentation for PYTORCH_CUDA_ALLOC_CONF

PyTorch explicitly detects and reports fragmentation in this error, pointing you at PYTORCH_CUDA_ALLOC_CONF for a reason: the caching allocator's memory is split into segments sized for past allocations, and a new allocation that does not match any free segment's size fails even with adequate total free memory. Setting expandable_segments:True and normalizing input shapes are the two highest-leverage fixes.

View the fix

How to reduce VRAM usage for LLM inference

Need to reduce GPU memory usage for LLM inference without a specific error, or: torch.cuda.OutOfMemoryError under production load that a smaller test load did not trigger

VRAM usage during inference comes from three independent budgets: model weights (params x bytes/param), KV cache (scales with batch x context length), and activation/workspace memory. Reducing usage means attacking whichever budget dominates: quantize weights to cut the largest fixed cost, cap max context length and concurrency to cut the largest variable cost, and use an efficient attention/serving backend to minimize workspace overhead.

View the fix

LLM inference is extremely slow after enabling CPU offload

Model runs without an out-of-memory error after enabling device_map="auto" with CPU offload, but generation throughput drops to a fraction of a token per second

CPU offload trades memory capacity for speed because every offloaded layer's weights must cross the PCIe bus (typically 16-64 GB/s) on every forward pass, versus terabytes-per-second on-GPU HBM bandwidth; this is not a bug, it is the fundamental cost of running more model than your GPU can hold. The real fix is usually to reduce how much needs to be offloaded (quantize first) rather than trying to make offloading itself faster.

View the fix

Out of memory when merging a LoRA adapter into the base model

torch.cuda.OutOfMemoryError: CUDA out of memory during model.merge_and_unload()

Merging a LoRA adapter mathematically requires the base weights in a real-valued (not 4-bit quantized) format so the adapter delta can be added in, which means a QLoRA workflow that trained happily in 4-bit suddenly needs a full fp16/bf16 copy of the base model just to merge, often doubling or more the memory footprint versus either training or inference alone. The fix is to merge on CPU, merge in a lower-footprint dtype, or skip merging entirely by serving the adapter unmerged.

View the fix
Installation & Environment10 fixes

Driver, CUDA, and Python dependency failures that block you before inference starts.

CUDA version mismatch between PyTorch and the system driver

RuntimeError: The NVIDIA driver on your system is too old (found version 11000). Please update your GPU driver by downloading and installing a new version

PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.

View the fix

nvidia-smi command not found or fails to communicate with the driver

bash: nvidia-smi: command not found

nvidia-smi not found or unable to communicate almost always means the NVIDIA kernel module never loaded, and the two most common reasons are that the driver was never installed, or the driver is installed but Secure Boot is blocking the unsigned kernel module from loading. WSL2 users hit a different variant: the driver must be installed on the Windows host, never inside the Linux guest.

View the fix

vLLM fails to install or import due to torch and CUDA mismatches

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. vllm 0.6.3 requires torch==2.4.0, but you have torch 2.5.1 which is incompatible

vLLM ships prebuilt wheels compiled against a specific, exact torch and CUDA version, and it uses custom CUDA kernels that only work with that pairing. Most install failures happen because torch was already installed separately, or an existing environment has an incompatible CUDA toolkit, so the fix is almost always a fresh virtual environment where pip resolves vLLM and its exact torch dependency together in one pass.

View the fix

FlashAttention install fails during compilation or gets killed

ninja: build stopped: subcommand failed

FlashAttention's pip install compiles CUDA kernels from source unless an exact prebuilt wheel exists for your torch, CUDA, Python, and C++ ABI combination, and that compilation is extremely RAM-hungry per parallel job. The build gets silently OOM-killed on machines without enough memory unless you limit MAX_JOBS, and separately fails if your CUDA toolkit does not match the version torch itself was built against.

View the fix

bitsandbytes cannot find or detect a working CUDA setup

RuntimeError: CUDA Setup failed despite GPU being available

bitsandbytes needs to locate the exact CUDA runtime shared library at import time and load a matching precompiled binary for that version. The setup fails most often because LD_LIBRARY_PATH points at a different CUDA installation than the one it detected, or because an older bitsandbytes version could not auto-detect the GPU correctly. Upgrading to the latest bitsandbytes and running its built-in diagnostic resolves the majority of cases.

View the fix

torch.cuda.is_available() returns False even though a GPU is present

torch.cuda.is_available() returns False

torch.cuda.is_available() returning False almost always means either the installed torch wheel is a CPU-only build, or the process cannot see the GPU due to a driver, container, or environment variable problem. Checking torch.version.cuda for None immediately tells you whether you have a CPU-only wheel, which is the single most common cause and the fastest thing to rule out.

View the fix

NVIDIA driver installation fails on Ubuntu

modprobe: ERROR: could not insert 'nvidia': Key was rejected by service

The single most common reason NVIDIA driver installation fails on Ubuntu is Secure Boot rejecting the unsigned or self-signed kernel module at load time, since most machines now ship with Secure Boot enabled out of the box. The fix is enrolling the MOK key the installer generates, or disabling Secure Boot in the BIOS, then clearing any lingering nouveau or mixed-install conflicts before rebooting.

View the fix

cuDNN version mismatch or library loading error in PyTorch

RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (8, 9, 0) but linked against (8, 5, 0)

PyTorch wheels bundle their own cuDNN version internally, so a separately installed system-wide cuDNN is usually unnecessary and often the actual cause of this error. When LD_LIBRARY_PATH exposes a different cuDNN version than the one torch was compiled against, torch loads the wrong one at runtime and throws a version incompatibility error. Removing the manual cuDNN path and letting torch use its bundled copy resolves most cases.

View the fix

Python dependency conflicts across transformers, tokenizers, and numpy

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. transformers 4.46.0 requires tokenizers<0.21,>=0.20, but you have tokenizers 0.19.1 which is incompatible

The Hugging Face stack (transformers, tokenizers, accelerate) is released in tightly coupled lockstep versions, so upgrading one package independently over time leaves combinations that were never tested together and break silently at import or runtime. The fix is resolving the entire stack in a single pip install pass from a fresh virtual environment, guided by a pinned requirements.txt, rather than incrementally patching individual packages.

View the fix

gcc or g++ version rejected while building CUDA extensions

error: unsupported GNU version! gcc versions later than 12 are not supported

Every CUDA toolkit release only supports compiling with a specific range of gcc and g++ major versions, and nvcc explicitly rejects anything outside that range rather than risk generating broken code. This most often surfaces on recently released Linux distributions whose default gcc is newer than what an older, already-installed CUDA toolkit supports, and the fix is installing a supported older gcc/g++ version alongside the default and pointing CC and CXX at it for the build.

View the fix
Inference Serving10 fixes

vLLM, SGLang, and OpenAI-compatible endpoint failures in production serving.

vLLM: model's max seq len is larger than the KV cache can hold

ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (16384). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

vLLM preallocates a fixed KV cache pool sized by gpu_memory_utilization and refuses to start a context length whose worst case (batch x max sequence length) doesn't fit in that pool. Fix it by raising --gpu-memory-utilization toward 0.9-0.95, lowering --max-model-len to what you actually need, or adding a GPU/quantizing weights to leave more headroom for cache.

View the fix

vLLM server won't start (port in use, auth, VRAM, or unsupported architecture)

ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use

vLLM server startup failures collapse into four buckets: the port is already bound by another process, Hugging Face auth is missing or expired for a gated repo, there isn't enough free VRAM for the requested model and context, or the installed vLLM version doesn't yet support the model's architecture. Read the last traceback line, not just the top, to tell them apart.

View the fix

vLLM tensor-parallel-size must divide the number of attention heads

ValueError: Total number of attention heads (32) must be divisible by tensor parallel size (3)

vLLM shards attention heads and KV heads evenly across GPUs for tensor parallelism, so --tensor-parallel-size must be a divisor of the model's head count and must equal the number of GPUs vLLM can actually see. Pick a TP size from the model's valid divisors, commonly 1, 2, 4, or 8, and make sure CUDA_VISIBLE_DEVICES exposes exactly that many GPUs.

View the fix

vLLM runs out of memory during startup, before serving any requests

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate X GiB (GPU 0; Y GiB total capacity; Z GiB already allocated)

vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.

View the fix

vLLM throughput is far below expected tokens per second

Avg generation throughput: 12.3 tokens/s (far below expected for the GPU and model combination)

Low vLLM throughput almost always traces back to max-num-seqs capping concurrent batching too low, chunked prefill being disabled so long prompts stall the decode batch, an unintended dtype that doesn't use tensor cores efficiently, CPU-bound tokenization or preprocessing, or requests spilling into swap. Diagnose with nvidia-smi and vLLM's own throughput logs before changing anything.

View the fix

vLLM OpenAI-compatible API returns 404 Not Found

{"object":"error","message":"The model `gpt-3.5-turbo` does not exist.","type":"NotFoundError","code":404}

This 404 is almost always a client-side mismatch, not a server bug: either the request hit the wrong route, such as missing the /v1 prefix, or the model field in the request body doesn't match the exact served-model-name (or default model repo id) vLLM registered at startup. Fix the URL and model name to match what /v1/models actually reports.

View the fix

vLLM error: no chat template found for this model

ValueError: As of transformers v4.44, default chat template is no longer allowed, so vLLM's chat api requires you to specify a chat template if the tokenizer does not define one

The /v1/chat/completions endpoint needs a Jinja chat template to turn a messages array into the model's expected prompt format, and base pretrained checkpoints plus some older fine-tunes simply don't ship one. Fix it by supplying --chat-template pointing at a template file matching the model family, or by switching to the raw /v1/completions endpoint with a manually formatted prompt.

View the fix

vLLM ignores tool or function calls, or returns them as plain text

Assistant message content contains literal function-call JSON text instead of a structured tool_calls field

Unlike the hosted OpenAI API, vLLM does not enable tool or function calling by default. You must launch with --enable-auto-tool-choice plus a --tool-call-parser matching your specific model family, and the model itself must have been trained to emit tool-call syntax its parser recognizes. Without both pieces, requests either error out or the model just writes the function call as plain text in its response content.

View the fix

vLLM keeps generating past the end of turn instead of stopping

Model output continues generating text, often repeating or hallucinating a new user turn, instead of stopping at the expected end of turn

Runaway generation almost always means the token the model actually emits to end a turn doesn't match what vLLM is told to stop on, either because generation_config.json's eos_token_id is stale, a custom stop string wasn't passed in the request, or a fine-tune introduced a new end-of-turn token the base config doesn't know about. Fix it by explicitly passing the correct stop token id or stop strings rather than relying on defaults.

View the fix

vLLM multi-LoRA serving fails to load or apply an adapter

ValueError: LoRA rank X is greater than max_lora_rank Y

Multi-LoRA serving in vLLM requires the server to be launched with --enable-lora plus explicit capacity flags such as max-lora-rank, max-loras, and max-cpu-loras sized to your actual adapters, and each adapter must be registered by name with --lora-modules so requests can reference it. Mismatches between an adapter's real rank or count and what the server was configured for produce hard failures rather than silent truncation.

View the fix
Local Runtimes10 fixes

Ollama, llama.cpp, and GGUF problems on workstations and edge boxes.

Ollama connection refused when calling the API

curl: (7) Failed to connect to 127.0.0.1 port 11434: Connection refused

Ollama connection refused almost always means the ollama serve process is not running, is listening on a different interface than expected, or is bound to 127.0.0.1 while your client is calling it from another host or container. Start or restart the service, confirm it is listening on 11434, and if you need remote access set OLLAMA_HOST to 0.0.0.0 explicitly.

View the fix

Ollama says a model was not found

Error: model 'llama3.2' not found, try pulling it first

Ollama model not found means the exact tag you requested, including the version suffix after the colon, does not exist locally or in the registry. Either the tag has a typo, the model was never pulled, or a custom Modelfile references a FROM path that does not resolve on this machine. Run ollama list to see what is actually installed, then pull or fix the Modelfile.

View the fix

Ollama not using the GPU, falls back to CPU

level=WARN source=gpu.go msg="no compatible GPUs were discovered"

Ollama falls back to CPU silently, without an obvious error, most often because the NVIDIA driver is missing inside a container, the model does not fit in available VRAM so Ollama offloads some or all layers to system RAM, or the GPU simply is not visible to the process. Check ollama ps for the CPU/GPU split and nvidia-smi for driver visibility before assuming the model itself is slow.

View the fix

Ollama generates tokens very slowly

total duration: 45.2s, eval rate: 1.84 tokens/s

Slow Ollama generation almost always traces back to the model running partly or fully on CPU instead of GPU, either because it does not fit in VRAM, the GPU was never detected, or n_gpu_layers is set too low in a llama.cpp-based config. Check the eval rate in verbose output and the CPU/GPU split in ollama ps before tuning anything else.

View the fix

Ollama runs out of memory loading or running a model

Error: model requires more system memory (5.4 GiB) than is available (3.9 GiB)

Ollama out of memory happens when the model's weights plus its KV cache exceed either GPU VRAM or system RAM, and the OOM killer or CUDA allocator terminates the process. The fix depends on which resource is exhausted: reduce quantization or context length for VRAM limits, or reduce concurrent model loads and context for system RAM limits.

View the fix

Configuring Ollama for remote access safely

curl: (28) Failed to connect to 192.168.1.50 port 11434: Connection timed out

Ollama binds to 127.0.0.1 by default, which blocks any connection from another machine, container, or network. Setting OLLAMA_HOST=0.0.0.0 makes it listen on all interfaces, but Ollama's API has no built-in authentication, so any remote-accessible instance must sit behind a reverse proxy or VPN that adds authentication and TLS before it is exposed beyond a fully trusted local network.

View the fix

llama.cpp fails to load a GGUF model file

error loading model: unexpectedly reached end of file

A GGUF load failure in llama.cpp is almost always one of three things: the file was truncated or corrupted during download, the file uses a quantization or metadata format newer than your llama.cpp build supports, or the model was split into multiple GGUF shards and only some of them were downloaded. Verify the file size and checksum first, then check your llama.cpp version against the GGUF version the file requires.

View the fix

llama.cpp fails to build from source (CMake, CUDA, Metal)

CMake Error: CUDA_TOOLKIT_ROOT_DIR not found

Most llama.cpp build failures come from missing or mismatched CUDA toolkit installs, wrong cmake backend flags (forgetting to enable GGML_CUDA or GGML_METAL), or a host compiler version newer than the CUDA toolkit supports. Confirm the toolkit is installed and on PATH, pass the correct backend flag for your hardware, and match your compiler version to what your CUDA version officially supports.

View the fix

GGUF quantization damages model output quality

quantized model produces repetitive, incoherent, or nonsensical output

GGUF quantization below roughly 4 bits per weight (Q2_K, Q3_K_S) trades accuracy aggressively for size and speed, and on smaller models or reasoning-heavy tasks this shows up as incoherent, repetitive, or factually unreliable output. Q4_K_M and Q5_K_M are the widely used sweet spots that keep most of the quality of the full-precision model while still cutting memory roughly in half or more, and Q2/Q3 should be reserved for cases where fitting in VRAM matters more than output quality.

View the fix

Ollama silently truncates earlier conversation turns

model responses ignore earlier context after a long conversation, as if it forgot the beginning

Ollama silently truncates conversation history once the total tokens exceed num_ctx, which defaults to a relatively small value in many client configurations, dropping the oldest turns without any error or warning to the user or the calling application. The fix is to explicitly set num_ctx to a value that matches both your actual conversation length needs and the model's supported maximum, and to monitor token counts rather than assuming the full history is always sent.

View the fix
Model Loading & Weights10 fixes

Gated repos, corrupt checkpoints, tokenizer mismatches, and air-gapped loading.

401/403 Unauthorized pulling a gated model from HuggingFace

OSError: You are trying to access a gated repo. Make sure to have access to it at https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct

A 401 or 403 on a gated HuggingFace repo means the request reached the Hub but was rejected for authorization, not because the model does not exist. The three real causes are: you have not accepted the model's license on the web page with the account tied to your token, the token exists but was created without read access to gated repos, or the token is valid but was never actually passed to the download call (no HF_TOKEN in the environment, no login run). Fix by accepting the license, generating a token with the right scope, and exporting it where the client library will find it.

View the fix

safetensors header too large or invalid header error loading model weights

safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

A safetensors HeaderTooLarge or InvalidHeaderDeserialization error means the file's first bytes are not the expected binary length-prefixed JSON header, almost always because the file on disk is not the actual model weights but a truncated partial download or a small Git LFS pointer text file that was never smudged into the real binary. The fix is to verify the file size matches what the Hub reports and re-download it properly, either with huggingface_hub or with git lfs pull, not to try to repair the file in place.

View the fix

Model type or architecture not recognized when loading a new model

ValueError: The checkpoint you are trying to load has model type `xyz` but Transformers does not recognize this architecture

An unrecognized model type or architecture error means the checkpoint's config.json declares a model_type or architectures field that your installed version of transformers or vLLM has never heard of, because that model's implementation code was added in a later library release than the one you have installed. The fix is almost always to upgrade the serving library (transformers, vLLM, or both) to a version released on or after that model's launch, not to modify the checkpoint.

View the fix

Tokenizer mismatch causing garbage output or wrong special tokens

The model outputs garbled or repetitive text even though the weights loaded successfully

Garbage or repetitive output with weights that loaded without error almost always means the tokenizer does not exactly match the model, either because the vocabulary size or token IDs differ from what the model was trained on, or because special tokens like BOS/EOS/chat markers are mapped to the wrong IDs. The fix is to always load the tokenizer from the exact same repo and revision as the model weights, never mix files between repos, and verify the chat template and special token IDs match the model card.

View the fix

trust_remote_code required, and the security tradeoff behind that error

ValueError: Loading this model requires you to execute the configuration file in that repo on your local machine. You can inspect the repository content at ... You can inspect the repository content at ... and set the option `trust_remote_code=True` to remove this error.

This error is transformers deliberately refusing to run arbitrary Python code from a model repository without explicit consent, because loading such a model means executing the repository author's custom modeling_*.py file directly in your process with full permissions, not just deserializing tensor data. Passing trust_remote_code=True removes the error but does not remove the risk; in a regulated or air-gapped environment the correct approach is to review that code yourself, vendor a pinned copy of it, and only then load with trust_remote_code=True against your reviewed copy.

View the fix

HuggingFace model download is extremely slow or stalls partway through

Download progress bar stops advancing and stays stuck at a partial percentage for minutes

Slow or stalled HuggingFace downloads are usually caused by huggingface_hub's default transfer path not using parallel chunked downloads, a corporate proxy or firewall throttling or dropping long-lived connections, or genuinely insufficient bandwidth for a hundreds-of-gigabytes model. Enable hf_transfer for a much faster Rust-based parallel downloader, rely on the client's built-in resume behavior rather than restarting from zero, and for regulated or air-gapped sites, download once and mirror internally instead of pulling repeatedly over the internet.

View the fix

Corrupted model checkpoint fails to load or loads with garbage weights

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

A corrupted checkpoint means the bytes on disk do not match the original artifact the model author published, whether from an interrupted download, a bad copy between systems, disk-level bit rot, or a failed write during a save operation. There is no reliable way to repair a corrupted deep learning checkpoint; the fix is always to verify the file against a known-good hash or size and re-obtain a clean copy, then build a verification step into your pipeline so the same failure does not silently recur.

View the fix

Model loading fails offline or in an air-gapped environment despite having local files

OSError: We couldn't connect to 'https://huggingface.co' to load this model and it looks like ... is not the path to a directory containing a config.json file

Passing a local path to from_pretrained does not guarantee an offline load, because transformers and related libraries (tokenizers, some model configs, auto-mapping code) can still issue background network calls to check for updates, fetch a referenced remote component, or resolve auto_map entries that point back at the original HuggingFace repo. The fix is to set HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 explicitly, use a complete local snapshot directory (not just the weights file), and verify no config field still references a remote repo ID.

View the fix

SentencePiece tokenizer conversion or loading error

ImportError: You need to install sentencepiece to use ... tokenizer: pip install sentencepiece

SentencePiece tokenizer errors come from three distinct causes: the sentencepiece Python package is simply not installed, the tokenizer.model protobuf file is missing, truncated, or from the wrong model entirely, or the automatic slow-to-fast tokenizer conversion process failed and needs to fall back explicitly. Install sentencepiece, verify tokenizer.model is present and matches the model's repo exactly, and use the model's own fast tokenizer files when available instead of relying on on-the-fly conversion.

View the fix

Model revision or commit not found when pinning a specific version

huggingface_hub.utils._errors.RevisionNotFoundError: 404 Client Error. Revision Not Found for url

A revision not found error means the exact commit hash, branch name, or tag you specified does not exist in that repository, most often because it was copied from a different repo, mistyped, or refers to a commit that was later force-pushed away or a tag that was deleted or renamed by the repo maintainer. Fix it by listing the repo's actual available revisions and re-pinning to a real, current one, and build your own immutable mirror if you need guarantees beyond what the source repo's maintainers commit to preserving.

View the fix
Fine-Tuning & Training10 fixes

Training runs that diverge, stall, overfit, or produce a worse model than the base.

Training loss not decreasing during fine-tuning

loss: 2.31, 2.30, 2.31, 2.30, 2.31 (loss flat across steps, not converging)

Training loss that stays flat is most often caused by a LoRA adapter that targets the wrong modules (missing q_proj/k_proj/v_proj/o_proj), a learning rate that is too low for LoRA or too high and bouncing, or labels that were never masked so the model is trying to learn the prompt tokens as if they were random noise. Check target_modules first, then the label mask, then the learning rate.

View the fix

Model forgets general knowledge after fine-tuning (catastrophic forgetting)

model answers basic general-knowledge questions incorrectly after fine-tuning, though it did fine before

Catastrophic forgetting happens when fine-tuning overwrites the general capabilities the base model already had, and it is driven by a learning rate that is too high, too many epochs over a narrow dataset, or a rank that gives the adapter too much capacity relative to the data. Fix it by lowering the LoRA rank, adding a learning rate decay schedule, mixing in general-purpose replay data, or simply training fewer epochs.

View the fix

LoRA adapter fails to load onto the base model

ValueError: Target modules {'q_proj', 'v_proj'} not found in the base model

A LoRA adapter usually fails to load because it was trained against a different base model than the one you're loading it onto, because the rank or alpha in adapter_config.json doesn't match what was actually trained, because the installed PEFT version is incompatible with how the adapter was saved, or because the adapter directory is missing its config file. Check adapter_config.json's base_model_name_or_path first.

View the fix

Loss becomes NaN during fine-tuning

loss: nan

NaN loss during training is most often caused by fp16 numeric overflow in gradients or activations, which bf16 avoids because of its wider exponent range. Other common causes are a learning rate spike (especially right after warmup), a small number of corrupt or malformed training samples, and unsafe division or log operations in a custom loss function. Switch to bf16 first if your hardware supports it, then check for corrupt samples and unstable LR.

View the fix

Gradient checkpointing errors during fine-tuning

UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly

Gradient checkpointing errors during fine-tuning almost always come from three sources: the use_reentrant parameter left unset (it now must be explicit and False is usually correct for transformer models), an attention implementation that isn't fully compatible with checkpointing's re-computation approach, or leaving use_cache=True enabled while checkpointing is on, which conflicts because checkpointing recomputes the forward pass and a live KV cache assumes it won't be recomputed. Set use_reentrant=False and use_cache=False together.

View the fix

Training dataset format errors during fine-tuning

KeyError: 'text' (or 'messages', 'prompt', 'completion')

Dataset format errors happen because the trainer expects a specific schema (either a messages list of role/content dicts, or a prompt/completion pair, or a single text field) and your JSONL doesn't match it, because samples are missing an EOS token so the model never learns to stop generating, or because a fixed max_length silently truncates long examples and cuts off labels partway through the intended response. Confirm your exact schema against what SFTTrainer or your data collator expects before training.

View the fix

Tokenizer padding and truncation errors during training

ValueError: Asking to pad but the tokenizer does not have a padding token

Padding errors during training happen because many base models ship without a defined pad token at all, because the common workaround of setting pad_token equal to eos_token teaches the model that end-of-sequence and padding look identical (so it can learn to never emit a real stop signal), or because left-padding is used when right-padding was needed for the training collator, or vice versa. Add a distinct pad token when possible, and always right-pad for causal LM training.

View the fix

Fine-tuning is much slower than expected

training throughput far below expected tokens/sec or steps/sec for the GPU and model size

Slow fine-tuning is usually a GPU sitting idle waiting on the CPU dataloader, a batch size too small to keep the GPU's compute units busy, missing FlashAttention or another fused kernel so attention runs in a much slower fallback path, or CPU/NVMe offloading (for models too large to fit fully in VRAM) that is inherently bound by PCIe bandwidth rather than GPU compute. Check GPU utilization with nvidia-smi first to know which of these you're actually facing.

View the fix

LLM overfits on a small fine-tuning dataset

train_loss keeps dropping toward zero while eval_loss rises after a few epochs

Overfitting on a small dataset shows up as training loss continuing to drop while evaluation loss rises after a few epochs, meaning the model is memorizing training examples rather than learning generalizable patterns. Fix it by holding out a genuine evaluation split, tracking eval loss every epoch, stopping early at the point where eval loss stops improving, and reducing epochs, rank, or learning rate if the crossover happens very early.

View the fix

Fine-tuned model scores worse than the base model

fine-tuned model underperforms the base model on the same evaluation benchmark

A fine-tuned model that scores worse than its own base model almost always means the evaluation is contaminated (test examples leaked into training) or unfair (a genuinely improved model getting compared under a broken harness), the inference prompt format doesn't match the exact format used during training, or the fine-tuning process optimized for surface style and tone rather than the underlying capability the benchmark actually measures. Check inference prompt formatting first, since it is the single most common cause.

View the fix
Multi-GPU & Distributed10 fixes

NCCL errors, multi-node hangs, tensor parallelism, and interconnect problems.

NCCL error during multi-GPU training or inference

RuntimeError: NCCL error: unhandled system error, NCCL version 2.18.1

An NCCL error during multi-GPU training or inference is almost always a symptom of a rank that crashed, a version mismatch across processes, or bad GPU topology, not a bug in NCCL itself. Enable NCCL_DEBUG=INFO first and read the per-rank logs before touching timeouts or retry logic.

View the fix

NCCL collective operation timeout during distributed training

torch.distributed.DistBackendError: NCCL communicator was aborted. Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1842, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1800000 milliseconds

An NCCL timeout means one or more ranks did not reach a collective operation (all-reduce, broadcast, all-gather) within the configured window, almost always because a straggler rank is slow or stuck, not because NCCL is malfunctioning. Raising NCCL_TIMEOUT can mask the symptom, but the durable fix is finding and removing the straggler: a data loading stall, an OOM-crashed rank, or a checkpoint write blocking one process.

View the fix

Tensor parallelism fails because the model does not split evenly across GPUs

AssertionError: Number of attention heads (32) must be divisible by tensor parallel size (6)

Tensor parallelism fails when the chosen degree does not evenly divide the model's attention heads, and often its key/value heads and hidden size, so the framework cannot split the projection weights across ranks. It also fails in practice, without an assertion, when the GPUs assigned to the TP group differ in VRAM or compute, since the even weight split then fits some ranks and not others.

View the fix

Multi-node training hangs with no error after rendezvous

Training hangs indefinitely after 'Rendezvous complete' with no error, no progress, and no crash across nodes

A multi-node job that hangs with no error almost always means not every rank actually joined the process group: a mismatched world size, a wrong MASTER_ADDR or MASTER_PORT, a firewall blocking the ephemeral ports NCCL negotiates after rendezvous, or a node that silently OOMed are the four most common causes. Because NCCL blocks silently while waiting for missing ranks, there is often no error at all until you manually intervene or hit a long default timeout.

View the fix

DeepSpeed ZeRO configuration errors at training startup

AssertionError: Check batch related parameters. train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size

DeepSpeed refuses to start when its config's batch-size fields do not agree with each other, since train_batch_size must equal train_micro_batch_size_per_gpu times gradient_accumulation_steps times world_size exactly. A second common failure comes from enabling ZeRO stage 3 with CPU or NVMe offload on hardware that lacks enough system RAM or fast enough storage, which surfaces as tensor or contiguity errors rather than a clear resource message.

View the fix

GPU peer-to-peer (P2P) access not working between GPUs on the same node

RuntimeError: NCCL WARN Cuda failure 'peer access is not supported between these two devices'

GPU peer-to-peer (P2P) access fails when the PCIe topology, IOMMU or ACS settings, or the GPU model itself does not support a direct memory path between two devices, forcing all transfers through the CPU and host memory instead of GPU-to-GPU. Setting NCCL_P2P_DISABLE=1 is a useful diagnostic to confirm P2P is the problem, but it only removes the crash by falling back to a slower path; it does not restore the P2P bandwidth you actually need for good multi-GPU performance.

View the fix

Expected all tensors to be on the same device error in multi-GPU code

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

This error means a tensor was created or moved to a specific device, often cuda:0 or cpu, that does not match the device another tensor in the same operation lives on, which happens most often when leftover manual .to(device) calls from single-GPU code collide with automatic sharding from device_map='auto' or DistributedDataParallel. The fix is to trace exactly which tensor has the wrong device with a quick print of tensor.device, not to guess and add more .to() calls.

View the fix

HuggingFace Accelerate config mismatch causes wrong distributed launch

ValueError: You can't train a model that has been loaded with `device_map='auto'` in any distributed mode

Accelerate errors during launch almost always mean the saved configuration does not match the current machine's actual GPU count, node count, or distributed type, or that the model was loaded with device_map='auto' inference-style sharding and then also handed to accelerate's training-mode preparation, which are two incompatible placement strategies. Regenerating the config for the current machine, or passing explicit CLI overrides, resolves most cases.

View the fix

InfiniBand not detected, NCCL falls back to slow TCP sockets

NCCL INFO NET/IB : No device found. NCCL INFO NET/Socket : Using [eth0]

NCCL falls back to slow TCP sockets when it cannot find a usable InfiniBand device, most often because the IB kernel modules or rdma-core drivers are not installed or loaded, the fabric's subnet manager is not running so ports stay down, or NCCL environment variables point at the wrong network interface. Checking ibstat to confirm the hardware and fabric are actually up is the first step, before touching any NCCL environment variables.

View the fix

Pipeline parallelism scales poorly, throughput does not improve with more stages

Pipeline parallel training throughput does not scale with additional stages, GPU utilization drops sharply as pipeline depth increases

Pipeline parallelism scales poorly when the number of microbatches per training step is too small relative to the number of pipeline stages, because the unavoidable fill and drain bubble at the start and end of each step is proportional to the stage count minus one divided by the number of microbatches. With too few microbatches, GPUs spend a large fraction of every step idle waiting for the pipeline to fill or drain, and adding more stages without also adding more microbatches makes this worse, not better.

View the fix
RAG & Retrieval10 fixes

Wrong retrievals, hallucination despite context, and vector database issues.

RAG retrieves irrelevant or wrong documents

why does my RAG pipeline keep returning irrelevant chunks

RAG retrieves the wrong documents most often because the embedding model used to index the corpus differs from the one used at query time, or because chunks are large enough that a single embedding averages away the passage that actually answers the question. Fix embedding consistency and chunk granularity first, then add a reranker and metadata filters before touching the LLM prompt.

View the fix

RAG hallucinates even though the correct context was retrieved

LLM hallucinates an answer even though the correct context was retrieved

RAG hallucination with good context in hand usually means the prompt never explicitly instructs the model to answer only from the provided passages, or the correct passage is buried in the middle of a long context window where attention is weakest. Add an explicit grounding instruction, place the most relevant passage first, resolve conflicting retrieved chunks before generation, and give the model an explicit abstain option.

View the fix

pgvector similarity queries are slow

pgvector query taking seconds instead of milliseconds

pgvector queries are almost always slow because of an index and operator class mismatch (an index built for one distance function while queries use a different operator), a missing index entirely so Postgres falls back to a sequential scan, or search-time parameters (ef_search, probes) set too low. Confirm EXPLAIN ANALYZE shows an index scan, match the operator class to your distance function, and tune maintenance_work_mem before building large HNSW indexes.

View the fix

Embedding dimension mismatch after switching models

ERROR: different vector dimensions 1536 and 768

Different embedding models produce vectors of different fixed dimensions, so swapping models without re-embedding the entire corpus produces a hard dimension mismatch error or, worse, silently meaningless similarity scores if the column is resized without re-indexing. There is no shortcut: changing the embedding model requires re-embedding every document and rebuilding the vector index from scratch.

View the fix

Chunking splits tables and headers, destroying their meaning

RAG answers are wrong for anything in a table even though the table is in the source document

Generic character-count or token-count chunking treats a document as an undifferentiated stream of text, so it routinely cuts a table's header row away from its data rows, leaving a retrieved chunk full of numbers with no column labels to explain what they mean. The fix is structure-aware chunking that detects table boundaries, keeps the header attached to every chunk of that table's rows, and never splits mid-row.

View the fix

Vector index memory usage is too high (HNSW blowing up RAM)

server ran out of memory while building or loading the HNSW index

Vector index memory usage exceeds raw embedding size because HNSW stores a graph of neighbor connections on top of every vector, typically adding 1.5 to 3 times the raw vector size in overhead depending on the M parameter, while IVFFlat has lower overhead but worse recall at the same speed. Estimate memory as dimension times row count times 4 bytes for raw vectors, then add graph overhead, and consider quantization if the total does not fit in available RAM.

View the fix

PDF text extraction produces garbled or out-of-order text

extracted PDF text has words run together or in the wrong order

Garbled PDF extraction almost always comes from using a text-layer extractor on a document that does not have the kind of text layer it expects: scanned or image-based PDFs need OCR, multi-column layouts need layout-aware extraction to preserve reading order, and ligature characters need Unicode normalization. Match the extraction method to the actual document type, and route scanned engineering drawings to a vision model instead of text extraction entirely.

View the fix

RAG citations point to the wrong source document

citation shown to the user does not match the content actually used in the answer

Wrong citations almost always come from a broken or ambiguous mapping between a retrieved chunk and its original source document, often introduced when chunks are re-ordered, deduplicated, or overlapped during ingestion without carrying a stable source id and offset through every processing step. Assign a stable, immutable id to every chunk at creation time, carry it through embedding, storage, retrieval, and generation, and verify at generation time that the cited id matches the chunk actually used.

View the fix

Adding a reranker isn't improving RAG results

reranker makes no measurable difference to retrieval accuracy

A reranker that shows no improvement usually means it is only reordering a candidate pool that was already too small, top-3 to top-5, to contain the correct answer, so there is nothing better to promote, or the wrong reranker model was chosen for the domain. Retrieve a wider candidate set of 20-50 before reranking, verify the reranker model actually outperforms your vector search on a labeled evaluation set, and budget the added latency deliberately rather than treating it as free.

View the fix

Vector database connection errors under load

FATAL: sorry, too many clients already

Vector database connection errors under production load are almost always pool exhaustion, either too many application processes each opening their own connections, or a pool sized for development traffic rather than real concurrent RAG query volume, not an actual network or database outage. Use a connection pooler sized for your real concurrency, add retry logic with exponential backoff for transient failures, and separate TLS/auth failures from timeout and pool errors since they need different fixes.

View the fix
Containers & Kubernetes10 fixes

GPU passthrough, NVIDIA container toolkit, and Kubernetes scheduling problems.

GPU not visible inside a Docker container

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

Docker containers cannot see a host GPU unless the NVIDIA Container Toolkit is installed and the nvidia runtime is registered with the daemon, since containers are isolated from host devices by default. The fix is almost always to install nvidia-container-toolkit, run nvidia-ctk runtime configure, restart Docker, and launch with --gpus all. If nvidia-smi already fails on the host itself, the problem is the driver, not Docker.

View the fix

nvidia-container-cli errors when starting a GPU container

nvidia-container-cli: initialization error: driver error: failed to process request

nvidia-container-cli initialization errors mean the host's NVIDIA kernel module failed to load or the driver's supported CUDA version does not meet the minimum your container image requires. Check nvidia-smi on the bare host first; if it fails there, fix the kernel module or driver before touching Docker. If the host is healthy, compare its CUDA support against your image's requirement and either upgrade the driver or use an older image tag.

View the fix

Kubernetes GPU pod stuck in Pending

0/5 nodes are available: 5 Insufficient nvidia.com/gpu

A GPU pod stays Pending when no node advertises the nvidia.com/gpu resource because the device plugin is down or missing, the pod requests more GPUs than any single node has, or a taint, toleration, or nodeSelector mismatch blocks placement on the GPU pool. Always start with kubectl describe pod, since the Events section states the exact blocking reason rather than leaving you to guess between these causes.

View the fix

LLM container image is tens of gigabytes and slow to pull

Error response from daemon: no space left on device (during docker pull)

LLM container images balloon past ten or twenty gigabytes almost always because model weights were copied directly into a layer instead of mounted at runtime, or because a devel CUDA base image and unstaged build tools shipped into production by mistake. Remove weights from the Dockerfile, switch to a runtime base image and a multi-stage build, and image size typically drops by an order of magnitude without any change to the serving code.

View the fix

Model takes minutes to load every time a container restarts

Loading checkpoint shards: 100%|... [took 8+ minutes]

A model that reloads slowly on every container restart almost always means the weight directory is not backed by a persistent volume, so each restart re-downloads or cold-reads the full checkpoint instead of hitting a warm cache. Mount a persistent volume for the model cache path, set HF_HUB_OFFLINE=1 once weights are local, and if load time is still slow, benchmark your storage backend's raw throughput since network storage is a common hidden bottleneck.

View the fix

NVIDIA GPU Operator pods stuck installing or crashlooping

nvidia-driver-daemonset pod: Error: failed to load kernel module nvidia: Device or resource busy

GPU Operator installation problems almost always come from a preinstalled host driver conflicting with the Operator's own driver container, or from Node Feature Discovery never labeling GPU nodes so downstream components stay unscheduled. Check kubectl get pods -n gpu-operator first to see which subsystem is failing, then confirm whether the node has a preexisting driver and whether NFD applied the expected NVIDIA labels.

View the fix

Dataloader crashes with a shared memory error inside a container

RuntimeError: DataLoader worker (pid 1234) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory

PyTorch DataLoader workers crash with a bus error inside Docker or Kubernetes because the default container shared memory allocation is only 64MB, far too small for multi-worker data loading. Set --shm-size explicitly in Docker, or add a Memory-medium emptyDir volume at /dev/shm in Kubernetes, sized to your batch size and worker count, and the crash disappears without touching your training code.

View the fix

Kubernetes PersistentVolumeClaim errors when serving model weights

Warning FailedMount ... MountVolume.SetUp failed for volume "model-weights": rpc error: code = Internal

PersistentVolumeClaim errors serving model weights almost always come from using a ReadWriteOnce volume with more than one replica, since that access mode only allows a single node to mount it at a time. Switch to a ReadOnlyMany-capable storage class, mount weights read-only, and set volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches; if the volume mounts fine but loading is still slow, the real problem is storage throughput, not access mode.

View the fix

Setting up a container registry for air-gapped Kubernetes deployments

Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: no such host

Air-gapped Kubernetes clusters cannot reach public registries or the Hugging Face Hub, so image pulls fail at DNS resolution unless a local registry mirror is stood up ahead of time and populated from a connected staging environment. Mirror container images and model weights as two distinct pipeline steps, distribute the internal registry's CA certificate to every node, and sign mirrored artifacts so provenance, not just reachability, is auditable, which is a mandatory control in ITAR and CMMC environments.

View the fix

Kubernetes readiness probe fails while the model is still loading

Readiness probe failed: HTTP probe failed with statuscode: 503

LLM service pods get killed or marked unready during startup because default Kubernetes readiness and liveness probes assume a service starts in seconds, while loading multi-gigabyte weights into GPU memory can take minutes. Add a startupProbe sized with a failureThreshold times periodSeconds budget that comfortably exceeds your worst-case load time; Kubernetes suppresses readiness and liveness checks entirely until the startup probe succeeds, which stops premature restarts without needing a fragile fixed initialDelaySeconds guess.

View the fix
Performance & Production10 fixes

Slow tokens, low GPU utilization, latency spikes, and runaway inference cost.

LLM inference is much slower in production than in benchmarks

vllm serving is way slower in production than the benchmark numbers showed

Production inference is usually slower than a benchmark because real traffic exposes problems a single-request test never hits: full-precision weights instead of BF16/FP16, no continuous batching so requests queue one at a time, CPU-bound tokenization or post-processing sitting in front of the GPU call, or hardware whose memory bandwidth cannot keep up with the model size and concurrency you actually see. Fix the dtype and batching first, they account for most of the gap, then profile the request path for CPU-bound steps.

View the fix

High time to first token (TTFT) on LLM inference requests

time to first token is several seconds even for short prompts

High time to first token almost always comes from one of four sources: a long prompt makes the prefill pass compute-bound and simply takes time to process, the server has no prefix caching so a repeated system prompt or RAG context is recomputed on every request, the model or GPU had to cold-start (weights loading, CUDA graph capture, JIT warmup), or the request sat in a queue behind other requests before its prefill even began. Prefix caching and admission-aware queueing fix most production cases.

View the fix

GPU utilization stays low during LLM inference even under load

nvidia-smi shows gpu utilization under 30 percent while the model is serving requests

Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.

View the fix

LLM serving throughput collapses once load increases past a certain point

throughput drops sharply and latency spikes once concurrent requests pass a threshold

Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.

View the fix

Inference gets much slower as context length grows toward 32k, 64k, or 128k tokens

latency increases sharply as prompt length or context window grows

Long-context slowness is not a bug, it is the fundamental cost structure of attention: self-attention compute scales roughly quadratically with sequence length in the prefill pass, and the KV cache that must be stored per token scales linearly with sequence length, multiplying memory pressure across every concurrent request. A 128k-token context is not the same cost as eight 16k-token contexts, it is dramatically more expensive per request in both compute and memory, which is why advertised max context length is rarely the practical operating point for concurrent production traffic.

View the fix

Not sure how to tune batch size for LLM inference throughput vs latency

increasing max-num-seqs or batch size does not improve throughput as expected

Batch size is a direct tradeoff between throughput and per-request latency: larger batches keep the GPU busier and raise aggregate tokens-per-second, but each additional concurrent sequence adds contention for the same compute and memory, increasing the latency of every individual request. The right batch size is not the largest one that fits in memory, it is the point on that curve, the knee, where added throughput per unit of batch size starts costing more latency than your SLO allows, and it should be derived from measurement against your actual latency target, not a fixed default.

View the fix

GPU or host memory usage keeps growing in a long-running LLM service

memory usage climbs steadily over hours or days until the service crashes or is restarted

Slow memory growth over hours or days in an LLM service is rarely a leak in the model itself, it is almost always one of: a KV cache pool that grows because completed sequences are not being freed correctly, client sessions or connections that are opened but never closed, LoRA adapters that accumulate in memory across many fine-tuned variants without eviction, or memory fragmentation that reduces effectively usable memory even though nothing is technically leaked. Isolating which of these it is requires tracking memory over time correlated with request volume, adapter count, and connection count separately.

View the fix

Same prompt produces different outputs across requests or replicas

identical prompt returns different responses each time it is sent

Inconsistent outputs for an apparently identical prompt usually come from one of three sources: sampling is not actually deterministic (temperature above zero and no fixed seed), continuous batching introduces small floating-point nondeterminism because the exact batch composition changes token-level numerics run to run, or different replicas behind a load balancer are quietly running different quantization or even different model revisions. Full bit-for-bit determinism is hard to guarantee in batched GPU inference, but the practical fix is to control sampling explicitly and make sure all replicas are provably running the same model artifact.

View the fix

Cost per token for self-hosted LLM inference is higher than expected

cost per million tokens on our own gpus is not much cheaper than a hosted api

Cost per token is dominated by GPU utilization far more than by hardware choice: an underutilized GPU serving one request at a time can cost more per token than a well-tuned smaller GPU serving at full continuous-batching concurrency. Before concluding self-hosting is not worth it, check utilization, whether the model is right-sized for the task, whether quantization and prefix/response caching are in use, and whether the comparison to an API is even apples-to-apples once amortization is accounted for.

View the fix

Model output quality dropped noticeably after quantization

quantized model gives noticeably worse answers than the full precision version

Quality degradation after quantization usually comes from choosing too aggressive a quantization level for the model size and task, quantizing layers that are unusually sensitive to precision loss (often attention output projections and the final layers), or from trusting perplexity as the only quality signal when perplexity can look nearly unchanged while task-specific accuracy drops meaningfully. FP8 is close to lossless for most models and tasks, while INT4 methods carry real risk that must be validated with a task-specific eval set before shipping, not assumed safe from a perplexity number alone.

View the fix

Debugging this stack is not your team's job

We design, deploy and operate on-prem AI so your engineers can build product instead of chasing CUDA errors.

Talk to an On-Prem AI EngineerFree Sizing Calculators