Why Kubernetes health checks fail while your LLM is still loading, and how to fix it
Readiness probe failed: HTTP probe failed with statuscode: 503
Also appears as
- Liveness probe failed: connection refused
- pod restarting in CrashLoopBackOff during model load
Short answer
LLM service pods get killed or marked unready during startup because default Kubernetes readiness and liveness probes assume a service starts in seconds, while loading multi-gigabyte weights into GPU memory can take minutes. Add a startupProbe sized with a failureThreshold times periodSeconds budget that comfortably exceeds your worst-case load time; Kubernetes suppresses readiness and liveness checks entirely until the startup probe succeeds, which stops premature restarts without needing a fragile fixed initialDelaySeconds guess.
Affects: Any containerized LLM server, vLLM, TGI, or a custom FastAPI wrapper, in Kubernetes where model load time exceeds default probe timing
Fastest path to a stable health check
- 1Check how long model load actually takes from your logs with kubectl logs <pod> | grep -i load.
- 2Add a startupProbe hitting your health endpoint with periodSeconds and failureThreshold sized to exceed that load time with margin.
- 3Confirm your health endpoint returns unhealthy while the model is loading, not just while the process is starting.
- 4Keep readinessProbe and livenessProbe configured normally; the startupProbe suppresses them until it first succeeds.
- 5Redeploy and watch the pod's restart count to confirm it no longer cycles during load.
How to confirm this is your problem
- pod cycles through CrashLoopBackOff specifically during model load, not after
- readiness probe failed with statuscode 503 repeating for several minutes then succeeding once
- liveness probe kills the pod right as loading was about to finish
- service is marked ready before it can actually serve a real inference request without errors
Root causes and fixes
Readiness or liveness probe fires before a multi-minute model load finishes
Kubernetes health probes default to short initial delays and timeouts designed for typical web services that start in seconds; an LLM server that spends one to several minutes loading multi-gigabyte weights into GPU memory has not opened its HTTP port or is still returning 503 during that window, so the default probe configuration marks the pod unready or, worse, kills and restarts it via the liveness probe before it ever finishes loading once.
Fix: Add a dedicated startupProbe with a generous failureThreshold times periodSeconds budget covering your worst-case load time, and let readiness and liveness probes only begin after the startup probe succeeds, since startupProbe suppresses the other two probes until it passes.
startupProbe: httpGet: {path: /health, port: 8000}, failureThreshold: 60, periodSeconds: 10
readinessProbe: httpGet: {path: /health, port: 8000}, periodSeconds: 5No startupProbe configured at all, only readinessProbe with a large initialDelaySeconds guess
Some manifests try to work around slow model loads by inflating readinessProbe's initialDelaySeconds to several minutes, but this is a static guess that either wastes time on fast restarts or still fails on unusually slow loads, such as a cold storage read or a larger model swapped in later, whereas a startupProbe actively polls until success rather than waiting a fixed duration blindly.
Fix: Replace a large fixed initialDelaySeconds with a startupProbe that polls periodically up to a generous failureThreshold, so the pod becomes ready as soon as loading actually finishes rather than after a worst-case fixed wait.
Health endpoint returns 200 before the model is actually ready to serve requests
A naive health endpoint that only checks whether the HTTP server process is alive, rather than checking whether the model has finished loading into memory, will report healthy immediately at process start; Kubernetes then routes real inference traffic to a pod that is still loading, causing request timeouts or errors even though the readiness probe itself is passing.
Fix: Implement the health endpoint to check actual model-ready state, for example a boolean set only after the model load call returns successfully, rather than just confirming the web server thread is running.
Liveness probe timeout too aggressive for a server briefly blocked during heavy inference load
A liveness probe with a short timeoutSeconds can misfire during legitimate periods where the server's event loop or worker thread is momentarily saturated handling a burst of inference requests, causing Kubernetes to kill and restart an otherwise healthy pod under load, which then compounds the problem by forcing a fresh multi-minute reload right when capacity is most needed.
Fix: Set a longer timeoutSeconds and a higher failureThreshold on the liveness probe specifically, separate from the startup probe's settings, so transient load spikes do not trigger unnecessary restarts.
Diagnostic commands
Check current probe configuration
kubectl get pod <pod> -o yaml | grep -A8 Probe
Reveals whether a startupProbe exists at all, and what initialDelaySeconds, periodSeconds, and failureThreshold are currently set.
Check how long model load actually takes
kubectl logs <pod> | grep -i load
Compare the logged load duration against your probe's total failureThreshold times periodSeconds budget; if load time exceeds that budget, the probe will always fail.
Check pod restart count and reason
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].restartCount}'A high restart count combined with logs showing repeated model loads confirms the liveness probe is killing the pod mid-load.
Manually hit the health endpoint during load
kubectl exec <pod> -- curl -s localhost:8000/health
Confirms whether the health endpoint itself distinguishes between process-alive and model-ready states.
Stopping it from happening again
- Always configure a startupProbe with a failureThreshold times periodSeconds budget that comfortably exceeds worst-case model load time
- Implement health endpoints that check real model-ready state, not just process liveness
- Set liveness probe timeouts and thresholds generously enough to tolerate legitimate load spikes, not just startup
- Log model load duration explicitly so probe budgets can be tuned from real data rather than guesses
When this becomes an architecture problem
If model load times are growing because models themselves are getting larger or storage is slower than expected, tuning probe timing is only a workaround; the real fix is addressing load time directly, faster storage, weight caching, or a smaller quantized model, and that becomes a capacity planning conversation rather than a probe configuration one.
Frequently asked questions
What is the difference between a startupProbe and a readinessProbe for a slow-loading service?
A startupProbe is specifically designed for slow-starting containers: while it is running, Kubernetes disables both the readiness and liveness probes entirely, so a long model load cannot trigger a premature restart. Once the startupProbe succeeds once, it stops running permanently for that container's lifetime, and the readiness and liveness probes take over for ongoing health checks. A readinessProbe alone, without a startupProbe, does not have this protective effect and can be misused as a liveness probe substitute, causing restarts during legitimate startup delays.
Why does my liveness probe kill the pod even though the model eventually loads fine?
Without a startupProbe, the liveness probe begins checking immediately, after its own initialDelaySeconds, and, if it fails enough times before the model finishes loading, Kubernetes will restart the container, resetting the load process back to zero. This creates a loop where the pod never gets far enough into loading to pass the probe before being killed again. Adding a startupProbe with a generous budget for your actual load time stops the liveness probe from evaluating at all until loading is confirmed complete.
How do I set failureThreshold and periodSeconds for a startupProbe correctly?
Multiply periodSeconds by failureThreshold to get the total time budget before Kubernetes gives up on startup; set that product to comfortably exceed your worst observed model load time, including cold cache scenarios, with margin. For example, periodSeconds 10 and failureThreshold 60 gives a 600 second, 10 minute, budget, which covers most large-model cold loads while still failing fast if the container is genuinely broken rather than just slow.
Should my health endpoint just check if the server process is running?
No, that alone produces false-positive readiness during model load. The health endpoint should reflect the actual application state, typically a flag or variable set to true only after the model load call completes successfully, so it returns unhealthy or a 503 while loading is in progress and only returns healthy once the service can actually handle inference requests correctly.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Kubernetes Cluster Cost Calculator
Estimate the true monthly and annual cost of a Kubernetes cluster, including compute, control plane fees, managed service premiums, and utilization waste.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Related problems
Model takes minutes to load every time a container restarts
A model that reloads slowly on every container restart almost always means the weight directory is not backed by a persistent volume, so each restart re-downloads or cold-reads the full checkpoint instead of hitting a warm cache. Mount a persistent volume for the model cache path, set HF_HUB_OFFLINE=1 once weights are local, and if load time is still slow, benchmark your storage backend's raw throughput since network storage is a common hidden bottleneck.
Kubernetes GPU pod stuck in Pending
A GPU pod stays Pending when no node advertises the nvidia.com/gpu resource because the device plugin is down or missing, the pod requests more GPUs than any single node has, or a taint, toleration, or nodeSelector mismatch blocks placement on the GPU pool. Always start with kubectl describe pod, since the Events section states the exact blocking reason rather than leaving you to guess between these causes.
vLLM server won't start (port in use, auth, VRAM, or unsupported architecture)
vLLM server startup failures collapse into four buckets: the port is already bound by another process, Hugging Face auth is missing or expired for a gated repo, there isn't enough free VRAM for the requested model and context, or the installed vLLM version doesn't yet support the model's architecture. Read the last traceback line, not just the top, to tell them apart.
LLM container image is tens of gigabytes and slow to pull
LLM container images balloon past ten or twenty gigabytes almost always because model weights were copied directly into a layer instead of mounted at runtime, or because a devel CUDA base image and unstaged build tools shipped into production by mistake. Remove weights from the Dockerfile, switch to a runtime base image and a multi-stage build, and image size typically drops by an order of magnitude without any change to the serving code.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideLLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.
GuideOn-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.