# Why Kubernetes health checks fail while your LLM is still loading, and how to fix it > Source: https://netray.co/troubleshoot/llm-service-health-check-failing > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer LLM service pods get killed or marked unready during startup because default Kubernetes readiness and liveness probes assume a service starts in seconds, while loading multi-gigabyte weights into GPU memory can take minutes. Add a startupProbe sized with a failureThreshold times periodSeconds budget that comfortably exceeds your worst-case load time; Kubernetes suppresses readiness and liveness checks entirely until the startup probe succeeds, which stops premature restarts without needing a fragile fixed initialDelaySeconds guess. ## Error Title Kubernetes readiness probe fails while the model is still loading ## Error Signature Readiness probe failed: HTTP probe failed with statuscode: 503 ## Alternate Signatures - Liveness probe failed: connection refused - pod restarting in CrashLoopBackOff during model load ## Category containers ## Stack - kubernetes - docker - vllm ## Affects Any containerized LLM server, vLLM, TGI, or a custom FastAPI wrapper, in Kubernetes where model load time exceeds default probe timing ## Quick Fix ### Fastest path to a stable health check **Steps:** - Check how long model load actually takes from your logs with kubectl logs | grep -i load. - Add a startupProbe hitting your health endpoint with periodSeconds and failureThreshold sized to exceed that load time with margin. - Confirm your health endpoint returns unhealthy while the model is loading, not just while the process is starting. - Keep readinessProbe and livenessProbe configured normally; the startupProbe suppresses them until it first succeeds. - Redeploy and watch the pod's restart count to confirm it no longer cycles during load. ## Symptoms - pod cycles through CrashLoopBackOff specifically during model load, not after - readiness probe failed with statuscode 503 repeating for several minutes then succeeding once - liveness probe kills the pod right as loading was about to finish - service is marked ready before it can actually serve a real inference request without errors ## Causes ### Readiness or liveness probe fires before a multi-minute model load finishes **Likelihood:** most-common **Explanation:** Kubernetes health probes default to short initial delays and timeouts designed for typical web services that start in seconds; an LLM server that spends one to several minutes loading multi-gigabyte weights into GPU memory has not opened its HTTP port or is still returning 503 during that window, so the default probe configuration marks the pod unready or, worse, kills and restarts it via the liveness probe before it ever finishes loading once. **Fix:** Add a dedicated startupProbe with a generous failureThreshold times periodSeconds budget covering your worst-case load time, and let readiness and liveness probes only begin after the startup probe succeeds, since startupProbe suppresses the other two probes until it passes. **Commands:** - startupProbe: httpGet: {path: /health, port: 8000}, failureThreshold: 60, periodSeconds: 10 - readinessProbe: httpGet: {path: /health, port: 8000}, periodSeconds: 5 ### No startupProbe configured at all, only readinessProbe with a large initialDelaySeconds guess **Likelihood:** common **Explanation:** Some manifests try to work around slow model loads by inflating readinessProbe's initialDelaySeconds to several minutes, but this is a static guess that either wastes time on fast restarts or still fails on unusually slow loads, such as a cold storage read or a larger model swapped in later, whereas a startupProbe actively polls until success rather than waiting a fixed duration blindly. **Fix:** Replace a large fixed initialDelaySeconds with a startupProbe that polls periodically up to a generous failureThreshold, so the pod becomes ready as soon as loading actually finishes rather than after a worst-case fixed wait. **Commands:** ### Health endpoint returns 200 before the model is actually ready to serve requests **Likelihood:** common **Explanation:** A naive health endpoint that only checks whether the HTTP server process is alive, rather than checking whether the model has finished loading into memory, will report healthy immediately at process start; Kubernetes then routes real inference traffic to a pod that is still loading, causing request timeouts or errors even though the readiness probe itself is passing. **Fix:** Implement the health endpoint to check actual model-ready state, for example a boolean set only after the model load call returns successfully, rather than just confirming the web server thread is running. **Commands:** ### Liveness probe timeout too aggressive for a server briefly blocked during heavy inference load **Likelihood:** occasional **Explanation:** A liveness probe with a short timeoutSeconds can misfire during legitimate periods where the server's event loop or worker thread is momentarily saturated handling a burst of inference requests, causing Kubernetes to kill and restart an otherwise healthy pod under load, which then compounds the problem by forcing a fresh multi-minute reload right when capacity is most needed. **Fix:** Set a longer timeoutSeconds and a higher failureThreshold on the liveness probe specifically, separate from the startup probe's settings, so transient load spikes do not trigger unnecessary restarts. **Commands:** ## Diagnostics ### Check current probe configuration **Command:** kubectl get pod -o yaml | grep -A8 Probe **Interpretation:** Reveals whether a startupProbe exists at all, and what initialDelaySeconds, periodSeconds, and failureThreshold are currently set. ### Check how long model load actually takes **Command:** kubectl logs | grep -i load **Interpretation:** Compare the logged load duration against your probe's total failureThreshold times periodSeconds budget; if load time exceeds that budget, the probe will always fail. ### Check pod restart count and reason **Command:** kubectl get pod -o jsonpath='{.status.containerStatuses[0].restartCount}' **Interpretation:** A high restart count combined with logs showing repeated model loads confirms the liveness probe is killing the pod mid-load. ### Manually hit the health endpoint during load **Command:** kubectl exec -- curl -s localhost:8000/health **Interpretation:** Confirms whether the health endpoint itself distinguishes between process-alive and model-ready states. ## Prevention - Always configure a startupProbe with a failureThreshold times periodSeconds budget that comfortably exceeds worst-case model load time - Implement health endpoints that check real model-ready state, not just process liveness - Set liveness probe timeouts and thresholds generously enough to tolerate legitimate load spikes, not just startup - Log model load duration explicitly so probe budgets can be tuned from real data rather than guesses ## When To Escalate If model load times are growing because models themselves are getting larger or storage is slower than expected, tuning probe timing is only a workaround; the real fix is addressing load time directly, faster storage, weight caching, or a smaller quantized model, and that becomes a capacity planning conversation rather than a probe configuration one. ## Faq ### What is the difference between a startupProbe and a readinessProbe for a slow-loading service? **Answer:** A startupProbe is specifically designed for slow-starting containers: while it is running, Kubernetes disables both the readiness and liveness probes entirely, so a long model load cannot trigger a premature restart. Once the startupProbe succeeds once, it stops running permanently for that container's lifetime, and the readiness and liveness probes take over for ongoing health checks. A readinessProbe alone, without a startupProbe, does not have this protective effect and can be misused as a liveness probe substitute, causing restarts during legitimate startup delays. ### Why does my liveness probe kill the pod even though the model eventually loads fine? **Answer:** Without a startupProbe, the liveness probe begins checking immediately, after its own initialDelaySeconds, and, if it fails enough times before the model finishes loading, Kubernetes will restart the container, resetting the load process back to zero. This creates a loop where the pod never gets far enough into loading to pass the probe before being killed again. Adding a startupProbe with a generous budget for your actual load time stops the liveness probe from evaluating at all until loading is confirmed complete. ### How do I set failureThreshold and periodSeconds for a startupProbe correctly? **Answer:** Multiply periodSeconds by failureThreshold to get the total time budget before Kubernetes gives up on startup; set that product to comfortably exceed your worst observed model load time, including cold cache scenarios, with margin. For example, periodSeconds 10 and failureThreshold 60 gives a 600 second, 10 minute, budget, which covers most large-model cold loads while still failing fast if the container is genuinely broken rather than just slow. ### Should my health endpoint just check if the server process is running? **Answer:** No, that alone produces false-positive readiness during model load. The health endpoint should reflect the actual application state, typically a flag or variable set to true only after the model load call completes successfully, so it returns unhealthy or a 503 while loading is in progress and only returns healthy once the service can actually handle inference requests correctly. ## Search terms kubernetes readiness probe failed llm, liveness probe failed connection refused model loading, startup probe model load kubernetes, llm health check failing pod restart, kubernetes crashloopbackoff during model load, initialdelayseconds model server --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json