# Why time to first token is high on your LLM endpoint, and how to fix it > Source: https://netray.co/troubleshoot/high-time-to-first-token > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer High time to first token almost always comes from one of four sources: a long prompt makes the prefill pass compute-bound and simply takes time to process, the server has no prefix caching so a repeated system prompt or RAG context is recomputed on every request, the model or GPU had to cold-start (weights loading, CUDA graph capture, JIT warmup), or the request sat in a queue behind other requests before its prefill even began. Prefix caching and admission-aware queueing fix most production cases. ## Error Title High time to first token (TTFT) on LLM inference requests ## Error Signature time to first token is several seconds even for short prompts ## Alternate Signatures - streaming response takes a long time before the first token appears - ttft spikes under load even though total generation time looks normal ## Category performance ## Stack - vllm - sglang - pytorch ## Affects Any streaming LLM endpoint, especially RAG applications with long injected context and multi-tenant endpoints serving many concurrent users ## Quick Fix ### Cut TTFT in the right order **Steps:** - Measure prefill time separately from queue time and decode time; most serving frameworks expose this in metrics or logs. - If the prompt (especially a shared system prompt or RAG context) is long and repeated across requests, enable prefix caching (vLLM's automatic prefix caching or SGLang's RadixAttention) so the shared prefix is computed once. - If TTFT is high only on the first request after startup, that is a cold start: warm the server with a few dummy requests before accepting real traffic. - If TTFT rises specifically under concurrent load, check queue depth and admission control; the request may be waiting for a batching slot, not actually computing. - For very long prompts, consider chunked prefill so a single huge prefill does not block other requests' decode steps in the same batch. ## Symptoms - First streamed token takes multiple seconds even on a lightly loaded server - TTFT scales with prompt length far more than expected - TTFT is fine on isolated tests but spikes as soon as several users hit the endpoint at once - The very first request after a deploy is dramatically slower than subsequent ones ## Causes ### Long prompt makes prefill compute-bound **Likelihood:** most-common **Explanation:** Prefill processes the entire input prompt in one forward pass, and unlike decode, its cost scales with the square of sequence length for the attention computation (before optimizations) plus linearly for the feed-forward layers. A long system prompt, few-shot examples, or a large RAG context can make prefill take hundreds of milliseconds to seconds on its own, before a single output token is produced. **Fix:** Shorten the prompt where possible, and cache the shared prefix (system prompt, RAG template) with prefix caching so only the unique tail of the prompt needs a fresh prefill. ### No prefix caching for a shared system prompt or repeated context **Likelihood:** common **Explanation:** Without prefix caching, every request recomputes the KV cache for the entire prompt from scratch, even if 90 percent of it (system prompt, tool definitions, RAG boilerplate) is identical to the previous request. That recomputation is pure wasted prefill time that directly adds to TTFT. **Fix:** Enable automatic prefix caching in vLLM or use SGLang's RadixAttention, and structure prompts so the shared portion comes first and is byte-identical across requests. **Commands:** - vllm serve MODEL_NAME --enable-prefix-caching ### Cold start: weight loading, CUDA graph capture, or kernel warmup on the first request **Likelihood:** common **Explanation:** The first request after server startup (or after autoscaling spins up a new replica) can trigger CUDA graph capture, JIT compilation of fused kernels, or lazy weight loading, all of which add one-time latency that has nothing to do with the prompt itself. **Fix:** Send a handful of warmup requests immediately after the server reports healthy, before routing real traffic to it, especially important for autoscaled replicas. ### Queueing delay before prefill starts **Likelihood:** occasional **Explanation:** If max-num-seqs or the scheduler's admitted-request limit is already at capacity, a new request waits in queue before its prefill even begins. From the client's perspective this looks identical to slow prefill, but the GPU has not touched the request yet. **Fix:** Track queue wait time as a separate metric from compute time, and size max-num-seqs and replica count to your actual concurrency and latency SLO rather than raw throughput alone. ### One large prefill blocking other requests' decode steps in the same batch **Likelihood:** rare **Explanation:** Without chunked prefill, a very long prompt's prefill can occupy an entire scheduling step, delaying the decode steps of other in-flight requests, which shows up as TTFT (and inter-token latency) spikes correlated with occasional long prompts from other users. **Fix:** Enable chunked prefill so long prefills are split across multiple scheduling steps and interleaved with ongoing decodes. **Commands:** - vllm serve MODEL_NAME --enable-chunked-prefill ## Diagnostics ### Break down latency into queue, prefill, and decode **Command:** curl -s localhost:8000/metrics | grep -E 'time_to_first_token|queue_time|prefill' **Interpretation:** If queue_time dominates, the fix is admission control or capacity, not prefill optimization. If prefill/TTFT dominates and correlates with prompt length, prefix caching or shortening the prompt is the fix. ### Compare TTFT on the first request after startup vs. steady state **Command:** time curl -s localhost:8000/v1/completions -d '{"prompt":"hello","max_tokens":1}' **Interpretation:** A large gap between the first call and later calls confirms a cold-start problem that a warmup routine will fix. ### Check whether prefix caching is active **Command:** curl -s localhost:8000/v1/models | python -m json.tool **Interpretation:** Confirm the server was started with prefix caching enabled; a surprising number of slow-TTFT reports trace back to this flag simply being off. ## Prevention - Structure prompts with the shared/static portion first so prefix caching can actually help - Warm every new replica with dummy requests before it receives production traffic - Set a TTFT-specific SLO and alert, separate from total latency, since it is driven by different causes - Budget prefill cost explicitly when designing RAG context sizes rather than treating context as free ## When To Escalate If TTFT stays high after enabling prefix caching and chunked prefill, and queue time is low, the prompts themselves are simply too long for your latency budget on the current hardware; that calls for a smaller/faster model for the latency-sensitive path, more GPUs to add parallel capacity, or a redesign of how much context you inject per request. ## Faq ### Does a bigger GPU always reduce time to first token? **Answer:** It helps for compute-bound long prompts, since prefill throughput scales with GPU compute, but it does not fix queueing delay or missing prefix caching. Diagnose which component (queue, prefill, cold start) dominates before assuming more hardware is the answer. ### Is TTFT the same thing as latency? **Answer:** No. TTFT is the time until the first output token streams back, driven mostly by prefill and queueing. Total latency also includes decode time for every subsequent token, which is driven by batching and throughput. A system can have low TTFT and still feel slow if decode is throttled, or the reverse. ### How much does prefix caching actually save? **Answer:** It scales with how much of the prompt is shared across requests. A RAG pipeline with a large fixed system prompt and tool schema can see the cached portion's prefill cost drop to near zero on cache hits, often cutting TTFT by more than half when the shared prefix is a large fraction of total prompt length. ## Search terms high time to first token, ttft slow llm, prefix caching vllm, streaming first token delay, chunked prefill, cold start llm inference, rag context slow prefill --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json