Prompt caching can cut both latency and cost, but only when your requests share a large, stable prefix. Here is how to get real wins instead of noise.
Cache the stable prefix
Put everything that rarely changes — system prompt, tool definitions, few-shot examples — at the very start of the request, and the volatile user input at the end. The longer the shared prefix, the bigger the saving.
Measure hit rate, not vibes
Track cached vs uncached input tokens per route. If your hit rate is low, your prefix is probably changing in ways you did not expect — a timestamp, a shuffled example order, or a per-user string sneaking into the prefix.
Where it will not help
Short prompts and one-off requests see little benefit; the prefix is too small to matter. Caching is a lever for high-volume, templated workloads — exactly where the savings compound.