Table of contents
What does caching optimization actually measure?
Not all traffic reaches the Gateway in the same state. Some applications and agents already have caching configured efficiently. Others send requests with no caching at all. And sometimes context isn’t reused enough for caching to generate net savings.
nexos.ai measures only the additional cost impact created by its cache optimization. If an agent already uses provider caching efficiently, those existing savings aren't attributed to nexos.ai optimization. If there’s no caching in place, nexos.ai steps in to cache repeated context. When that context is reused, it creates additional savings.
That’s why the numbers can look very different depending on your traffic:
Cost optimization | Cache outcomes with nexos.ai Gateway |
|---|---|
Positive | nexos.ai captured additional caching savings |
Zero | Little or nothing additional to optimize |
Negative | Cache wasn’t reused enough to offset its cost |
Scenario 1: Repeated context isn't being cached
Your application sends some of the same context with every request, but caching isn't configured. A typical agent request looks something like this:
System prompt
- Tool definitions
- MCP configuration
- Project context
- Conversation history
- New user request
Only part of the "new request" is new. A large part of the context may be repeated across multiple requests. Without caching, the provider processes that repeated context as regular input again and again, and you pay for it each time.
nexos.ai Gateway identifies the repeated context and applies the right caching configuration for the provider handling the request. When that context is reused in subsequent requests, it can be served from cache instead of being processed again at the full input-token price.
Result: positive optimization. nexos.ai introduced caching where it wasn't being used before and the cached context was reused enough to generate savings.
Scenario 2: Caching is already optimized
Some agents handle this well on their own. Claude Code is a useful example, since its traffic may already arrive with strong caching behavior built in. When it does, there's little additional opportunity left for the Gateway to capture. Some applications already make effective use of provider caching, and in those cases there simply isn't much left on the table.
That means you can have a high cache hit rate and real provider-level savings while seeing ~$0 nexos.ai caching optimization.
The savings are still happening. They simply weren’t created by nexos.ai, so we don’t attribute them to nexos.ai Gateway optimization.
Result: ~$0 optimization. Caching is already working efficiently, and nexos.ai found little or nothing additional to optimize.
Run every AI model in one Gateway
One secure, lightweight layer to access, route, and manage 200+ models.
Scenario 3: Cache wasn’t reused enough
Creating a cache has an upfront cost. The savings come afterward, when the same context is reused and can be read from cache at a lower cost than processing it again as regular input.
Text | Deployment |
|---|---|
Cache created | write cost incurred |
00:00 - 00:05 | cache window open |
Insufficient reuse | too few matching requests |
Expired | window closes |
Net cost | write costs not recovered |
nexos.ai uses a 5-minute cache window, a commonly used standard for LLM prompt caching. For caching to generate savings, the cached context needs to be reused enough within those five minutes to offset the initial cache-write cost.
But sometimes there isn't enough reuse within that window. The user or agent may not send another request within five minutes, the next request may use different context, or the workflow may simply move on. In those cases, the cache expires before enough reuse happens to recover the initial write cost.
Result: negative optimization. Within that cache window, the context wasn't reused enough for the savings to offset the initial write cost.
Note: This doesn’t mean caching was configured incorrectly. Reuse depends on subsequent application, agent, and user activity. It's also worth checking the time range you're looking at. A short window can show negative optimization while a wider one turns positive, since periods of strong reuse offset the caches that didn't pay off.
Caching everything is the wrong goal
Worth stating plainly, because it sits underneath most of the confusion about negative numbers – more caching does not automatically mean better results. So, if a request is unlikely to be repeated, caching it can add cost without generating enough savings to offset it. You pay the write cost and never see a read
That calls for a judgment about probability rather than a switch to flip. Every caching decision is made before the next request arrives, based on how the traffic has behaved so far.
This is also why the number can move in both directions. The metric measures net cost impact, so it can be positive or negative. It reflects whether caching paid off, not how much context was cached.
Why caching gets complicated across providers
Caching sounds straightforward when looking at a single provider, but in practice real-world traffic is more complex. Providers implement caching differently, agents can route across multiple models, and traffic arrives from applications with very different caching behavior.
- Providers handle caching differently: caching APIs, TTLs, pricing, cache-write cost, cache-read cost, whether caching is automatic or explicit, and what cache-control information needs to be passed. OpenAI caches automatically on supported models, while Anthropic offers both automatic behavior and explicit cache control. Setup tuned for one cannot be assumed to transfer to the other.
- Agents add another layer of complexity: one application may route different tasks across multiple providers – one model might handle classification, another reasoning, and another coding. Caching behavior then varies by route rather than by application. Multi-model frameworks like OpenClaw illustrate the point well, since the same framework can produce quite different caching outcomes depending on which provider serves the request.
- The traffic arrives from different applications: coding agents, internal applications, third-party tools, and agent frameworks can all send traffic through the same Gateway, each with its own request structure and caching behavior, or with no caching configured at all. Two applications hitting the same model can present completely different optimization opportunities, which is why the number reflects your whole traffic mix rather than any single integration.
Managing this application by application means accounting for provider-specific caching behavior, request structure, TTLs, and the caching patterns of every tool and agent and revisiting that configuration as models, providers, and workflows change.
nexos.ai Gateway puts that optimization in one place. Instead of every application and team having to manage caching independently, nexos.ai identifies and optimizes caching opportunities across the traffic flowing through the Gateway.
Conclusion: What caching optimization numbers are telling you
Caching optimization measures one thing: the additional cost impact nexos.ai created on top of what your traffic was already doing. A positive number means the Gateway found caching opportunities you weren't capturing and turned them into savings. Zero means there was little left to improve, which is often a sign your own configuration is already working well. A negative number means a cache was created and the context wasn't reused enough within the window to cover what it cost to write.
All three are normal readings. Caching you configured stays attributed to you, and only the additional impact nexos.ai created is counted as ours. That's what gives a positive number its meaning: it represents savings that didn't exist before your traffic reached the Gateway.
