For the first three months of Nilux's beta, every session started the same way: the system prompt, tool definitions, and conversation context were sent fresh to the LLM on every single request. That's 60-80k tokens of identical prefix, billed at full price, every time a user typed a command.
KV caching changed everything. Here's how it works, why it matters, and what it means for your bill.
What is KV caching?
Large language models process text by converting it into key-value pairs at each layer of the transformer. When you send a prompt, the model computes these KV pairs for every token. On the next request, if the prompt starts with the same prefix, the provider can skip recomputing those pairs and reuse them from cache.
This isn't a Nilux feature — it's built into modern LLM providers. But how much of your prompt hits the cache depends on how the client structures its requests. That's where we come in.
Why Nilux gets 94% cache hit rates
Most AI tools rebuild the prompt from scratch on every request. We made three design decisions that keep the prefix stable:
- System prompt is built once per session and never changes.
- Tool definitions are cached separately, reused for every LLM call.
- Messages are append-only. Compact uses the same system prompt to preserve the KV prefix.
The result: in production, 94.17% of input tokens hit the cache. Standard model: 95.3%. Pro: 96.6%.
What this means for your bill
Cached tokens cost about 90% less. A typical request sends ~65k input tokens, and with a 94% cache rate only ~3.9k of them are fresh. That turns a ~$0.018 request into roughly $0.003 — about 6x cheaper.
Real production numbers
From 6,794 real LLM calls (April 24 – May 22, 2026): 373M input tokens, 351M cached (94.17%). Total cost: $8.53. Without caching, that same workload would have cost roughly $52 in provider fees — about a 6x reduction.
What breaks the cache
KV cache works by prefix matching. If any token changes, cache invalidates from that point. Common mistakes: changing system prompt mid-session, reordering tools, mutating earlier messages, or using a different prompt for compact.
The bottom line: KV caching isn't a feature we built. It's a property of how LLMs work that we designed our entire architecture around. You pay for your actual work, not for re-explaining your project on every keystroke.