> Source: https://vitna.ai/docs/runtime/cache
> Raw Markdown: https://vitna.ai/docs/runtime/cache.md
> Corpus index: https://vitna.ai/llms.txt

# The response cache

A repeat of a deterministic request, answered from memory rather than by a
provider. Off unless you turn it on, and never a saving.

```bash
VITNA_CACHE_TTL_SECONDS=300 vitna-runtime
vitna-relay --cache 300
```

## What it answers

Only a request that asked for the most likely answer, once, and whole:

- `temperature` set to `0`. An absent temperature is the provider's default,
  which samples, so a request that did not say `0` is never answered from
  memory.
- One choice: `n` absent or `1`.
- Not streamed.

And only one the runtime reads as a chat completion. An Anthropic request that
names a model is [passed to Anthropic as it was written](serving.md#the-other-dialect),
and the cache leaves it alone as well.

Zero is not a promise that a provider answers the same way twice. What the cache
promises is narrower: an answer it gives is the answer a provider gave to exactly
this request, within the last `VITNA_CACHE_TTL_SECONDS`.

## Exactly this request

The key is the whole body the runtime would forward, in a canonical form in which
key order does not matter and everything else does: the messages to the last
character, the tool calls and their ids, `max_tokens`, `stop`, `tool_choice`,
every field. For a lane it also holds the policy digest, the lane, and the
caller's `x-vitna-cohort`, since those decide which attempt serves the request.
So a reload that changes what a lane serves never answers with what the previous
decision put there, and two callers on different sides of a traffic share never
answer each other.

## A hit

The answer is the one stored, with `x-vitna-cache: hit` on the response. Its
[ledger](ledger.md) row says what happened and nothing more:

| Field | On a hit |
|---|---|
| `serve`, `via` | `cache` |
| `cache_hit` | `true` |
| `outcome` | `served`, because the caller got an answer |
| `units` | `0`, because no provider served it |
| `inputTokens`, `outputTokens` and the cache parts | null, because nobody was asked anything |
| `sku`, `decision`, `provider`, `model` | the serve whose answer it repeats, so the answer still resolves to the decision that authorized it |

`vitna-runtime-push` and `vitna-runtime-profile` count hits apart from their
figures and say how many, so a lane answered from memory neither reads as a sku
that failed nor inflates what the sku did. `/status` counts hits, misses and
entries. Nothing anywhere adds up what a hit would have cost: a price needs the
date it applied on, and the runtime counts nothing in money.

## Asking for a fresh answer

A caller's `cache-control` header is honoured: `no-cache` asks the provider even
where an answer is stored, and replaces it; `no-store` keeps this request's
answer out of memory.

## What it holds

At most 1024 answers and 64 MB, the least recently used going first, in the
serving process's memory. Nothing is written to disk, and a restart starts it
empty. An answer larger than the whole allowance is not kept.

Answers are kept after [redaction](redaction.md) has put a request's secrets back
into them, so a cached answer may hold a secret the caller sent. It stays in the
same process that already held it.
