Get started

The response cache

Source

A repeat of a deterministic request, answered from memory rather than by a provider. Off unless you turn it on, and never a saving.

bash
VITNA_CACHE_TTL_SECONDS=300 vitna-runtime
vitna-relay --cache 300

What it answers

Only a request that asked for the most likely answer, once, and whole:

  • temperature set to 0. An absent temperature is the provider's default, which samples, so a request that did not say 0 is never answered from memory.
  • One choice: n absent or 1.
  • Not streamed.

And only one the runtime reads as a chat completion. An Anthropic request that names a model is passed to Anthropic as it was written, and the cache leaves it alone as well.

Zero is not a promise that a provider answers the same way twice. What the cache promises is narrower: an answer it gives is the answer a provider gave to exactly this request, within the last VITNA_CACHE_TTL_SECONDS.

Exactly this request

The key is the whole body the runtime would forward, in a canonical form in which key order does not matter and everything else does: the messages to the last character, the tool calls and their ids, max_tokens, stop, tool_choice, every field. For a lane it also holds the policy digest, the lane, and the caller's x-vitna-cohort, since those decide which attempt serves the request. So a reload that changes what a lane serves never answers with what the previous decision put there, and two callers on different sides of a traffic share never answer each other.

A hit

The answer is the one stored, with x-vitna-cache: hit on the response. Its ledger row says what happened and nothing more:

FieldOn a hit
serve, viacache
cache_hittrue
outcomeserved, because the caller got an answer
units0, because no provider served it
inputTokens, outputTokens and the cache partsnull, because nobody was asked anything
sku, decision, provider, modelthe serve whose answer it repeats, so the answer still resolves to the decision that authorized it

vitna-runtime-push and vitna-runtime-profile count hits apart from their figures and say how many, so a lane answered from memory neither reads as a sku that failed nor inflates what the sku did. /status counts hits, misses and entries. Nothing anywhere adds up what a hit would have cost: a price needs the date it applied on, and the runtime counts nothing in money.

Asking for a fresh answer

A caller's cache-control header is honoured: no-cache asks the provider even where an answer is stored, and replaces it; no-store keeps this request's answer out of memory.

What it holds

At most 1024 answers and 64 MB, the least recently used going first, in the serving process's memory. Nothing is written to disk, and a restart starts it empty. An answer larger than the whole allowance is not kept.

Answers are kept after redaction has put a request's secrets back into them, so a cached answer may hold a secret the caller sent. It stays in the same process that already held it.