The response cache
A repeat of a deterministic request, answered from memory rather than by a provider. Off unless you turn it on, and never a saving.
VITNA_CACHE_TTL_SECONDS=300 vitna-runtime
vitna-relay --cache 300What it answers
Only a request that asked for the most likely answer, once, and whole:
temperatureset to0. An absent temperature is the provider's default, which samples, so a request that did not say0is never answered from memory.- One choice:
nabsent or1. - Not streamed.
And only one the runtime reads as a chat completion. An Anthropic request that names a model is passed to Anthropic as it was written, and the cache leaves it alone as well.
Zero is not a promise that a provider answers the same way twice. What the cache promises is narrower: an answer it gives is the answer a provider gave to exactly this request, within the last VITNA_CACHE_TTL_SECONDS.
Exactly this request
The key is the whole body the runtime would forward, in a canonical form in which key order does not matter and everything else does: the messages to the last character, the tool calls and their ids, max_tokens, stop, tool_choice, every field. For a lane it also holds the policy digest, the lane, and the caller's x-vitna-cohort, since those decide which attempt serves the request. So a reload that changes what a lane serves never answers with what the previous decision put there, and two callers on different sides of a traffic share never answer each other.
A hit
The answer is the one stored, with x-vitna-cache: hit on the response. Its ledger row says what happened and nothing more:
| Field | On a hit |
|---|---|
serve, via | cache |
cache_hit | true |
outcome | served, because the caller got an answer |
units | 0, because no provider served it |
inputTokens, outputTokens and the cache parts | null, because nobody was asked anything |
sku, decision, provider, model | the serve whose answer it repeats, so the answer still resolves to the decision that authorized it |
vitna-runtime-push and vitna-runtime-profile count hits apart from their figures and say how many, so a lane answered from memory neither reads as a sku that failed nor inflates what the sku did. /status counts hits, misses and entries. Nothing anywhere adds up what a hit would have cost: a price needs the date it applied on, and the runtime counts nothing in money.
Asking for a fresh answer
A caller's cache-control header is honoured: no-cache asks the provider even where an answer is stored, and replaces it; no-store keeps this request's answer out of memory.
What it holds
At most 1024 answers and 64 MB, the least recently used going first, in the serving process's memory. Nothing is written to disk, and a restart starts it empty. An answer larger than the whole allowance is not kept.
Answers are kept after redaction has put a request's secrets back into them, so a cached answer may hold a secret the caller sent. It stays in the same process that already held it.
